Merge pull request #149 from GeoscienceAustralia/geomedian

Geomedian 4.0.0 - new geomedian version WIP

docs/_components/shift-in-summary-product-grid-tech-alert.md

@@ -0,0 +1,8 @@
+To accommodate an expanded area of coverage of Australia's external territories, the [DEA Summary Product Grid](/guides/reference/collection_3_summary_grid/) will soon be shifted. The south-west origin point of the grid will be shifted from `-5472000.0, -2688000.0` to `-6912000.0, -4416000.0` (EPSG:3577). Therefore, all tile grid references will shift 18 tiles west and 15 tiles south. For instance, a tile reference of `x10y10` will change to `x28y25`. For a preview, see the [provisional version of the expanded DEA Summary Product Grid](https://maps.dea.ga.gov.au/#share=s-avXJqwjUtf55qGUmweYY5KYoVnI) on DEA Maps.
+
+All new versions of our 'summary derivative products' will be affected by this change (but existing versions of the products will not be changed).
+
+This change has already been applied to the latest versions of the following products: [v4.0.0 of DEA Geometric Median and Median Absolute Deviation (GeoMAD)](/data/product/dea-geometric-median-and-median-absolute-deviation-landsat/).
+
+This change will be applied to the new versions of the following products, when released: [DEA Water Observations Statistics (Landsat)](/data/product/dea-water-observations-statistics-landsat/), [DEA Fractional Cover Percentiles](/data/product/dea-fractional-cover-percentiles-landsat/), [DEA Mangroves](/data/product/dea-mangrove-canopy-cover-landsat/), and [DEA Land Cover](/data/product/dea-land-cover-landsat/).
+

docs/_files/geomedian/lineage.mermaid

@@ -0,0 +1,39 @@
+flowchart LR
+
+subgraph DataSources["Data Source(s)"]
+    direction TB
+    L5Data["`Landsat 5 (TM)`"]
+    L7Data["`Landsat 7 (ETM+)`"]
+    L8Data["`Landsat 8 (OLI-TIRS)`"]
+    L9Data["`Landsat 9 (OLI-TIRS)`"]
+end
+
+subgraph InputProducts["Input Product(s)"]
+    direction TB
+    L5SurfaceReflectance["`Landsat 5 Surface Reflectance`"]
+    L7SurfaceReflectance["`Landsat 7 Surface Reflectance`"]
+    L8SurfaceReflectance["`Landsat 8 Surface Reflectance`"]
+    L9SurfaceReflectance["`Landsat 9 Surface Reflectance`"]
+end
+
+subgraph ProcessingMethods["Processing Method(s)"]
+    direction TB
+    Processing1["`GeoMAD algorithms with morphological cloud operations on fmask (opening and dilation)`"]
+end
+
+subgraph Outputs["Output(s)"]
+    direction TB
+    GeomadLandsat5["`DEA GeoMAD Landsat 5`"]
+    GeomadLandsat7["`DEA GeoMAD Landsat 7`"]
+    GeomadLandsat8And9["`DEA GeoMAD Landsat 8 and 9`"]
+end
+
+L5Data --> L5SurfaceReflectance --> ProcessingMethods
+L7Data --> L7SurfaceReflectance --> ProcessingMethods
+L8Data --> L8SurfaceReflectance --> ProcessingMethods
+L9Data --> L9SurfaceReflectance --> ProcessingMethods
+
+ProcessingMethods --> GeomadLandsat5
+ProcessingMethods --> GeomadLandsat7
+ProcessingMethods --> GeomadLandsat8And9
+

docs/_static/styles/global/_utilities.scss

@@ -0,0 +1,4 @@
+.overlow-x-hidden {
+    overflow-x: hidden;
+}
+

docs/conf.py

@@ -88,6 +88,7 @@
 myst_heading_anchors = 6
 myst_all_links_external = True
 
+nbsphinx_requirejs_path = ""
 nbsphinx_execute = "never"
 
 external_toc_path = "table_of_contents.yaml"

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_access.md

@@ -0,0 +1,13 @@
+% ## Access constraints
+
+% ## Use constraints
+
+:::{dropdown} Accessing the OGC web services
+
+When accessing via the [Digital Earth Australia OGC Web Services](https://ows.dea.ga.gov.au/), the layer names for the three Landsat 8, Landsat 7 and Landsat 5 products respectively are:
+* `ga_ls8c_nbart_gm_cyear_3`    
+* `ga_ls7e_nbart_gm_cyear_3`    
+* `ga_ls5t_nbart_gm_cyear_3`
+
+:::
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_credits.md

@@ -0,0 +1,10 @@
+## Acknowledgments
+
+The high-dimensional statistics algorithms incorporated in this product is the work of Dr Dale Roberts, Australian National University.
+
+## License and copyright
+
+© Commonwealth of Australia (Geoscience Australia).
+
+Released under [Creative Commons Attribution 4.0 International Licence](https://creativecommons.org/licenses/by/4.0/).
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_data.yaml

@@ -0,0 +1,102 @@
+# See the Product metadata fields documentation:
+# https://docs.dev.dea.ga.gov.au/public_services/dea_knowledge_hub/product_metadata_fields.html
+
+# Overview
+
+title: DEA Geometric Median and Median Absolute Deviation (Landsat)
+long_title: Geoscience Australia Landsat Geometric Median and Median Absolute Deviation Collection 3
+header_image: /_files/cmi/Geomedian_Australia_2014.jpg
+version: 3.1.0
+is_latest_version: false
+latest_version_link: /data/product/dea-geometric-median-and-median-absolute-deviation-landsat/
+is_provisional: false
+product_type: Derivative
+spatial_data_type: Raster
+time_span:
+  start: 1986
+  end: 2022
+update_frequency: Yearly
+next_update: null
+product_ids:
+  - ga_ls5t_nbart_gm_cyear_3
+  - ga_ls7e_nbart_gm_cyear_3
+  - ga_ls8c_nbart_gm_cyear_3
+
+parent_products:
+  name: Landsat 5, 7 and 8 NBART and Observational Attributes
+  link: /data/category/dea-surface-reflectance/
+collection:
+  name: Geoscience Australia Landsat Collection 3
+  link: null
+doi: 10.26186/146261
+ecat: "146261"
+nci: null
+licence:
+  name: Creative Commons Attribution 4.0 International Licence
+  link: https://creativecommons.org/licenses/by/4.0/
+
+citations:
+  data_citation: "Geoscience Australia, 2022. DEA Geometric Median and Median Absolute Deviation (Landsat). Geoscience Australia, Canberra. https://dx.doi.org/10.26186/146261"
+  paper_citation: null
+custom_citations:
+  - name: Geometric Median paper citation
+    citation: "Roberts, D., Mueller, N., Mcintyre, A., 2017. High-dimensional pixel composites from earth observation time series. IEEE Transactions on Geoscience and Remote Sensing, 55(11), 6254–6264. https://doi.org/10.1109/TGRS.2017.2723896"
+  - name: Median Absolute Deviation paper citation
+    citation: "Roberts, D., Dunn, B., Mueller, N., 2018. Open data cube products using high-dimensional statistics of time series. IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, 8647–8650. https://doi.org/10.1109/IGARSS.2018.8518312"
+
+tags:
+  - geoscience_australia_landsat_collection_3
+  - satellite_images
+  - geomedian
+  - geo_mad
+  - geometric_median
+  - median_absolute_deviation
+
+# Access
+
+maps:
+  - link: https://maps.dea.ga.gov.au/#share=s-fzHT65z4jscx9U1Nfb1FoKlQ7uU
+    name: See it on a map
+
+explorers:
+  - link: https://explorer.dea.ga.gov.au/products#baseline-satellite-data-group
+    name: Data explorer
+
+data:
+  - link: https://data.dea.ga.gov.au/?prefix=derivative
+    name: Access the data on AWS
+  - link: https://thredds.nci.org.au/thredds/catalog/jw04/ga_ls5t_nbart_gm_cyear_3/catalog.html
+    name: Annual Landsat 5 observations on NCI
+  - link: https://thredds.nci.org.au/thredds/catalog/jw04/ga_ls7e_nbart_gm_cyear_3/catalog.html
+    name: Annual Landsat 7 observations on NCI
+  - link: https://thredds.nci.org.au/thredds/catalog/jw04/ga_ls8c_nbart_gm_cyear_3/catalog.html
+    name: Annual Landsat 8 observations on NCI
+
+code_examples: null
+
+web_services:
+  - link: https://ows.dea.ga.gov.au/
+    name: Web Services
+
+custom: null
+
+# History
+
+old_versions:
+  - version: 2.1.0
+    title: DEA Surface Reflectance Median Absolute Deviation (Landsat)
+    slug: dea-surface-reflectance-median-absolute-deviation-landsat-2.1.0
+  - version: 2.0.0
+    title: DEA Surface Reflectance Geomedian (Landsat)
+    slug: dea-surface-reflectance-geomedian-landsat-2.0.0
+
+# Settings
+
+enable_overview: true
+enable_access: true
+enable_details: true
+enable_quality: true
+enable_history: true
+enable_faqs: false
+enable_credits: true
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_details.md

@@ -0,0 +1,105 @@
+## Background
+
+Large-scale image composites are increasingly important for a variety of applications such as land cover mapping, change detection, and the generation of high-quality data to parameterise and validate bio-physical and geophysical models. A number of compositing methodologies are being used in remote sensing in general; however challenges such as maintaining the spectral relationship between bands, mitigating against boundary artifacts due to mosaicking scenes from different epochs, and ensuring spatial regularity across the mosaic image still exist.
+
+The creation of good composite images is a particularly important technology since the opening of the Landsat archive by the United States Geological Survey. The greater availability of satellite imagery has resulted in demand to provide large regional mosaics that are representative of conditions over specific time periods while also being free of clouds and other unwanted image noise. One approach is the stitching together of a number of clear images. Another is the creation of mosaics where pixels from different epochs are combined based on some algorithm from a time series of observations. This 'pixel composite' approach to mosaic generation provides a more consistent result compared with stitching clear images due to the improved color balance created by the combining of one-by-one pixel representative images. Another strength of pixel-based composites is their ability to be automated for application to very large data collections and time series such as national satellite data archives.
+
+When deriving information from satellite images it may be beneficial to see how an area usually looks over a given year rather than how it was viewed at a single point in time. Other useful interpretations include how dynamic or variable the Australian landscape is over any given year, especially to help with understanding how the landscape changes, or to discriminate parts of the landscape that stay the same throughout the year (like bare rock), from those areas that go through big changes in cover (such as cropping areas). 
+
+By understanding general conditions (which can be thought of as "average" for the year) and the different patterns of variation, we can characterise various types of land cover and land use, and detect changes of significance in the landscape.
+
+The Geometric Median provides information on the general conditions of the landscape for a given year.
+
+The Median Absolute Deviation provides information on how the landscape is changing in the same year.
+
+The DEA GeoMAD product combines the Geometric Median and the Median Absolute Deviation in a single, annually produced package.
+
+## What this product offers
+
+This product provides statistical tools to exploit the time series of Earth observation data available in Digital Earth Australia, providing annual images of general conditions and how much an area changes for a given year.
+
+The geomedian part of the product provides an "average" cloud-free image over the given year. The geomedian image is calculated with a multi-dimensional median, using all the spectral measurements from the satellite imagery at the same time in order to maintain the relationships among the measurements.
+
+The median absolute deviation part of the product uses three measures of variance, each of which provides a "second order" high dimensional statistical composite for the given year. The three variance measures show how much an area varies from the "average" in terms of "distance" based on factors such as brightness and spectra:
+* Euclidean distance (EMAD)
+* Cosine (spectral) distance (SMAD)
+* Bray Curtis dissimilarity (BCMAD)
+
+Together, they provide information on variance in the landscape over the given year and are useful for change detection applications.
+
+% ## Data description
+
+## Applications
+
+The GeoMAD product is useful for:
+* Land cover mapping
+* Change detection and classification (such as for burn area mapping, crop mapping, urban area mapping)
+* General variance and change, so it can be used in machine learning for change detection
+* Environmental monitoring
+
+## Technical information
+
+### Geometric Median
+
+The surface reflectance geometric median (geomedian) provides an approach that leverages high-dimensional statistical theory to deliver a spectrally consistent, artefact-free pixel composite product.
+
+The geomedian is a pixel composite mosaic of a time series of earth observations. Essentially the value of a pixel in a geomedian image is the statistical median of all observations for that pixel from a period of time. For example the 2016 Landsat 8 geomedian image over an area will be the median of Landsat 8 pixels observed for that area in 2016.  
+
+An annual geometric median is a high-dimensional median calculated from the reflectance values drawn from a calendar year. The years available are each full calendar year since 1988. The annual geomedians of surface reflectance measurements are calculated per calendar year, on individual sensors (ie. only Landsat 5 TM or Landsat 7 ETM+ or Landsat 8 OLI).
+
+Multispectral satellite imagery, such as is available from Landsat and Sentinel-2, consists of multiple measurements per pixel: one for each spectral band. In order to create a meaningful median, a median pixel must take all concurrent spectral measurements into account simultaneously as a multi-dimensional set, rather than each measurement independently as occurs with a simple median. The geomedian is a high-dimensional statistic, which calculates a multi-dimensional median from all the spectral measurements from the satellite imagery at the same time, and maintains the relationships among the measurements. This provides a median of the physical conditions measured by the earth observations used to create it, and provides a good representation of a typical pixel observation devoid of outliers, and exhibits reduced spatial noise. 
+
+The annual geomedian provides an annual surface reflectance composite for any area of interest within the area covered by the DEA data cube, up to the entire spatial extent available (continental). It provides a cloud-free overview of the middle surface reflectance value for each year and as it is equivariant (i.e. a linear algorithm applied to a geomedian image is equal to a geomedian applied to a set of images on which the same linear algorithm was applied) can be used in further analyses such as Tasseled Cap, Principal Components Analysis and Normalised Difference Indices. It is useful in analyses requiring baseline conditions such as change detection.
+
+Surface reflectance geometric median products are derived from the DEA Surface Reflectance (SR) products, and provide a representation of the middle of surface reflectance over the time period, which can be considered "average" conditions. This is a synthetic representation of a time series rather than an actual observed pixel. 
+
+The geomedian (Roberts et al 2017) is used in preference to the mean, as the mean can be adversely effected by extrema whereas the median is less sensitive to outliers. The geomedian is used in preference to the basic median, as it preserves the physical relationship between spectral measurements which the basic median does not. The geometric median is used in preference to the medoid (Flood 2013) where a low noise composite is required. Where the provenance of each pixel in a composite is required the medoid is the preferred method and the geomedian should not be used.
+
+The input data used to calculate the geomedian are filtered to remove poor quality observations including cloud, cloud shadow, band saturation (of any band) using the Pixel Quality Assessment (PQA) product. To account for satellite availability and status the statistics are calculated using the following satellites/sensors for the following periods of time:
+* 1988-1999 : Landsat 5 only
+* 2000-2003 : Landsat 7 only
+* 2004-2007 : Landsat 5 and Landsat 7 (as separate geomedian products)
+* 2008: Landsat 7 only
+* 2009-2011: Landsat 5 and Landsat 7 (as separate geomedian products)
+* 2012 : Landsat 7 only
+* 2013 onward : Landsat 7 and Landsat 8 (as separate geomedian products)
+
+The primary uses of geomedian pixel composites are for baselines for change detection, and for broad regional image composites (such as national and continental mosaics).
+
+### Median Absolute Deviation
+
+The 'first order' statistics of a set of data include the mean and the median. Instead of the mean, for multi-dimensional datasets such as satellite imagery, the median provides a method to find general conditions for a time period while also maintaining the physical relationships between spectral measurements.
+
+A 'second order' statistic associated with a mean of a set of data is the standard deviation, providing a measure of data variance. The equivalent second order statistic associated with the median is the Median Absolute Deviation (MAD), which provides the associated variance measures for the median.
+
+The MAD is the median of absolute differences of the individual values in a set of data from their overall median. To calculate the MAD for a multi-dimensional dataset, such as the set of satellite images captured in a year, measures of "distance" from each multi-dimensional measurement (i.e. the set of spectral measurements for a pixel through time, being blue, green, red, near-infra-red, and short-wave-infra-red) to the mean are needed. However multi-dimensional distances can be calculated in different ways, providing different insights into the behaviour of pixels through time. The DEA GeoMAD product includes three MADs produced from different measures of distance as follows: 
+* Euclidean distance (EMAD) is more sensitive to changes in target brightness.
+* Cosine (spectral) distance (SMAD) is more sensitive to change in target spectral response.
+* Bray Curtis dissimilarity (BCMAD) is more sensitive to the distribution of the observation values through time.
+
+Each MAD provides information on different land cover change features and is useful in classification.
+
+The mathematical derivation of the three MADs can be found in Roberts et al (2018).
+
+## Lineage
+
+The GeoMAD is derived from Landsat surface reflectance data. The data are masked for cloud, shadows and other image artefacts using the associated pixel quality product to help provide as clear a set of observations as possible from which to calculate the medians. 
+
+The three MAD layers of the GeoMAD are calculated by computing the multidimensional distance between each observation in a time series of multispectral (or higher dimensionality such as hyperspectral) satellite imagery with the multidimensional median of the time series. The median used for this calculation is the geometric median corresponding to the time series. 
+
+The GeoMAD is calculated over annual time periods on Earth observations from a single sensor by default (such as the annual time series of Landsat 8 observations); however, it is applicable to multi-sensor time series of any length that computing resources can support.
+
+For the purposes of the default Digital Earth Australia product, GeoMADs are computed per calendar year, per sensor (Landsat 5, Landsat 7 and Landsat 8) from terrain-illumination-corrected surface reflectance data (Analysis Ready Data), and compared to the annual geometric median of that data.
+
+Note: the constituent pixels in the GeoMAD pixel composite mosaics are synthetic, meaning that the pixels have not been physically observed by the satellite. Rather they are the computed high-dimensional medians of a time series of pixels.
+
+% ## Processing steps
+
+% ## Software
+
+## References
+
+Roberts, D., Mueller, N., & Mcintyre, A. (2017). High-dimensional pixel composites from earth observation time series. *IEEE Transactions on Geoscience and Remote Sensing*, *55*(11), 6254–6264. [https://doi.org/10.1109/TGRS.2017.2723896](https://doi.org/10.1109/TGRS.2017.2723896)
+
+Roberts, D., Dunn, B., & Mueller, N. (2018). Open data cube products using high-dimensional statistics of time series. *IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium*, 8647–8650. [https://doi.org/10.1109/IGARSS.2018.8518312](https://doi.org/10.1109/IGARSS.2018.8518312)
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_faqs.md

@@ -0,0 +1,5 @@
+% ## Frequently asked questions
+
+% :::{dropdown} What is the question?
+% :::
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_history.md

@@ -0,0 +1,2 @@
+% ## Changelog
+

docs/data/old-version/dea-geometric-median-and-median-absolute-deviation-landsat-3.1.0/_overview_1.md

@@ -0,0 +1,4 @@
+## About
+
+The DEA Geometric Median and Median Absolute Deviation products use statistical analyses to provide information on variance in the landscape over the given year. They provide insight into the "average" conditions observed over Australia in a given year, as well as the amount of variability experienced around that average. These products are useful for monitoring change detection, such as from cropping, urban expansion or burnt area mapping. 
+