update documentation for release

docs/dependencies.rst

@@ -1,16 +1,20 @@
 Dependencies
 ------------
 
-The following dependencies need to be on your system prior to `PyRate` installation:
+The following dependencies need to be on your system (or in your working
+environment) prior to `PyRate` installation:
 
 - Python_, versions 3.6, 3.7 or 3.8.
 - GDAL_, versions 3.0.2 or 3.0.4
 - `Open MPI`_, versions 2.1.6, 3.0.4, 3.1.4 or 4.0.2
 
-The versions of each package stated above have been tested to work.
+The versions of each package stated above have been tested to work using
+`Travis CI`_ continuous integration testing.
 
 .. _Python: https://www.python.org/downloads/
 .. _GDAL: https://gdal.org/download.html
 .. _`Open MPI`: https://www.open-mpi.org/software/ompi/v4.0/
+.. _`Travis CI`: https://travis-ci.org/github/GeoscienceAustralia/PyRate
 
-Other Python dependencies that will be installed are listed in ``PyRate/requirements.txt``.
+Other Python dependencies that will be installed are listed in
+``PyRate/requirements.txt``.

docs/history.rst

@@ -3,7 +3,7 @@
 Release History
 ===============
 
-0.5.0 (2020-08-31)
+0.5.0 (2020-09-08)
 ------------------
 Added
 +++++
@@ -34,11 +34,16 @@ Changed
   files are saved to disk, regardless of whether ``cohmask`` is 0 or 1.
 - Parallelisation capability is refactored - MPI and multiprocessing both now use a common
   tiling framework for ``stack``, ``timeseries`` and ``mst`` algorithms.
+- Introduced a simplified and standardised file naming format for files produced by the
+  ``prepifg`` step. Information previously saved in the filename (e.g. ``ifglksx``, ``ifglksy``,
+  and ``ifgcropopt`` values) is now saved to the geotiff header instead.
 
 Removed
 +++++++
 - Redundant ``tscal`` config parameter was deprecated - not needed now there is a ``timeseries``
   step invokable on the command line.
+- Unused ``Pillow``, ``cython`` and ``glob2`` dependencies.
+- Deprecated function ``pyrate.prepifg_helper.prepare_ifgs``, which is no longer needed.
 
 0.4.3 (2020-08-04)
 ------------------

docs/hpc.rst

@@ -7,47 +7,34 @@ HPC platforms may differ.
 
 .. _`National Computational Infrastructure (NCI)`: https://nci.org.au/ 
 
-Login to Gadi and clone the `PyRate` repository:
-
-::
+Login to Gadi and clone the `PyRate` repository::
 
     ssh <user_name>@gadi.nci.org.au
     cd ~
     git clone https://github.com/GeoscienceAustralia/PyRate.git
 
-Load the required Gadi modules (this will also remove the default NCI GDAL
-Python bindings so we can build and use our own):
-
-::
-
-    source ~/PyRate/scripts/nci_load_modules.sh
-
-Create a Python virtual environment:
-
-::
+Create a Python virtual environment::
 
     python3 -m venv ~/PyRateVenv
     source ~/PyRateVenv/bin/activate
 
-Install `PyRate`:
+Load the required Gadi modules (this will also remove the default NCI GDAL
+Python bindings so we can build and use our own)::
+
+    source ~/PyRate/scripts/nci_load_modules.sh
 
-::
+Install `PyRate`::
 
     cd ~/PyRate
     python3 setup.py install
 
-
 Following this, `PyRate` will be available for PBS jobs. To verify the 
-installation, first run an interactive session:
-
-::
+installation, first run an interactive session::
 
     qsub -I -q express -l walltime=01:00:00,mem=16Gb,ncpus=4,wd
 
 Once the session has started, you will need to reactivate your virtual 
-environment and reload the required modules:
-
-::
+environment and reload the required modules::
 
     source ~/PyRateVenv/bin/activate
     source ~/PyRate/scripts/nci_load_modules.sh

docs/installation.rst

@@ -2,7 +2,8 @@ Installation
 ============
 
 This is an installation guide to get `PyRate` running on various platforms.
-Follow the instructions to install `PyRate` and run a small toy example.
+Follow the instructions to install `PyRate` and run a small toy example using
+the test dataset included in the repository.
 
 .. include:: dependencies.rst
 
@@ -12,14 +13,13 @@ Platforms
 PyPI
 ^^^^
 
-`PyRate` and its Python dependencies can be installed directly from the Python Package Index (PyPI_):
-
-.. _PyPI: https://pypi.org/project/Py-Rate/
-
-::
+`PyRate` and its Python dependencies can be installed directly from the
+Python Package Index (PyPI_)::
 
     pip install Py-Rate
 
+.. _PyPI: https://pypi.org/project/Py-Rate/
+
 .. include:: ubuntu.rst
 .. include:: docker.rst
 .. include:: hpc.rst
@@ -28,21 +28,17 @@ PyPI
 Verify Installation
 -------------------
 
-To verify `PyRate` has been successfully installed, run the workflow with the
-example config file and the small dataset included in the repository:
+To verify `PyRate` has been successfully installed, run the full processing
+workflow with the example config file and the small dataset included
+in the repository. If you compiled the ``pyrate`` executable program::
+
+    pyrate workflow -f input_parameters.conf
 
-::
+If you installed from the Python Package Index (PyPI_)::
 
-    pyrate conv2tif -f input_parameters.conf
-    pyrate prepifg -f input_parameters.conf
-    pyrate correct -f input_parameters.conf
-    pyrate timeseries -f input_parameters.conf
-    pyrate stack -f input_parameters.conf
-    pyrate merge -f input_parameters.conf
+    python3 pyrate/main.py workflow -f input_parameters.conf
 
 If the installation has been successful, this workflow will complete without 
-errors and geotiff files will be available in:
+errors and geotiff files will be available in::
 
-:: 
-
     ~/PyRate/out

docs/pyrate.process.rst → docs/pyrate.correct.rst

@@ -1,7 +1,7 @@
-pyrate correct Script
+PyRate Correct Script
 ========================
 
-.. automodule:: pyrate.process
+.. automodule:: pyrate.correct
     :members:
     :undoc-members:
     :show-inheritance:

docs/pyrate.__main__.rst → docs/pyrate.main.rst

@@ -1,12 +1,14 @@
 PyRate Main Script
 ==================
 
-.. automodule:: pyrate.__main__
+.. automodule:: pyrate.main
 
    .. rubric:: Functions
    .. autosummary::
 
       conv2tif
       prepifg
-      process
+      correct
+      timeseries
+      stack
       merge

docs/scripts.rst

@@ -6,6 +6,6 @@ PyRate Scripts
 
    pyrate.conv2tif
    pyrate.prepifg
-   pyrate.process
+   pyrate.correct
    pyrate.merge
-   pyrate.__main__
+   pyrate.main

docs/troubleshooting.rst

@@ -7,41 +7,6 @@ If your issue is not covered below, please contact the `Geoscience Australia InS
 
 .. _`Geoscience Australia InSAR Team`: mailto:[email protected]
 
-Corrections being skipped
--------------------------
-**Problem**: When running the ``process`` step, many corrections are reported as ``Skipped: interferograms already corrected``, but I want to try different processing parameters!
-
-::
-
-    >> pyrate correct -f input_parameters.conf
-    16:43:16 main 97 24732 INFO 0/0 Verbosity set to INFO.
-    16:43:16 shared 1294 24732 INFO 0/0 Running process serially
-    16:43:17 process 86 24732 INFO 0/0 Found 13 unique epochs in the 17 interferogram network
-    16:43:17 process 134 24732 INFO 0/0 Searching for best reference pixel location
-    16:43:18 process 155 24732 INFO 0/0 Selected reference pixel coordinate: (38, 58)
-    16:43:18 process 170 24732 INFO 0/0 Calculating orbital correction
-    16:43:18 shared 1255 24732 INFO 0/0 Skipped: interferograms already corrected
-    16:43:18 process 198 24732 INFO 0/0 Calculating reference phase
-    16:43:19 shared 1255 24732 INFO 0/0 Skipped: interferograms already corrected
-    16:43:19 process 105 24732 INFO 0/0 Calculating minimum spanning tree matrix
-    16:43:19 process 342 24732 INFO 0/0 Calculating the temporal variance-covariance matrix
-    16:43:20 process 391 24732 INFO 0/0 Calculating time series using SVD method
-    16:43:20 timeseries 152 24732 INFO 0/0 Calculating timeseries in serial
-    16:43:21 process 323 24732 INFO 0/0 Calculating rate map from stacking
-    16:43:21 process 326 24732 INFO 0/0 Stacking of tile 0
-    16:43:21 stack 64 24732 INFO 0/0 Calculating stack rate in serial
-    16:43:21 process 314 24732 INFO 0/0 PyRate workflow completed
-
-**Reason**: `PyRate` updates the phase values in the input interferogram geotiff files as corrections are applied during the ``process`` step. Metadata is then added to the geotiff header to indicate the correction has been applied. This metadata is then checked upon subsequent runs to see if the correction should be applied.
-
-**Solution**: Start again from ``prepifg`` step, creating new cropped/multi-looked interferograms that have not been corrected.
-
-.. note::
-
-    We plan to change this workflow behaviour in a future `PyRate` release, recognising that
-    it would be convenient to be able to quickly test the impact of parameter changes.
-
-
 ValueError: too many values to unpack (expected 2)
 --------------------------------------------------
 **Problem**: During ``prepifg`` step, the following error is encountered:
@@ -104,12 +69,12 @@ Stack Rate map appears to be blank/empty
 
 **Reason**: The ``maxsig`` parameter is too low, resulting in stack rate values being replaced by NaNs. ``maxsig`` is a threshold for masking stack rate pixels according to the corresponding stack error estimate saved in ``out/tmpdir/stack_error_*.npy``.
 
-**Solution**: Increase ``maxsig``, then re-run ``process`` and ``merge`` steps. Maximum permittable value for ``maxsig`` is 1000 mm.
+**Solution**: Increase ``maxsig``, then re-run the ``merge`` step. Maximum permittable value for ``maxsig`` is 1000 mm.
 
 
 Failure of APS spatial low pass filter
 ---------------------------------------
-**Problem**: Atmospheric corrections during “process“ fails due to the interpolated grid used for correction being empty:
+**Problem**: Atmospheric corrections during ``correct`` step fails due to the interpolated grid used for correction being empty:
 
 ::
 
@@ -155,8 +120,8 @@ Failure of APS spatial low pass filter
 In general, users are advised to input a network of small-baseline interferograms
 that has at least 2 interferometric connections per SAR image epoch. Furthermore,
 make sure that ``ts_pthr``, ``pthr`` and ``tlpfpthr`` are smaller than the number
-of image epochs. To check that ``process`` worked correctly, users can check that
-the ``tsincr_*.npy`` and ``tscuml*.npy`` arrays in the ``/<outdir>/tmpdir`` contain numeric values and not NaNs.
+of image epochs. To check that the spatio-temporal filters worked correctly, users can check that
+the numpy arrays saved at ``/<outdir>/aps_error/*aps_error.npy`` contain numeric values and not NaNs.
 
 
 Out of memory errors
@@ -167,12 +132,12 @@ Out of memory errors
 
     joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {EXIT(1), EXIT(1), EXIT(1)}
 
-**Solution**: Increase the amount of memory available. On HPC systems this can be done by increasing the value provided to the ``mem`` argument when submitting a PBS job, e.g.:
-
-::
+**Solution**: Increase the amount of memory available. On HPC systems this can be done by increasing the value provided to the ``mem`` argument when submitting a PBS job, e.g.::
 
     mem=32Gb
 
+If no more memory can be called upon, users can try running the job in serial, or reducing the size of the interferograms by increasing the multi-looking factors applied during ``prepifg`` (parameters ``ifglksx`` and ``ifglksy``).
+
 Incorrect modules loaded on Gadi
 ----------------------------------
 **Problem**: `PyRate` requires certain versions of Python, GDAL and Open MPI to be loaded on Gadi and other HPC systems. While sourcing the `PyRate/scripts/nci_load_modules.sh` script will load the correct modules, you may need to unload previously unloaded modules.

docs/ubuntu.rst

@@ -17,3 +17,5 @@ Clone the repository, install required packages and install `PyRate`:
     export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/include/gdal
 
     python3 setup.py install
+
+These commands will compile the executable program ``pyrate``.