Update NowcastWorker mocks in unit tests #81
Comments
douglatornell
added a commit
that referenced
this issue
Oct 4, 2022
douglatornell
added a commit
that referenced
this issue
Oct 5, 2022
douglatornell
added a commit
that referenced
this issue
Nov 8, 2022
Test suite maintenance. re: issue #81
douglatornell
added a commit
that referenced
this issue
Nov 8, 2022
Test suite maintenance. re: issue #81
douglatornell
added a commit
that referenced
this issue
Feb 26, 2023
douglatornell
added a commit
that referenced
this issue
Mar 15, 2023
* Chg weather config to 2.5km HRDPS continental product ECCC stopped producing 2.5km HRDPS west product on 22-Feb-2023. It was part of the polar stereographic grid product suite that they replaced with the rotated lat-lon grid product, but that product is only available for the continental domain. * Add --run-date option to download_weather worker Replaces --yesterday option. ECCC hpfx server holds ~60 days of archived model products. So, we now have the option of downloading for dates further into the past than just the previous day. * Update worker failure docs * Fix download_weather worker description unit test * Drop logger patches in download_worker tests re: issue #82 * Chg to monkeypatch os.stat() in download_worker tests * Update NowcastWorker mocks in grib_to_netcdf tests re: issue #81 * Update logging mocks in grib_to_netcdf unit tests re: issue #82 * Drop unused monkeypacth args in download_weather tests * Refactor _define_forecast_segments_nowcast() Includes part of work required for issue #23 * Refactor _define_forecast_segments_forecast2() Includes part of work required for issue #23 * Add pywgrib2_xr package as dependency Use SalishSeaCast branch in SalishSeaCast/pywgrib2_xr fork to work under Python 3.10 and xarray>=2022.11. Also requires installation of libwgrib2 from SalishSeaCast branch in SalishSeaCast/libwgrib2 fork. * Refactor _rotate_grib_wind() Includes part of work required for issues #23 and #43 * Improve GHA pytest-with-coverage workflow comments Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr fork. * Install vendored libwgrib2 in GHA pytest-with-coverage Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr fork. * Fix pywgrib2_xr clone path in GHA pytest-with-coverage * Fix pywgrib2_xr clone syntax in GHA pytest-with-coverage * Revert premature config test changes * Drop GRIB2 file existence & 0 length checks Artifact of early days of SalishSeaNowcast development before automation included assurance that * Factor out _wgrib2_append() function Handles finicky details of pywgrib2_xr.wgrib(): * all pywgrib2_xr.wgrib() args must be strings * files must be freed after use to close them * Refactor _collect_grib_scalars() Includes part of work required for issues #23 and #43 * Refactor _concat_hourly_gribs() Includes part of work required for issue #23. * Drop TCDC_Sfc variable from weather collection Total cloud in percent that was used for parametrization of radiation missing from 2007-2014 GEMLAM. No longer needed. * Refactor _crop_to_watersheds() Includes factoring out _wgrib2_crop() function to handles finicky details of pywgrib2_xr.wgrib(): * all pywgrib2_xr.wgrib() args must be strings * files must be freed after use to close them Also includes part of work required for issue #23. * Temporarily remove launch of upload_forcing worker Need this so that weather collection, etc. can run in automation before grib_to_netcdf rework is ready for use. * Change sarracenia to hrdps-continental from hpfx * Revert addition of pywgrib2_xr pkg as dependency Using forks of pywgrib2_xr and libwgrib2 was yucky, but ultimately, the inability of wgrib2 to export the rotated lat-lon grid to netCDF was the death knell. * Add cfgrib package as dependency * Drop pygrib2_xr import & unit tests * Move HRDPS subdomain lon/lat index ranges to config * Drop wgrib2-style grid description from config * Ignore private GitLab FVCOM41 repo in docs linkcheck * Add _calc_grib_file_paths() * Add _calc_nemo_var_ds() & _trim_grib() * Change 2.5km weather variable names to triplets We need the MSC file variable name, the GRIB standard variable name, and the variable name to use in the NEMO forcing file for processing of the HRDPS continental GRIB files. * Chg forcing filename template from ops_ to hrdps_ So that we can clearly distinguish from HRDPS continental grid in contrast to those from HRDPS west for GEMLAM experimental grids. * Prototype code for hrdps_yYYmMMdDD.nc file creation * Set logging level for cfgrib library to WARNING DEBUG is too noisy. * Add _apportion_accumulation_vars() Add list of 2.5km forecast accumulation variables to weather config. Also fix bugs in grib files date and hours selection. * Factor out _update_checklist() * Add debug level logging messages * Clean up apportioning of accumulated variables * Improve lon/lat metadata * Add _calc_grid_angle() * Fix missing deg2rad() bug in _calc_grid_angle() * Add _calc_earth_ref_winds() * Chg longitude from grib's -180-180 to nemo's 0-360 * Improve trimmed dataset in _calc_earth_ref_winds() * Divide apportioned variables by 60*60 s/hr * Drop unneeded coordinate variables "time", "longitude", and "latitude" variables from GRIB files are replaced by "time_counter", "nav_lon", and "nav_lat" in NEMO datasets, so they are unneeded. * Add _improve_metadata() * Move setting dataset history attr to _write_netcdf() Necessary so that we can use xarray.combine_by_coords() to build 3-section run-date dataset. * Refactor NEMO dataset prep into _calc_nemo_ds() * Make timezone mock more robust * Handle 2 cases for offset in _apportion_accumulation_vars() * Calculate full 24hr nowcast date forcing file * Add calc of 24hr day 1 forcing file for nowcast+ * Add calc of 12hr day 2 forcing file for nowcast+ * Chg to fcst=False default in _update_checklist() * Add calc of 24hr day 1 forcing file for forecast2 * Add calc of 6hr day 2 forcing file for forecast2 * Delete old code that has been replaced * Drop wgrib2 logging configuration * Drop wgrib2 setup from deployment docs * Add logging port config for grib_to_netcdf on salish * Correct swapped lon/lat indices for grid cropping * Use persistent dask cluster on salish for processing Reduces grib_to_netcdf memory footprint and processing time a little compared to using dask's default threads scheduler. * Chg _write_netcdf() file created log msg to info level Also improve test coverage of _write_netcdf().
douglatornell
added a commit
that referenced
this issue
Feb 29, 2024
… v3 (#234) * Modernize test_get_onc_ctd Replace unittest.mock.patch decorator with pytest.fixture for mock worker. Add unit tests for production YAML config file elements related to worker; re: issue #117. Replace unittest.mock.patch decorator with pytest caplog fixture for tests of logging; re: issue #82. * Add unit test for YAML config file elements Added unit test for production YAML configuration file ctd data observations elements used in after_* () functions. * Improve get_onc_ctd.main() function Update docstring re: issue #121. Return worker so that modernized unit tests for main() work; re issue #81. * Update get_onc_ctd worker to use ONC API v3 Modified the get_onc_ctd worker to use ONC API v3 'getByLocation' method. The new method uses a location code instead of a station name. Also, these changes include adding a 'dateTo' parameter to limit the data to a specific time range for the day. Additionally, small changes to variable and parameter names were made to match the new method requirements. * Remove test skipping due to resolved issue #174 The pytest skip marker for "_resample_nav_coord()" function in "tests/workers/test_get_onc_ferry.py" file has been removed. This has happened following the successful resolution of issue number #174, making it unnecessary to skip these specific unit tests anymore. Fixes issue #174 * Add unit tests for YAML config file elements Add unit tests for production YAML config file elements related to worker; re: issue #117. * Update YAML config with Tsawwassen - Duke Point ferry route The 'nowcast.yaml' configuration file has been updated to include the Tsawwassen - Duke Point ferry route details including devices, sensors, and other relevant data. * Update dataset ID for TWDP-ferry The dataset ID for TWDP-ferry was updated in the 'nowcast.yaml' configuration file and in respective test. The change is reflected accurately to match the new details for the Tsawwassen - Duke Point ferry route. * Update get_onc_ferry worker to use ONC API v3 Modified the get_onc_ferry worker to use ONC API v3 'getByLocation' method. The new method uses a location code instead of a station name. Also, these changes include adding a 'dateTo' parameter to limit the data to a specific time range for the day. Server-side averaging into 1-second bins is used to ensure that an entire day's observations from each sensor can be obtained by a single API request; there is a limit of 100,000 "rows" per sensor. Additionally, small changes to variable and parameter names were made to match the new method requirements. * Update ONC_data_product_url attr in ferry datasets Updated the ONC_data_product_url attribute in the datasets produced by the get_onc_ferry worker to reflect changes in the data source's domain name and API query parameter names. Code changes include updating the ONC data API domain name, adding the 'locationCode' query parameter, and changing the 'deviceCategory' query parameter name to 'deviceCategoryCode'. All of those changes are for compatibility with the ONC data API v3. * Correct representation of relative humidity attribute Changed the naming representation of the relative humidity attribute in the 'get_onc_ferry' worker. The attribute was initially named "REL_HUMIDITY" and it was renamed to "rel_humidity", matching the common low-caps format of the other attributes for consistency of naming convention. * Update QA/QC filter criteria in get_onc_ferry worker Adjust the QA/QC filter settings in the 'get_onc_ferry' worker to now include data where the 'qaqcFlag' attribute is less than or equal to 1 or greater than or equal to 7. This update is necessary because of the change to server-side averaging in the request. * Improve empty data array handling ONC API v3 has more/different ways of returning empty responses to data requests that we have to handle. It's not as clean as I would like, but it is good enough for the purpose.
douglatornell
added a commit
that referenced
this issue
Mar 29, 2024
Test suite maintenance. re: issue #81
douglatornell
added a commit
that referenced
this issue
Mar 30, 2024
* Change NowcastWorker mock to pytest fixture Test suite maintenance. re: issue #81 * Update make_plots worker main() function docstring Removed not informative "Set up and run the worker." line at the beginning. re: issue #121 * Change logging mocks to pytest caplog fixture Replace unittest.mock.patch decorator with pytest caplog fixture for tests of logging. Test suite maintenance re: issue #82. * Add unit tests for YAML config file elements Add unit tests for production YAML config file elements related to worker; re: issue #117. * Remove unused ferry_data_dir configuration retrieval The ferry_data_dir configuration was retrieved but not used in the make_plots.py worker script. This change removes the unneeded line to tidy up the code and avoid potential confusion in the future. * Update to V21-11 dataset URLs in config & tests The URLs for '3d tracer fields' and '3d biology fields' have been updated in the test_make_plots.py and nowcast.yaml files. * Rename physics dataset keys to '3d physics fields' This commit changes the '3d tracer fields' key in the nowcast.yaml configuration file, the make_plots.py worker and test_make_plots.py tests to '3d physics fields'. This change better reflects in the relevant data source URLs. * Update zooplankton field var names in make_plots Update zooplankton field variable names for the time series plot function in the make_plots worker. Specifically, change "mesozooplankton" and "microzooplankton" to "z1_zooplankton" and "z2_zooplankton". This update ensures consistency with the V21-11 model output variable names. * Replace Mesodinium rubrum w/ Diatoms in time series plots In the 'make_plots' worker, the field variable 'mesodinium' was changed to 'diatoms' for the time series plots. The 'diatoms_flagellates_timeseries' dictionary key is adjusted accordingly. This is necessary due to the removal of the Mesodinium rubrum variable from the V21-11 model calculations and output. * Add z1 & z2 zooplankton to color dict in website_theme Two new types of zooplankton, 'z1_zooplankton' and 'z2_zooplankton', have been added to the color dictionary of nowcast/figures/website_theme.py file. This change would allow the correct color to be displayed for these new types in the corresponding plots.
douglatornell
added a commit
that referenced
this issue
May 17, 2024
Test suite maintenance. re: issue #81
douglatornell
added a commit
that referenced
this issue
May 19, 2024
* Update make_plots worker main() function docstring Removed not informative "Set up and run the worker." line at the beginning. re: issue #121 * Refactor to use textwrap for config file writing This commit introduces the use of `textwrap.dedent` to maintain the configuration file's content in the tests for the surface current tiles worker module. Using `textwrap.dedent` makes the inline configuration text more manageable and readable by removing the initial indentations. * Change NowcastWorker mock to pytest fixture Test suite maintenance. re: issue #81 * Change logging mocks to pytest caplog fixture Replace unittest.mock.patch decorator with pytest caplog fixture for tests of logging. Test suite maintenance re: issue #82. * Replace `PyPDF2` package with `pypdf` Updated the project dependencies and scripts to use the `pypdf` package instead of `PyPDF2`. Also, adapted the code in `make_surface_current_tiles.py` to use `pypdf`'s `PdfWriter` instead of `PyPDF2`'s `PdfFileMerger`. This resolves the Jun-2023 security alert re: CVE-2023-36464 infinite loop vulnerability and deprecation of PyPDF2.
douglatornell
added a commit
that referenced
this issue
Jul 16, 2024
Test suite maintenance. re: issue #81
douglatornell
added a commit
that referenced
this issue
Jul 16, 2024
* Add get_vfpa_hadcp after 06 weather collection Collect the previous day's observations from the VFPA HADCP located at the 2nd Narrows railway bridge early in the morning after the 06Z weather forecast products have been downloaded. This restores daily collection of the HADCP obs that were inadvertently stopped when we stopped running the VHFR FVCOM model in Mar-2023. * Update get_vfpa_hadcp main() function docstring Remove non-informative "Set up and run the worker." line at the beginning. re: issue #121 * Change NowcastWorker mock to pytest fixture Test suite maintenance. re: issue #81 * Change logging mocks to pytest caplog fixture Replace unittest.mock.patch decorator with pytest caplog fixture for tests of logging. Test suite maintenance re: issue #82. * Modernize unit tests for get_vfpa_hadcp worker Modified the unit tests in test_get_vfpa_hadcp.py to use pytest fixtures and monkeypatching for more accurate and isolated tests. This includes new mocks for make_hour_dataset and write_netcdf, as well as changes in logging and managing file paths. Particular attention given was given to ensuring accurate capture of log messages.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Replace
@patch("nowcast.workers.*.NowcastWorker", spec=True)withmock_nowcast_worker()fixture and refactorTestMainclasses in the following unit test modules:test_download_fvcom_results.pytest_download_wwatch3_results.pytest_get_onc_ctd.pytest_get_vfpa_hadcp.pytest_grib_to_netcdf.pytest_launch_remote_worker.pytest_make_CHS_currents_file.pytest_make_feeds.pytest_make_fvcom_atmos_forcing.pytest_make_fvcom_boundary.pytest_make_fvcom_rivers_forcing.pytest_make_live_ocean_files.pytest_make_plots.pytest_make_runoff_file.pytest_make_surface_current_tiles.pytest_make_turbidity_file.pytest_make_ww3_current_file.pytest_make_ww3_wind_file.pytest_run_fvcom.pytest_run_NEMO_agrif.pytest_run_NEMO_hindcast.pytest_run_NEMO.pytest_update_forecast_datasets.pytest_upload_fvcom_atmos_forcing.pytest_watch_fvcom.pytest_watch_NEMO_agrif.pytest_watch_NEMO_hindcast.pytest_watch_NEMO.pytest_watch_ww3.pyThe text was updated successfully, but these errors were encountered: