Refactor GRIB and netCDF variables names from grib_to_netcdf to config file #43
Labels
Milestone
Comments
|
Original changes by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell). changed state from "new" to "open" |
douglatornell
added a commit
that referenced
this issue
Mar 15, 2023
* Chg weather config to 2.5km HRDPS continental product ECCC stopped producing 2.5km HRDPS west product on 22-Feb-2023. It was part of the polar stereographic grid product suite that they replaced with the rotated lat-lon grid product, but that product is only available for the continental domain. * Add --run-date option to download_weather worker Replaces --yesterday option. ECCC hpfx server holds ~60 days of archived model products. So, we now have the option of downloading for dates further into the past than just the previous day. * Update worker failure docs * Fix download_weather worker description unit test * Drop logger patches in download_worker tests re: issue #82 * Chg to monkeypatch os.stat() in download_worker tests * Update NowcastWorker mocks in grib_to_netcdf tests re: issue #81 * Update logging mocks in grib_to_netcdf unit tests re: issue #82 * Drop unused monkeypacth args in download_weather tests * Refactor _define_forecast_segments_nowcast() Includes part of work required for issue #23 * Refactor _define_forecast_segments_forecast2() Includes part of work required for issue #23 * Add pywgrib2_xr package as dependency Use SalishSeaCast branch in SalishSeaCast/pywgrib2_xr fork to work under Python 3.10 and xarray>=2022.11. Also requires installation of libwgrib2 from SalishSeaCast branch in SalishSeaCast/libwgrib2 fork. * Refactor _rotate_grib_wind() Includes part of work required for issues #23 and #43 * Improve GHA pytest-with-coverage workflow comments Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr fork. * Install vendored libwgrib2 in GHA pytest-with-coverage Mostly to trigger another workflow run with tweaked setup.py in the pywgrib2_xr fork. * Fix pywgrib2_xr clone path in GHA pytest-with-coverage * Fix pywgrib2_xr clone syntax in GHA pytest-with-coverage * Revert premature config test changes * Drop GRIB2 file existence & 0 length checks Artifact of early days of SalishSeaNowcast development before automation included assurance that * Factor out _wgrib2_append() function Handles finicky details of pywgrib2_xr.wgrib(): * all pywgrib2_xr.wgrib() args must be strings * files must be freed after use to close them * Refactor _collect_grib_scalars() Includes part of work required for issues #23 and #43 * Refactor _concat_hourly_gribs() Includes part of work required for issue #23. * Drop TCDC_Sfc variable from weather collection Total cloud in percent that was used for parametrization of radiation missing from 2007-2014 GEMLAM. No longer needed. * Refactor _crop_to_watersheds() Includes factoring out _wgrib2_crop() function to handles finicky details of pywgrib2_xr.wgrib(): * all pywgrib2_xr.wgrib() args must be strings * files must be freed after use to close them Also includes part of work required for issue #23. * Temporarily remove launch of upload_forcing worker Need this so that weather collection, etc. can run in automation before grib_to_netcdf rework is ready for use. * Change sarracenia to hrdps-continental from hpfx * Revert addition of pywgrib2_xr pkg as dependency Using forks of pywgrib2_xr and libwgrib2 was yucky, but ultimately, the inability of wgrib2 to export the rotated lat-lon grid to netCDF was the death knell. * Add cfgrib package as dependency * Drop pygrib2_xr import & unit tests * Move HRDPS subdomain lon/lat index ranges to config * Drop wgrib2-style grid description from config * Ignore private GitLab FVCOM41 repo in docs linkcheck * Add _calc_grib_file_paths() * Add _calc_nemo_var_ds() & _trim_grib() * Change 2.5km weather variable names to triplets We need the MSC file variable name, the GRIB standard variable name, and the variable name to use in the NEMO forcing file for processing of the HRDPS continental GRIB files. * Chg forcing filename template from ops_ to hrdps_ So that we can clearly distinguish from HRDPS continental grid in contrast to those from HRDPS west for GEMLAM experimental grids. * Prototype code for hrdps_yYYmMMdDD.nc file creation * Set logging level for cfgrib library to WARNING DEBUG is too noisy. * Add _apportion_accumulation_vars() Add list of 2.5km forecast accumulation variables to weather config. Also fix bugs in grib files date and hours selection. * Factor out _update_checklist() * Add debug level logging messages * Clean up apportioning of accumulated variables * Improve lon/lat metadata * Add _calc_grid_angle() * Fix missing deg2rad() bug in _calc_grid_angle() * Add _calc_earth_ref_winds() * Chg longitude from grib's -180-180 to nemo's 0-360 * Improve trimmed dataset in _calc_earth_ref_winds() * Divide apportioned variables by 60*60 s/hr * Drop unneeded coordinate variables "time", "longitude", and "latitude" variables from GRIB files are replaced by "time_counter", "nav_lon", and "nav_lat" in NEMO datasets, so they are unneeded. * Add _improve_metadata() * Move setting dataset history attr to _write_netcdf() Necessary so that we can use xarray.combine_by_coords() to build 3-section run-date dataset. * Refactor NEMO dataset prep into _calc_nemo_ds() * Make timezone mock more robust * Handle 2 cases for offset in _apportion_accumulation_vars() * Calculate full 24hr nowcast date forcing file * Add calc of 24hr day 1 forcing file for nowcast+ * Add calc of 12hr day 2 forcing file for nowcast+ * Chg to fcst=False default in _update_checklist() * Add calc of 24hr day 1 forcing file for forecast2 * Add calc of 6hr day 2 forcing file for forecast2 * Delete old code that has been replaced * Drop wgrib2 logging configuration * Drop wgrib2 setup from deployment docs * Add logging port config for grib_to_netcdf on salish * Correct swapped lon/lat indices for grid cropping * Use persistent dask cluster on salish for processing Reduces grib_to_netcdf memory footprint and processing time a little compared to using dask's default threads scheduler. * Chg _write_netcdf() file created log msg to info level Also improve test coverage of _write_netcdf().
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Original report by Doug Latornell (Bitbucket: douglatornell, GitHub: douglatornell).
There are 3 elements in the mapping:
grib_to_netcdfwill have to be refactored for sure. If anything other than 1 is chosen as the key,download_weatherwill also have to be refactored.The text was updated successfully, but these errors were encountered: