v3.0.0

This release includes updates to be used for the IMAP SDC Software Integration Test 3 (SIT-3). More information can be found in this galaxy page.

Features

• Added tests for backup S3 buckets
• Added RDS database schemas and tables
• Now using shared resources in a common environment/account
• Added support for ancillary data access from EFS
• Added cloudwatch monitoring
• Added API endpoint for the universal spin table
• Various updates to /upload, /download, and /query endpoints
• Added documentation on how to use personal AWS accounts for deployment
• Removed use of OpenSearch and StepFunctions
• Replaced processing step stack with Batch job stack
• Added batch job event handler to indexer lambda
• Added pre-processing dependency table
• Added I-ALiRT architecture for IOIS
• Standardized logging across the package
• Now using imap-data-access for file validation
• Added SNS Notification
• Added ECS to I-ALiRT Architecture
  Enabled SPICE file uploads to the SDC
• Added dependencies for all instruments
• Removed char limit on descriptor
• Automated Docker processing deployments

What's Changed

• Remove API Wrapper Scripts by @sdhoyt in #126
• Backup bucket tests by @maxinelasp in #146
• SIT-2 Release [dev] by @bourque in #151
• First pass at processing architecture by @laspsandoval in #154
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #155
• Starting point for RDS by @laspsandoval in #180
• Share resources in a common environment/account and auto deploy resources by @greglucas in #181
• CI: Update codecov configuration by @greglucas in #189
• CI: Use poetry run when deploying resources by @greglucas in #190
• Ancillary data access from EFS by @tech3371 in #188
• Remove old processing pipeline from SIT1 by @tech3371 in #195
• ENH: Add cloudwatch monitoring by @greglucas in #197
• Added endpoint to the API for universal spin table by @tech3371 in #199
• RDS with Schema Creation Custom Resource by @sdhoyt in #196
• CI: Remove version pin on npm cdk version installation by @greglucas in #201
• Processing Trigger Mechanisms by @laspsandoval in #191
• fix to upload folder path by @tech3371 in #198
• Setup Personal AWS Deploy Documentation by @sdhoyt in #200
• Added some more info to personal AWS account docs by @bourque in #205
• Upload config remove by @tech3371 in #202
• Create spin table by @tech3371 in #204
• ENH: Add redirect response from download url by @greglucas in #206
• Remove OpenSearch by @sdhoyt in #207
• Metadata Schema by @sdhoyt in #209
• update to batch env by @laspsandoval in #212
• Metadata Ingest by @sdhoyt in #210
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #211
• Remove Step Functions by @laspsandoval in #214
• Reorganize sds manager stack by @tech3371 in #215
• Replace processing step stack with Batch job stack by @tech3371 in #216
• Indexer events by @tech3371 in #217
• Api dockerfile for testing by @laspsandoval in #219
• InstrumentLambda from Construct to Stack by @laspsandoval in #220
• Update to indexer lambda to make pytest work locally by @tech3371 in #224
• Update File Catalog and Add Status Table by @sdhoyt in #223
• Query RDS with API by @sdhoyt in #228
• Added pre-processing dependency table by @tech3371 in #231
• I-ALiRT: First pass at architecture for IOIS by @laspsandoval in #208
• Batch starter by @laspsandoval in #230
• Indexer events by @tech3371 in #229
• added batch job event handler to indexer lambda by @tech3371 in #233
• Add event rule to batch starter lambda by @tech3371 in #236
• Various updates to database schema by @bourque in #232
• Added custom PutEvent to Batch Starter Lambda by @tech3371 in #234
• Updates to Batch Job by @laspsandoval in #238
• setup to send event from indexer lambda. by @tech3371 in #237
• FIX: Change to path params instead of query params in APIs by @greglucas in #235
• Batch starter fix by @tech3371 in #239
• Rename s3 uri by @tech3371 in #240
• Return list of dicts in query API by @bourque in #241
• Added name for database extensions enum by @sdhoyt in #243
• Prevent database duplicates by @sdhoyt in #244
• Fixes from Integration tests by @tech3371 in #245
• FIX: Change download API responses by @greglucas in #248
• MNT: Standardize logging across the package by @greglucas in #247
• Indexer lambda update post demo by @tech3371 in #253
• Updated ruff by @sdhoyt in #256
• update to use imap-data-access file validator by @tech3371 in #255
• Batch starter updates by @tech3371 in #257
• updated --data_level to --data-level to follow best practices by @tech3371 in #259
• Update dependency from json to database by @tech3371 in #260
• Consistent ruff checks with imap_processing by @bourque in #258
• added few instruments to list by @tech3371 in #261
• Infrastructure changes to match filename convention updates by @tech3371 in #262
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #263
• Added SNS Notification by @laspsandoval in #266
• Add ECS to I-ALiRT Architecture by @laspsandoval in #265
• ENH: Enable SPICE file uploads to the SDC by @greglucas in https://githu...

v2.0.0

This release marks the successful completion of SIT-2. Further information about what was tested in SIT-2 can be found in this galaxy page and the SIT-2 GitHub Project Board.

Features

• Ability to deploy and test software in the SDC Staging environment.
• Software can be successfully promoted from the SDC Staging Environment to the SDC Data Production Environment.
• Delivery of data files to the SPDF for routine archival purposes.
  • Backing up both OpenSearch and the data bucket to the backup account.
• Use of the API data interfaces by both SDC personnel as well as instrument teams. APIs:
  • The /upload endpoint for uploading new data to SDC
  • The /download endpoint for downloading data from SDC
  • The /query endpoint for querying information about data and data location.
• Demonstrate the SDC's ability to trigger jobs based on data availability and launch necessary processing containers.
  • Implementing a data watcher (Database) that keeps track of processing status.
  • Establishing a processing pipeline that initiates processing tasks.

What's Changed

• Primarily adding more fine-grained permissions to the lambda by @bryan-harter in #30
• Adding a way to create unique stack names each deploy by @bryan-harter in #29
• Adding the upload API to the stack and adding the Upload API code by @bryan-harter in #33
• MNT: Style updates by @greglucas in #19
• Adding poetry to the project by @maxinelasp in #38
• TST: Add test for s3 bucket properties by @greglucas in #42
• MNT: Ignore deprecation warnings by @greglucas in #44
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #51
• 1 create a spot to store configjson in a centralized location on aws by @sdhoyt in #50
• Improve stack creation testing by @sdhoyt in #52
• Update issue templates to include spike by @maxinelasp in #58
• 24 mock opensearch for unit testing by @sdhoyt in #69
• PR template by @sdhoyt in #70
• moved pull request template one directory up by @sdhoyt in #78
• Code of Conduct by @bourque in #77
• Modularize CDK code by @laspsandoval in #101
• API Gateway by @laspsandoval in #102
• TST: Reorganizing tests and moving to pytest-style by @greglucas in #103
• Removing templates now that we have a project-wide template for issues by @maxinelasp in #106
• Documentation by @maxinelasp in #109
• Add pre-commit hooks by @bourque in #115
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #80
• Updated infrastructure tests by @laspsandoval in #118
• adding .rst to docs/cdk by @GFMoraga in #114
• MNT: Add more information in the app template file by @greglucas in #112
• Implementation of Track Data Availability by @tech3371 in #120
• Changing the Upload API to use a GET rather than a POST by @bryan-harter in #110
• Initial Implementation of Trigger jobs based on data availability by @tech3371 in #119
• Opensearch snapshot by @sdhoyt in #121
• S3 backups by @maxinelasp in #125
• Updating permissions for data bucket role - SIT2 fix by @maxinelasp in #128
• SIT-2 Release [main] by @bourque in #150

Full Changelog: v1.0.1...v2.0.0

v1.0.0

This is the first successful release of the SDS Data Management software. It was released released after SOC SDC System Integration Test #1.

What's Changed

• 2 have indexerpy store file metadata in the opensearch cluster by @sdhoyt in #11
• TST/CI: Add automated testing for pull requests by @greglucas in #18
• 7 create an api for downloading the data by @tech3371 in #20
• 4 create api for querying the data by @sdhoyt in #21

Full Changelog: https://github.com/IMAP-Science-Operations-Center/sds-data-manager/commits/1.0.0

v1.0.1

SIT1 Fixes

Only one defect was found in SOC SDC System Integration Test (SIT) 1. The defect was that we failed to block public access on the data bucket. It was fixed in this release.

What's Changed

• Post SIT1 fix by @tech3371 in #39

Full Changelog: 1.0.0...v1.0.1