[Aug/ Sep] [Pubs] Curation Workflow Tracking

[aditigopalan]

This ticket tracks curation workflow progression.

Note: It is possible for work to take place simultaneously in the three sections with overlapping periods, allowing curation workflows for different months to coincide.

1. Curation and Annotation

• Run Pubmed crawler to generate PublicationView manifest [205 publications generated, long sprint anticipated]
• Send Amber and Jineta a copy of the PublicationView manifest from latest crawl to review for MC2 Center Newsletter publication highlights
• Send Amber "News from CCKP" for MC2 Center Newsletter
• Annotate publications in PublicationView manifest [In progress]
• Generate ToolView and DatasetView manifests based on PublicationView manifest
• Run the automated curation workflow to upload publications, datasets and tools [This includes splitting manifests, processing and validating manifests, generating target synapse IDs for upload, schema updates, upload to synapse and (in progress) a validation check for uploads)
• Generate UNION tables
• QC of staging tables
• Performing automate portal sync to CCKP
• Validate data on the CCKP

Status check [Plan to report numbers for each category following pubmed crawl]:

• Publication upload [ ]

No data model updates

[aclayton555]

24-9 - all have been annotated, up to the Sept crawl. Action needed is to push sync.

Orion has run union/QC checks. Have found a couple of new values that will necessitate a small data model release, but this can be decoupled from pushing the sync (should be done by tomorrow). Sync will also include some of the datasets (e.g. light sheet)