Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IBCDPE-587] adds bash script for syncing LENS reference files #198

Merged
merged 16 commits into from
Jul 7, 2023

Conversation

BWMac
Copy link
Contributor

@BWMac BWMac commented Jun 7, 2023

This PR adds a bash script that can be used for reproducibly sync-ing the LENS Nextflow Workflow reference files to our sage-igenomes S3 bucket. The files and their structure within the subdirectories can be viewed here.

I followed the example provided by Bruno as much as I could, but there are a couple of differences.

  1. There is no all-encompassing location where these reference files are stored. Instead, the LENS devs provide 3 bash scripts that are used to generate the reference files from different locations. These scripts are downloaded and executed in this script.
  2. The file structure is different in that there are both files and folders present in the prefix directories used in this script, so I decided against sync-ing with longer prefixes as Bruno did in the example for simplicity.

I considered not including prefixes and the sync loop at all because I am not 100% certain that the prefixes won't change in the future.

After a follow-up discussion with iAtlas stakeholders, we elected to look closer at the reference file-downloading bash scripts used to set up the reference directory. I have made some updates to these scripts and included them in this PR. before running these scripts, it is assumed that you have followed the setup instructions provided by the LENS team here and that the raft directory has been created.

Documentation of the other additional steps I needed in order to run these scripts is also a WIP.

NOTE: The generate_mouse_references.sh script is still a WIP, owing to the fact that the starfusion section needs to be updated. As a result, I have commented out the download_mouse_references.sh executing lines.

@BWMac BWMac changed the title adds bash script for syncing LENS reference files [IBCDPE-575] adds bash script for syncing LENS reference files Jun 7, 2023
@BWMac BWMac marked this pull request as ready for review June 7, 2023 17:08
@BWMac BWMac requested a review from a team as a code owner June 7, 2023 17:08
@BWMac
Copy link
Contributor Author

BWMac commented Jun 7, 2023

I made another small commit to this branch but the PR does not seem to be updating for some reason...

bin/mirror-lens-references.sh Outdated Show resolved Hide resolved
bin/mirror-lens-references.sh Outdated Show resolved Hide resolved
bin/mirror-lens-references.sh Outdated Show resolved Hide resolved
@BWMac BWMac marked this pull request as draft July 5, 2023 19:06
@BWMac BWMac changed the title [IBCDPE-575] adds bash script for syncing LENS reference files [IBCDPE-587] adds bash script for syncing LENS reference files Jul 5, 2023
@BWMac BWMac marked this pull request as ready for review July 6, 2023 20:29
@BWMac BWMac requested a review from thomasyu888 July 6, 2023 20:32
Copy link
Collaborator

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is a draft, but some prelim comments.

bin/lens/download_general_references.sh Show resolved Hide resolved
bin/lens/download_human_references.sh Outdated Show resolved Hide resolved
bin/lens/download_human_references.sh Show resolved Hide resolved
Copy link
Collaborator

@thomasyu888 thomasyu888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ this looks great! Thanks!

@BWMac BWMac merged commit 50917ec into dev Jul 7, 2023
9 checks passed
@thomasyu888 thomasyu888 deleted the bwmac/IBCDPE-575/lens_ref_files branch October 3, 2023 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants