No data in chunks despite gzip conversion

[Haidermanzer]

Hello,

I've been trying to use Nanodisco and have ran into the issue that many others have had where the calculate differences step gives the "no data in chunk" error. It seems like almost all of the others that have ran into this issue have resolved it by using the ONT tools functionality to convert the files to .gzip, and I tried that as well but am still running into the same error.

Do you have any alternative solutions to this problem?

[touala]

Hello @Haidermanzer,

Thank you for trying nanodisco. Do all chunks give this No data in chunk issue? How do native and WGA mappings looks like (e.g. in IGV for example)? You could have this issue when the reference do not match the datasets.

Best,

Alan

[Haidermanzer]

Hi Alan, Thanks for the quick response. Every single chunk for both datasets say “no data.” I’m new to all of this so I’m not sure what IGV is, but it’s saying that every chunk is mapping correctly in the same step. I've also been able to map my reads to the reference genome with no problem in other programs. Best, Haider S. Manzer University of Colorado - Anschutz Medical Campus Department of Immunology and Microbiology Doctoral Candidate - Doran Lab

…

________________________________ From: touala ***@***.***> Sent: Monday, January 16, 2023 2:01:27 AM To: fanglab/nanodisco ***@***.***> Cc: Manzer, Haider ***@***.***>; Mention ***@***.***> Subject: Re: [fanglab/nanodisco] No data in chunks despite gzip conversion (Issue #64) [External Email - Use Caution] Hello @Haidermanzer<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FHaidermanzer&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cf0e6aa24248f4174c50408daf7a040a0%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094564921103892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2F0hTVbkrxPtRtnDTlmlsXfwmYGGx%2BxK0LWMhvmBHLOQ%3D&reserved=0>, Thank you for trying nanodisco. Do all chunks give this No data in chunk issue? How do native and WGA mappings looks like (e.g. in IGV for example)? You could have this issue when the reference do not match the datasets. Best, Alan — Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffanglab%2Fnanodisco%2Fissues%2F64%23issuecomment-1383693575&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cf0e6aa24248f4174c50408daf7a040a0%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094564921103892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0lFqxMHhV4y0ziHIR3oQBYJHV0sxAFpfVcdc5Xr9%2Fiw%3D&reserved=0>, or unsubscribe<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA5DWUDVECBHNUBNF2QTAUY3WSUE6PANCNFSM6AAAAAAT4LZOSI&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cf0e6aa24248f4174c50408daf7a040a0%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094564921103892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tTM3Ojxl7wc5uJakAweTA%2FaW9jrWhKJfP9VhvaOQmvk%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[touala]

IGV is a mapping visualisation software. You can find the documentation and download link at this address: https://software.broadinstitute.org/software/igv/UserGuide. Basically, I wanted to make sure reads were properly mapped by nanodisco preprocess command (e.g. -o <path_output> will produced .bam at <path_output>/<sample_name>.sorted.bam).

Is your dataset public?

Alan

[Haidermanzer]

Sorry for the delayed response. I pulled the sorted.bam file and just looked at it Geneious Prime and the reads did map correctly. We’re pretty close to publishing so the dataset will be out soon, but if you have any extra time to help troubleshoot I could send you the files privately via a onedrive link if you let me know which email address to send it to. Best, Haider S. Manzer University of Colorado - Anschutz Medical Campus Department of Immunology and Microbiology Doctoral Candidate - Doran Lab

…

________________________________ From: touala ***@***.***> Sent: Monday, January 16, 2023 9:16:19 AM To: fanglab/nanodisco ***@***.***> Cc: Manzer, Haider ***@***.***>; Mention ***@***.***> Subject: Re: [fanglab/nanodisco] No data in chunks despite gzip conversion (Issue #64) [External Email - Use Caution] IGV is a mapping visualisation software. You can find the documentation and download link at this address: https://software.broadinstitute.org/software/igv/UserGuide<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsoftware.broadinstitute.org%2Fsoftware%2Figv%2FUserGuide&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cb48ab33e409c42b59ae208daf7dd0097%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094825810607627%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5XmvfKwSjnYkwiNtmwlS63W%2FO8vzD%2BaNx4Ft4%2BAQdHA%3D&reserved=0>. Basically, I wanted to make sure reads were properly mapped by nanodisco preprocess command (e.g. -o <path_output> will produced .bam at <path_output>/<sample_name>.sorted.bam). Is your dataset public? Alan — Reply to this email directly, view it on GitHub<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffanglab%2Fnanodisco%2Fissues%2F64%23issuecomment-1384272988&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cb48ab33e409c42b59ae208daf7dd0097%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094825810607627%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lAB8%2FprrG3WeGrzhTD%2Bu%2FDJcvZyYfABxGhhc5fPtllQ%3D&reserved=0>, or unsubscribe<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA5DWUDSZZQRV4OBNFZBCCUDWSVX5HANCNFSM6AAAAAAT4LZOSI&data=05%7C01%7Chaider.manzer%40cuanschutz.edu%7Cb48ab33e409c42b59ae208daf7dd0097%7C563337caa517421aaae01aa5b414fd7f%7C0%7C0%7C638094825810607627%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wHE%2Bm2ZwpxArTza8ubjJk%2FojH4dWEyxRmc2LsI66If4%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[touala]

Hi @Haidermanzer,

You can send me the files and I'll look ASAP. To diagnose the issue, I would need native and WGA fast5, and the reference you used.

Best,

Alan

[Haidermanzer]

Great! Which email address should I share the one drive link with?

[touala]

You can share them at: [email protected]

[touala]

Hi @Haidermanzer,

I got your data and I was able to run nanodisco. The main issue seems related to the .vbz vs .gzip compression. Please find the commands below:

# Download and unzip your dataset
# Using most recent, but compatible, guppy version i.e. 6.3.8
# Later version discontinued --fast5_out
guppy_basecaller --disable_pings --recursive -i Nativefast5_pass --fast5_out -s Nativefast5_basecalled -c dna_r9.4.1_450bps_sup.cfg --device cuda:0
guppy_basecaller --disable_pings --recursive -i WGAfast5_pass --fast5_out -s WGAfast5_basecalled -c dna_r9.4.1_450bps_sup.cfg --device cuda:0

compress_fast5 -t 20 --recursive -c gzip -i Nativefast5_basecalled -s Nativefast5_basecalled_gzip
compress_fast5 -t 20 --recursive -c gzip -i WGAfast5_basecalled -s WGAfast5_basecalled_gzip

# Using an older singularity version to avoid having to bind directories
conda create -n singularity_3.5.2 -c conda-forge singularity=3.5.2
conda activate singularity_3.5.2

run_nanodisco="singularity exec <path_to/nanodisco> nanodisco"
$run_nanodisco preprocess -p 20 -f Nativefast5_basecalled_gzip -s native -o results/preprocessed_gzip -r Reference.fasta
$run_nanodisco preprocess -p 20 -f WGAfast5_basecalled_gzip -s wga -o results/preprocessed_gzip -r Reference.fasta

$run_nanodisco difference -nj 2 -nc 1 -p 5 -f 281 -l 290 -i results/preprocessed_gzip -o results/difference -w wga -n native -r Reference.fasta
# Make sure to process more if not all the genome chunks for better performances
# $run_nanodisco chunk_info -r Reference.fasta

# Check output within R:
# summary(readRDS("results/difference/chunk.281.difference.rds"))

I'm not sure why it didn't work for you earlier. Was the compress_fast5 command done before or after basecalling? Maybe Guppy reintroduce the newer .vbz compression when outputting basecalled fast5.

Please let me know if this fixed your issue or if you have any other problem.

Best,

Alan

[Haidermanzer]

Hi Alan,

Great! The steps that you outlined look pretty similar to what I tried other than some version differences, but I'll try using the versions you used now. In the meantime, would it be possible for you to use the one drive to send the output from those steps back to me so I could try to troubleshoot which part our output deviates from yours?

[touala]

Ok. It could be version specific indeed. I've uploaded two current differences files: for a chunk and for the whole genome. Do you know if a motif is modified in this sample so I can do a sanity check? Feel free to continue the discussion by email if you don't want to talk specific here.

Alan

[ecpierce]

@touala just in case it makes a difference- which version of ont-fast5-api/compress_fast5 are you using?

[touala]

@ecpierce Unfortunately I didn't kept track of that and I've multiple versions available... Although the most likely is 4.0.0.

[BioRB]

I have same issie. when I do nanodisco difference I get no data for any chunk. I followed the mentioned procedure also trying to change the singularity version but notting. could you try a test on my data? thanks

[touala]

Hello @BioRB,

Sure, feel free to share a subset of data. I've posted my email above.

Alan

[BioRB]

ok , I just sent you the files in a google drive folder.

[BioRB]

Dear @touala , do you have news about the subset I've sent you? Did you give it a try?
thanks