Demultiplexing genetic-pooled samples fails when VCF is provided

[mhulke]

I am currently using the demultiplexing workflow in Terra (snapshot 32) to demultilpex genetic-pooled samples. I previously used this workflow on these samples by running souporcell without a VCF files and samples were separated without any issue. Now that I have a VCF for these samples, the pipeline crashes after a few errors. The error log is attached below. It first seems to have issues with the bam file not have an associated bai file, which is located in the same directory and follows the expected naming convention. It then has issues with the VCF file not being in the expected format. The workflow finally crashes with a nondescript error related to doublet calling. Do you have any idea what might be going wrong?

I also tried using this workflow with the demuxlet package in popsicle. That seems to run config successfully and then stops without any error message. All I can find in the workflow logs is this:

**
2022-03-17 13:12:23,530 INFO - MaterializeWorkflowDescriptorActor [UUID(f28d06c9)]: Parsing workflow as WDL 1.0
2022-03-17 13:12:23,939 INFO - MaterializeWorkflowDescriptorActor [UUID(f28d06c9)]: Call-to-Backend assignments: demuxEM.run_demuxEM -> PAPIv2-CloudNAT, souporcell.run_souporcell -> PAPIv2-CloudNAT, popscle.popscle_task -> PAPIv2-CloudNAT, souporcell.match_donors -> PAPIv2-CloudNAT, demultiplexing.Config -> PAPIv2-CloudNAT
2022-03-17 13:12:26,178 INFO - WorkflowExecutionActor-f28d06c9-2e02-44d8-8c54-f7d7a428b6bc [UUID(f28d06c9)]: Starting demultiplexing.Config
2022-03-17 13:12:38,241 INFO - f28d06c9-2e02-44d8-8c54-f7d7a428b6bc-EngineJobExecutionActor-demultiplexing.Config:NA:1 [UUID(f28d06c9)]: Failure copying cache results for job BackendJobDescriptorKey_CommandCallNode_demultiplexing.Config: -1:1 (EnhancedCromwellIoException: [Attempted 1 time(s)] - StorageException: No such object: fc-6e5b0ad8-754f-4fec-8776-e4295f797fd9/f420e11b-db49-4bb3-8131-e6399780c2e4/demultiplexing/4cbec5e5-e7ee-4461-aff9-cc0476d6a598/call-Config/stderr)
2022-03-17 13:12:39,962 INFO - f28d06c9-2e02-44d8-8c54-f7d7a428b6bc-EngineJobExecutionActor-demultiplexing.Config:NA:1 [UUID(f28d06c9)]: Failure copying cache results for job BackendJobDescriptorKey_CommandCallNode_demultiplexing.Config: -1:1 (EnhancedCromwellIoException: [Attempted 1 time(s)] - StorageException: No such object: fc-6e5b0ad8-754f-4fec-8776-e4295f797fd9/204cbe8a-2f9d-4f48-8b21-af8ae1cc6e29/demultiplexing/ed54466e-bb88-42a4-ba93-39733d584774/call-Config/rc)(this job has already failed to copy from another 1 other hits, of which 1 were copy failures and 0 were already blacklisted)
2022-03-17 13:12:42,963 INFO - f28d06c9-2e02-44d8-8c54-f7d7a428b6bc-EngineJobExecutionActor-demultiplexing.Config: NA:1 [UUID(f28d06c9)]: Failure copying cache results for job BackendJobDescriptorKey_CommandCallNode_demultiplexing.Config: -1:1 (EnhancedCromwellIoException: [Attempted 1 time(s)] - StorageException: No such object: fc-6e5b0ad8-754f-4fec-8776-e4295f797fd9/5b142df4-b61a-4631-b03b-2d2572278562/demultiplexing/865d8766-9b29-4486-bbd8-e41a08fd6087/call-Config/cacheCopy/stdout)(this job has already failed to copy from another 2 other hits, of which 2 were copy failures and 0 were already blacklisted)
2022-03-17 13:12:43,120 INFO - f28d06c9-2e02-44d8-8c54-f7d7a428b6bc-EngineJobExecutionActor-demultiplexing.Config:NA:1 [UUID(f28d06c9)]: Failure copying cache results for job BackendJobDescriptorKey_CommandCallNode_demultiplexing.Config: -1:1 (EnhancedCromwellIoException: [Attempted 1 time(s)] - StorageException: No such object: fc-6e5b0ad8-754f-4fec-8776-e4295f797fd9/e216a11c-4902-47a0-bb76-a686b1ff2bcf/demultiplexing/bdf00970-9bb4-4216-b614-20e453709b18/call-Config/cacheCopy/script)(this job has already failed to copy from another 3 other hits, of which 3 were copy failures and 0 were already blacklisted)
2022-03-17 13:12:48,325 INFO - f28d06c9-2e02-44d8-8c54-f7d7a428b6bc-EngineJobExecutionActor-demultiplexing.Config:NA:1 [UUID(f28d06c9)]: Call cache hit process had 4 total copy failures before completing successfully (of which, 4 were copy failures, 0 were already blacklisted)
2022-03-17 13:12:49,920 INFO - WorkflowExecutionActor-f28d06c9-2e02-44d8-8c54-f7d7a428b6bc [UUID(f28d06c9)]: Job results retrieved (CallCached): 'demultiplexing.Config' (scatter index: None, attempt 1)
2022-03-17 13:12:59,887 INFO - WorkflowExecutionActor-f28d06c9-2e02-44d8-8c54-f7d7a428b6bc [UUID(f28d06c9)]: Workflow demultiplexing complete. Final Outputs:
{
"demultiplexing.output_folders": [],
"demultiplexing.output_zarr_files": []
}
2022-03-17 13:12:59,904 INFO - $a [UUID(f28d06c9)]: Copying workflow logs from /cromwell-workflow-logs/workflow.f28d06c9-2e02-44d8-8c54-f7d7a428b6bc.log to gs://fc-6e5b0ad8-754f-4fec-8776-e4295f797fd9/13d650cb-9891-49e1-97fd-81d535878b7c/workflow.logs/workflow.f28d06c9-2e02-44d8-8c54-f7d7a428b6bc.log
**

Thank you for any help you can offer

souporcell_error.txt