-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High memory usage for Detector1Pipeline #8667
Comments
Thanks for the excellent detective work! Similar to #8668 I think one issue is a bug in the However one thing to note is the (sometimes very) large size of the reference files. To use one example, running the |
I have promising results. I am now pulling The upgrade to stpipe does reduce the overall memory usage. In many cases it brings it in line with expectations, where the Dark Current step represents the largest usage, for example in MIRI slit observations: However, in other cases, the usage is still incredibly high. The MIRI usage is now down to 40GB (from 80GB) which represents a significant improvement. However, the Jump step still represents the majority of the usage. Perhaps this is still in line with expectations, but I would expect closer to a 5-10X size in memory with respect to the input image, rather than a 30X usage. I've attached again a zip file of all of single run tests: |
I can confirm that the memory usage of the pipeline is insanely high in stage 1 at the moment. I am reducing NIRISS WFSS data right now and all my processes get killed (I run the pipeline in parallel with a bash script). I tried to run it over one single uncal file and the memory usage went up to 65 Gb (for one single uncal file!), which makes the parallelization impossible on my cluster. Actually, it makes impossible using the pipeline also on a normal machine. How should I tackle this? |
Are you pulling I agree that this makes it untenable on a "normal" machine, so hopefully there are still improvements in memory usage coming soon. |
Summary:
The Detector1Pipeline is excepionally high, nearly 2 orders of magnitude larger than the input image size, regardless of instrument/mode or input file size. I am using the JWST pipeline installed from HEAD and testing this on macOS and Linux.
Context:
I have been running on NIRISS IMAGE/WFSS data, which have input file sizes of ~100MB. I would frequently use multiprocessing on cluster environments and receive Out Of Memory job cancellations regularly on nodes with 250GB of RAM. To test what was going on, I ran the Stage1 Pipeline on a subset of my NIRISS data to examine the memory usage. I was surprised to find memory footprints well into 8GB, a nearly two orders of magnitude size difference compared to the input file size. I decided to try and test this more robustly, and see if this was true for all JWST modes, or just NIRISS. I have therefore done the following:
Investigation:
These also span quite a wide range of input sizes, from 172K to 1.4G.
ps
to track the memory usage of the the JWST pipeline over time with a granularity of 1s. By comparing against the JWST pipeline output, it is possible to track memory usage as a function of pipeline step and I can create the following summary graph:NIRISS-WFSS_short.pdf
The problem becomes even more severe when looking at large input files, for example this MIRI/IFU observation requires 80GB of RAM to run the Stage1Pipeline:
MIRI-IFU_short.pdf
Regardless of any mode tried, the memory requirement scales to nearly 2 orders of magnitude larger than the input model size.
After discussion with @jdavies-st, it appears that instictively this is quite a bit higher than one might expect give the operations conducted within the Stage 1 Pipeline. Are there are any strategies for memory footprint mitigation? I am running the JWST pipeline over thousands of files and this is my main limitation.
I have attached the resultant plots from all of my runs in a .zip archive: short.zip
Code
The text was updated successfully, but these errors were encountered: