You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, all blocks need to be processed before passing onto the post-processing step (just stored into a file as of now, see #11).
This makes it unconvenient when working with large block ranges or with a lot of data that needs to be stored in memory.
Also, if an unexpected error occurs at the block processing step, all previously processed block data will be lost and nothing will be available in the final output.
Proposition
Allow for having batches of processed blocks (customizable) to be sent to the post-processing step.
The text was updated successfully, but these errors were encountered:
Problem
Currently, all blocks need to be processed before passing onto the post-processing step (just stored into a file as of now, see #11).
This makes it unconvenient when working with large block ranges or with a lot of data that needs to be stored in memory.
Also, if an unexpected error occurs at the block processing step, all previously processed block data will be lost and nothing will be available in the final output.
Proposition
Allow for having batches of processed blocks (customizable) to be sent to the post-processing step.
The text was updated successfully, but these errors were encountered: