Fix blocking async interfaces for v2 ingest framework #3315
+43
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Right now, in a lot of places, where something does not support async, we just call the sync version of it.
This is potentially dangerous and could cause us to have blocking code.
I propose we wrap all those calls in an executor so we don't get unexpected behavior.
A big place we're missing async is in all our file IO(we have a lot of it). We could use a package like aiofiles; however, I have a feeling there is just a lot of code that won't be coverable with async interfaces. If files are small, it might be okay.