Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File upload appears to hold everything in memory until the end #3517

Open
bertbarabas opened this issue Jun 19, 2024 · 1 comment
Open

File upload appears to hold everything in memory until the end #3517

bertbarabas opened this issue Jun 19, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@bertbarabas
Copy link

Describe the bug

  1. For large files, memory upload consumes significant memory and it grows even more when writing out the file iobuffer.
  2. Upload is very slow and consumes significant CPU when everything is on the same host (both reflex and the browser). Uploading files shouldn't really consume much CPU at all...

To Reproduce
Steps to reproduce the behavior:

  • Just use the standard upload example and upload a multi-gigabyte file.

Expected behavior
Very little memory should be used because the data should be incrementally written to a file.
To avoid re-writing the file after upload, I'd expect to be able to specify the target or if you choose to write to a temporary file I'd expect to be able to rename the temporary file to it's final home.

Specifics (please complete the following information):

  • Python Version: 3.12.4
  • Reflex Version: 0.5.4
  • OS: WSL Ubuntu 22.04.4 LTS on Windows 11 pro
  • Browser (Optional): Brave 1.67.116 ( Chromium 126.0.6478.71)
@bertbarabas bertbarabas added the bug Something isn't working label Jun 19, 2024
@bertbarabas
Copy link
Author

Another odd behavior for upload is the on_upload_progress which claims the upload is 100% done but then spends another 10% of the time before returning from rx.upload_files and passing the result on to the handle_upload function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant