Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move batch loads to slow worker #1420

Open
aequitas opened this issue May 30, 2024 · 2 comments · May be fixed by #1426
Open

Move batch loads to slow worker #1420

aequitas opened this issue May 30, 2024 · 2 comments · May be fixed by #1426
Assignees

Comments

@aequitas
Copy link
Collaborator

aequitas commented May 30, 2024

To solve memory issues on batch installations:

  • Move batch_slow queue to slow worker
  • Make concurrency for slow worker configurable and reduce to 1 2
@baknu
Copy link
Contributor

baknu commented May 30, 2024

Two remarks:

  • Note that according to an older mail from GT (subject: "worker_slow configuration" / dated 3 December 2021), the concurrency for the slow worker that handled the batch_slow queue with the async_register and async_generate_results was 2.
  • We should also make sure that each of the workers gets sufficient max memory. The total available memory on each of the Docker based batch servers (dev-docker.batch and docker.batch) is currently 16 GB.

@baknu baknu assigned gthess and aequitas and unassigned gthess May 30, 2024
@aequitas aequitas linked a pull request Jun 4, 2024 that will close this issue
@aequitas
Copy link
Collaborator Author

aequitas commented Jun 4, 2024

PR #1426 moves batch_slow to slow worker and reduces concurrency on that worker.

Memory for both the normal and slow worker can be configured already in local.env by overriding the WORKER_MEMORY_LIMIT and WORKER_SLOW_MEMORY_LIMIT respectively.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging a pull request may close this issue.

3 participants