resalloc-server container hits resource limits under heavy load

[siteshwar]

While performing a mass scan on 3rd July 2024, resalloc server container started failing with these exceptions:

ERROR 2024-07-03 14:52:14,011 PID:1:140040003295040(Manager) Exception raised in Manager().run()
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/resallocserver/main.py", line 140, in main
    Manager(sync).run()
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 1119, in run
    self._loop()
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 1092, in _loop
    pool.detect_closed_tickets(self.sync.ticket)
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 748, in detect_closed_tickets
    ReleaseWorker(event, self, resource_id).start()
  File "/usr/lib64/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 208, in run
    self.job()
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 274, in job
    out = run_command(self.pool.id, resource.id, resource_name, id_in_pool,
  File "/usr/lib/python3.9/site-packages/resallocserver/manager.py", line 71, in run_command
    return {'status': subprocess.call(command, env=env, shell=True,
  File "/usr/lib64/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.9/subprocess.py", line 1770, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
BlockingIOError: [Errno 11] Resource temporarily unavailable

This problem was worked around by restarting the container by killing it.

This issue tracks finding root cause and fix for these failures.

[siteshwar]

This issue seems to be caused by using threads which are not daemon, and not joining them when they finish. So, the stale threads reach maximum number of threads limit.