Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Python3-only reusable thread executor #214

Open
wants to merge 32 commits into
base: master
Choose a base branch
from

Conversation

pierreglaser
Copy link
Collaborator

Supersedes #189.

This is a Python3-only implementation of a reusable ThreadPoolExecutor. Making this Python3-only limits the maintainability burden of loky by not making us backport threading utilities for Python2.

In particular, this PR implements a new _ReusableThreadPoolExecutor class. The appropriate tests from the loky as well as the concurrent.futures.ThreadPoolExecutor test suite are ran against it.

As a result, most of the changes in this code are test-suite additions-refactoring:

  • adding all cross-backend (processes/threads) tests into new base test classes, while separating all process-specific tests (tests about serialization, deadlocks, process termination etc. ) into new [Theme]ProcessExecutorTest classes
  • backporting some threading-specific tests from concurrent.futures

There is still a bunch of style/renaming to do, but the major part of the work is done.

@pierreglaser
Copy link
Collaborator Author

Umm, so a bunch of features and bugfixes were added to the ThreadPoolExecutor over time (such as initiializer). Do we want to backport the current ThreadPoolExecutor of python3.8 to older python versions, or should we match the features of loky's ThreadPoolExecutor with cpython's ThreadPoolExecutor for each python version? The first option is going to require code backporting. The second option requires skipping some test in old python versions. Pinging @tomMoral

@pierreglaser
Copy link
Collaborator Author

After an IRL talk with @tomMoral, I'm going to not backport all features/bugfixes made to the ThreadPoolExecutor to loky. I'll add skipping directives to the tests specific to these bugfixes/features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant