Numpy 2 compatibility

[matbryan52]

Looks like we are OK for numpy 2 here, just hdbscan which is not yet compatible. Their new version 0.8.37 released a few days ago pins numpy < 2, so new installs will work. Older installs that are upgraded will likely crash with the ValueError for the dtype size change on the import of hdbscan.

See scikit-learn-contrib/hdbscan#642

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

[sk1p]

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

I didn't look in detail yet, but the description sounds good - we already depend on numba, so installation (and maintenance on their side) should be a lot less painful. The better performance is then just the cherry on top.

[matbryan52]

We might also want to checkout fast_hdbscan mentioned in that issue: https://github.com/TutteInstitute/fast_hdbscan

I didn't look in detail yet, but the description sounds good - we already depend on numba, so installation (and maintenance on their side) should be a lot less painful. The better performance is then just the cherry on top.

In the end not such a simple change, yet.

Looks like fast_hdbscan is not quite yet numpy2 compatible (use of np.bool8 at least). Dropping to older numpy does mean that tests pass with fast_hdbscan, which is a good sign, but the import time is huge (10+ seconds on ptycho), I think because they run a fit to warmup numba at import : https://github.com/TutteInstitute/fast_hdbscan/blob/main/fast_hdbscan/__init__.py

Might want to wait and see, for now.