O(n²) Memory Requirements for _stamp_parallel()

[JaKasb]

The function _stamp_parallel() computes the distance-profiles via a pool of workers.
Each worker returns a list of distance-profiles.

The reduction of the distance-matrix occurs after all workers return their lists-of-distance-profiles.
The memory requirements of all distance-profiles is O( (sampling_rate*n) * n )

reduce(map(mass_distance_profile_parallel(indices)))

matrixprofile-ts/matrixprofile/matrixProfile.py

Line 128 in 207aa94

def _stamp_parallel(tsA, m, tsB=None, sampling=0.2, n_threads=-1, random_state=None):

matrixprofile-ts/matrixprofile/distanceProfile.py

Line 74 in bcba7dc

def mass_distance_profile_parallel(indices, tsA=None, tsB=None, m=None):

To reduce the memory requirements, each worker must reduce the list-of-distances to an intermediate matrix-profile.
Afterwards the pool-spawner must reduce the intermediate matrix-profiles to the final matrix-profile.

reduce(map(reduce(mass_distance_profile_parallel(indices))))