Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved build_global_exposure/4 #9255

Merged
merged 2 commits into from
Nov 27, 2023
Merged

Improved build_global_exposure/4 #9255

merged 2 commits into from
Nov 27, 2023

Conversation

micheles
Copy link
Contributor

@micheles micheles commented Nov 27, 2023

Part of #9227. By using pandas directly on small files I get much faster:

Received 16291 * 472.46 KB {'tot': '7.34 GB'} in 209 seconds
<Monitor [michele], duration=294.58053159713745s, memory=13.18 GB>

| calc_56312, maxmem=28.8 GB | time_sec | memory_mb | counts |
|----------------------------+----------+-----------+--------|
| total exposure_by_geohash  | 1_667    | 26.5      | 3_365  |
| total gen_tasks            | 444.2    | 6.26842   | 13_001 |

| operation-duration  | counts | mean | stddev | min     | max   | slowfac |
|---------------------+--------+------+--------+---------+-------+---------|
| exposure_by_geohash | 75     | 22.2 | 23%    | 0.25935 | 31.2  | 1.40277 |
| gen_tasks           | 19     | 23.4 | 180%   | 0.33477 | 152.3 | 6.51578 |

@micheles micheles added this to the Engine 3.19.0 milestone Nov 27, 2023
@micheles micheles self-assigned this Nov 27, 2023
@micheles micheles merged commit f397e4b into master Nov 27, 2023
@micheles micheles deleted the expo3 branch November 27, 2023 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant