Load balancing for multiprocessing

[tso-martin]

Dear @mkhraijah
I had a look into the function solve_dopf_mp and would like to know the design considerations behind it. It seems that the areas are attached to the workers at initialization. Depending on the size of the area the workload may be different among the workers. I wondered if it may be possible to use pmap to process a collection of areas. pmap offers load-balancing. Preliminary tests of mine, trying to repeatedly solve JuMP models on arbitrary workers failed however. Did you have similar experiences?

[mkhraijah]

Hi @tso-martin
You are right, we assign each area to a specific worker in the initialization function which may cause load unbalance. You might be able to use pmap to distribute the load, but that means you need to make the data available to all workers at all times or send the area data to the worker when needed (each iteration). This in turn might cause higher computation time due to transfering the data from/to the workers. That's why we assign the areas to workers and transfer all the area data to the assigned worker at the beginning of the algorithm and only share the results at the end of the iterations. Having said that, I think the workers in the Distributed package are not allocated to physical cores or processors. You might be able to get the best performance when you have the number of workers equal to the number of areas. I'm not sure if that is accurate, but this is the easiest way to get a faster computation.

[tso-martin]

Thank you very much for your insights, @mkhraijah .