Skip to content

using pathos pool in multipleruns

Lasse Bonnrequested to merge
pool into master

Save around 45 lines of multiprocessing by using pathos pp.pool.

pathos pool is the same as multiprocessing pool where there is a set number of processes. With the map function you assign a list of arguments to a function and the map gives one argument(-vector) to each process. If there is more arguments than the size of the pool, the program waits and when a worker has finished one process it can start on the next one. The result is a list of return from the function that was mapped onto the arguments.

I use pathos because normal python multiprocessing uses pickle for data management but pickle cannot deal with methods of classes. Pathos multiprocessing uses dill, which can serialise methods.

pros:

  • simpler and less code
  • no manual creating processes and joining etc

cons:

  • new dependency if you use multiprocessing

Merge request reports