python - How to apply a function to a 2D numpy array with multiprocessing -
suppose have following function:
def f(x,y): return x*y how apply funtion each element in nxm 2d numpy array using multiprocessing module? using serial iteration, code might follows:
import numpy np n = 10 m = 12 results = np.zeros(shape=(n,m)) x in range(n): y in range(m): results[x,y] = f(x,y)
here's how might parallelize example function using multiprocesssing. i've included identical pure python function uses non-parallel for loops, , numpy one-liner achieves same result:
import numpy np multiprocessing import pool def f(x,y): return x * y # helper function needed because map() can used functions # take single argument (see http://stackoverflow.com/q/5442910/1461210) def splat_f(args): return f(*args) # pool of 8 worker processes pool = pool(8) def parallel(m, n): results = pool.map(splat_f, ((i, j) in range(m) j in range(n))) return np.array(results).reshape(m, n) def nonparallel(m, n): out = np.zeros((m, n), np.int) in range(m): j in range(n): out[i, j] = f(i, j) return out def broadcast(m, n): return np.prod(np.ogrid[:m, :n]) now let's @ performance:
%timeit parallel(1000, 1000) # 1 loops, best of 3: 1.67 s per loop %timeit nonparallel(1000, 1000) # 1 loops, best of 3: 395 ms per loop %timeit broadcast(1000, 1000) # 100 loops, best of 3: 2 ms per loop the non-parallel pure python version beats parallelized version factor of 4, , version using numpy array broadcasting absolutely crushes other two.
the problem starting , stopping python subprocesses carries quite lot of overhead, , test function trivial each worker thread spends tiny proportion of lifetime doing useful work. multiprocessing makes sense if each thread has substantial amount of work before killed. might, example, give each worker bigger chunk of output array compute (try messing around chunksize= parameter pool.map()), such trivial example doubt you'll see big improvement.
i don't know actual code looks - maybe function big , expensive enough warrant using multiprocessing. however, bet there much better ways improve performance.
Comments
Post a Comment