numpy - Python multiprocessing (joblib) best way for argument passing -


i've noticed huge delay when using multiprocessing (with joblib). here simplified version of code:

import numpy np joblib import parallel, delayed  class matcher(object):     def match_all(self, arr1, arr2):         args = ((elem1, elem2) elem1 in arr1 elem2 in arr2)          results = parallel(n_jobs=-1)(delayed(_parallel_match)(self, e1, e2) e1, e2 in args)         # ...      def match(self, i1, i2):         return i1 == i2  def _parallel_match(m, i1, i2):     return m.match(i1, i2)  matcher = matcher() matcher.match_all(np.ones(250), np.ones(250))

so if run shown above, takes 30 secs complete , use 200mb. if change parameter n_jobs in parallel , set 1 takes 1.80 secs , barely use 50mb...

i suppose has related way pass arguments, haven't found better way it...

i'm using python 2.7.9

i have re-written code without using joblib library , works supposed work, although not "beautiful" code:

import itertools import multiprocessing import numpy np   class matcher(object):     def match_all(self, a1, a2):         args = ((elem1, elem2) elem1 in a1 elem2 in a2)         args = zip(itertools.repeat(self), args)          pool = multiprocessing.pool()         results = np.fromiter(pool.map(_parallel_match, args))         # ...      def match(self, i1, i2):         return i1 == i2  def _parallel_match(*args):     return args[0][0].match(*args[0][1:][0])  matcher = matcher()  matcher.match_all(np.ones(250), np.ones(250))

this version works charm, , takes 0.58 secs complete...

so, why isn't working @ joblib? can't understand it, guess joblib making copies of whole array every single process...


Comments

Post a Comment

Popular posts from this blog

jquery - How do you format the date used in the popover widget title of FullCalendar? -

Image
in fullcalendar, how format date used in popover widget title when clicking '+n more'? code: <div class="fc-popover fc-more-popover"> <div class="fc-header fc-widget-header"> <span class="fc-close fc-icon fc-icon-x"></span> <span class="fc-title">tuesday, april 7</span> ... </div> ... </div> you use daypopoverformat that. determines date format of title of popover created eventlimitclick option. must date formatting string. default value "dddd, mmmm d" english , "ll" other languages.
Read more

Bubble Sort Manually a Linked List in Java -

Image
this first question here. trying manually sort linked list of integers in java , can not figure out wrong code. suggestions? don't error, still have output unordered. tried few different ways, nothing worked. appreciate if can me that. public class node { int data; node nextnode; public node(int data) { this.data = data; this.nextnode = null; } public int getdata() { return this.data; } } // node class public class datalinkedlist implements datainterface { private node head; private int size; public datalinkedlist(){ this.head = null; this.size = 0; } public void add(int data) { node node = new node(data); if (head == null) { head = node; } else { node currentnode = head; while(currentnode.nextnode != null) { currentnode = currentnode.nextnode; } currentnode.nextnode = node; ...
Read more

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a...
Read more