python - SQLite reads with multiprocessing, not starting the imap -
i've tried think of, can not figure out why following multiprocessing code not start loop:
import sqlite3, itertools # dummy table conn = sqlite3.connect(":memory:") conn.execute('create table numbers (num integer)') conn.executemany("insert numbers values (?)", ((x,) x in range(5))) conn.commit() cmd_search = "select * numbers" cursor = conn.execute(cmd_search) def nothing(x): return x import multiprocessing p = multiprocessing.pool() #itr = p.imap(nothing,cursor) # parallel version itr = itertools.imap(nothing, cursor) # serial version x in itr: print x when run "serial" version (using itertools.imap) expected output of (0,) (1,) (2,) (3,) (4,). using multiprocessing.imap version, nothing , loop exits silently. has sqlite cursor, switching cursor=range(5) works.
why won't multiprocessing work here?
by default sqlite3 won't let access of object outside of hte thread created it. multiprocessing.pool uses background thread queue objects, violates rule. can disable check, though passing check_same_thread=false sqlite3.connect:
conn = sqlite3.connect(":memory:", check_same_thread=false) once made change, code ran fine. without change, seeing error:
programmingerror: sqlite objects created in thread can used in same thread.the object created in thread id 140082824808256 , thread id 140082770409216 i'm not sure why weren't seeing error message; way didn't 1 if removed for x in itr: print x line, since without don't try retrieve results pool, suppress errors occurred inside it.
Comments
Post a Comment