python - Update for big count of NDB Entities fails -


i have simple task. after migration , adding new field (repeated , composite property) existing ndb entity (~100k entities) need setup default value it.

i tried code first:

q = dm.e.query(ancestor=dm.e.root_key) user in q.iter(batch_size=500):      user.field1 = [dm.e2()]      user.put()

but fails such errors:

2015-04-25 20:41:44.792 /**** 500 599830ms 0kb appengine-google; (+http://code.google.com/appengine) module=default version=1-17-0 w 2015-04-25 20:32:46.675 suspended generator run_to_queue(query.py:938) raised timeout(the datastore operation timed out, or data temporarily unavailable.) w 2015-04-25 20:32:46.676 suspended generator helper(context.py:876) raised timeout(the datastore operation timed out, or data temporarily unavailable.) e 2015-04-25 20:41:44.475 traceback (most recent call last): file "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in

the task runs on separate task queue have @ least 10 mins execute seems not enough it. strange other thing: warnings ndb. may there dead lock because of updates same entities other instances (initiated users) not sure.

anyway want know best practices (and simplest) such task. know mapreduce looks me overcomplicated such task.

update:

also tried use put_multi grabbing entities in array gae stops instance exceeds ~600 mb of memory (with 500 mb limit). seems not enough memory store entities (~100k).

after execute _migrate_users() process 50 users , create task process next 50 users , on. may use bigger batch size 50 depending on size of entities.

def _migrate_users(curs=none):   users, next_curs, more = user.query().fetch_page(50, start_cursor=curs)   user in users:     user.field1 = 'bla bla'   ndb.put_multi(users)   if more:     deferred.defer(_migrate_users, next_curs, _queue='default')

Comments

Post a Comment

Popular posts from this blog

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a
Read more

Python Tkinter keyboard using bind -

i have problem using bind option can click button in keyboard, , call function once key pressed. tried taking other codes have similar purpose , saw i'm doing pretty same, though still have problem must've missed. thankful if me this. here's code: # -*- coding: utf-8 -*- #imports tkinter import * pil import imagetk, image import time import tkmessagebox #===========================================================================================================================================# #tkinter class class app(frame): def __init__(self, master=none): frame.__init__(self, master) self.pack() #===========================================================================================================================================# #game pieces classes class board: def __init__(self, rows = 7, columns = 6, picture_height = none, picture_width = none): self.rows = rows self.columns = columns self.picture
Read more

ubuntu - Selenium Node Not Connecting to Hub, Not Opening Port -

i'm trying set demo selenium grid. have digital ocean ubuntu 14.02 droplet. (i have 2 , tried set using 2 different machines got nowhere.) i'm trying run hub , node both locally. here input both hub, node, , nmap results. https://gist.github.com/mekhami/d5bbecca5ff0de9d2dc9 so, node starting tcp connection reset , port it's supposed operate on closed. i have had no luck in number of different options , configurations work. but... of course, worked randomly once. don't know why, don't know did, know 1 brief moment, node connected , saw on grid console. tried run test , went down again , i've never seen come up. please let me know more information can provide. update: randomly came online again. don't know happened, might related timeout or time_wait or going on. this because vm/server not have enough entropy available. to resolve problem, install haveged (or other similar things). the instructions of how install haveged on ubu
Read more