memory efficient way to change and parse a large XML file in python -


i want parse large xml file (25 gb) in python, , change of elements.

i tried elementtree xml.etree takes time @ first step (elementtree.parse).

i read somewhere sax fast , not load entire file memory parsing not modifying.

'iterparse' should parsing not modifying.

is there other option fast , memory efficient?

what important here need streaming parser, sax is. (there built in sax implementation in python , lxml provides one.) problem since trying modify xml file, have rewrite xml file read it.

an xml file text file, can't go , change data in middle of text file without rewriting entire text file (unless data exact same size unlikely)

you can use sax read in each element , register event write each element after been read , modified. if changes simple may faster not bother xml parsing , match text looking for.

if doing signinficant work large of xml file, shouldn't using xml file, should using database.

the problem have run here same issue cobol programmers on mainframes had when working file based data


Comments

Post a Comment

Popular posts from this blog

shopping cart - Page redirect not working PHP -

im trying redirect user when click "empty cart". redirect working "add cart" not "empty cart". ideas great. in main .php file set current url seen $current_url = base64_encode("http://$_server[http_host]$_server[request_uri]"); then below cartaction.php file handles actions such "add cart" , "empty cart" <?php session_start(); include_once 'config.php'; //add item in shopping cart if(isset($_post["add"])) { $product_id = $_post["product_id"]; //product id $return_url = base64_decode($_post["return_url"]); //return url $qty = $_post["qty"]; $product = $mysqli->query("select product_name, unit_price, unit_quantity products product_id = '$product_id'"); $obj = $product->fetch_object(); if(!isset($_session['products'])) { $_session['products'] = array(); } if(array_key_exists($product_id...
Read more

php - How to modify a menu to show sub-menus -

i have custom php menu made person, displays categories menu. however, need display subcategories well. tried adding hierarchical parameter didn't work. hoping assistance! here code: <ul> <?php /* categories wp e-commerce products */ $wpec_product_categories = get_terms( 'wpsc_product_category', 'hide_empty=0&parent=0&exclude=298'); $myarray = array(); foreach($wpec_product_categories $wpec_categories){ $myarray[$wpec_categories->term_id]['name'] = $wpec_categories->name; $myarray[$wpec_categories->term_id]['slug'] = $wpec_categories->slug; $myarray[$wpec_categories->term_id]['term_id'] = $wpec_categories->term_id; } foreach($myarray $wpec_categories1){ $wpec_term_name = $wpec_categories1['name']; $wpec_term_slug = $wpec_categories1['slug']; $wpec_term_id = $wpec_categories1['term_id']; echo '<li> <a href="'.get_page_link(19)....
Read more

python - Installing PyDev in eclipse is failed -

unable read repository @ http://pydev.sf.net/updates/content.xml. sun.security.validator.validatorexception: pkix path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable find valid certification path requested target i installing pydev in eclipse using python, doesn't install successfully. gives above error. happened? how solve error? i had error when trying install on laptop behind corporate firewall. couldn't tell setup of firewall, not sure of root cause. i got passed error installing pydev eclipse when out behind firewall. another way got around have eclipse on non-firewalled machine, install pydev eclipse, , copy whole eclipse directory non-firewalled machine firewalled machine. there better way, , way fix properly, didn't have time work out firewall doing. this on windows 7 machine (yeah know) not work *nix versions installed through package manager yum or apt.
Read more