python - scrapy append to linkextractor links -


i using crawlspider linkextractor crawl links.

how go appending parameters links linkextractor finds? add '?pag_sortorder=0&pag_perpage=999' each link linkextractorextracts.

this perfect use case using process_value argument:

from scrapy.contrib.linkextractors import linkextractor  addition = "?pag_sortorder=0&pag_perpage=999"  linkextractor(process_value=lambda x: x + addition)

Comments

Post a Comment

Popular posts from this blog

shopping cart - Page redirect not working PHP -

im trying redirect user when click "empty cart". redirect working "add cart" not "empty cart". ideas great. in main .php file set current url seen $current_url = base64_encode("http://$_server[http_host]$_server[request_uri]"); then below cartaction.php file handles actions such "add cart" , "empty cart" <?php session_start(); include_once 'config.php'; //add item in shopping cart if(isset($_post["add"])) { $product_id = $_post["product_id"]; //product id $return_url = base64_decode($_post["return_url"]); //return url $qty = $_post["qty"]; $product = $mysqli->query("select product_name, unit_price, unit_quantity products product_id = '$product_id'"); $obj = $product->fetch_object(); if(!isset($_session['products'])) { $_session['products'] = array(); } if(array_key_exists($product_id...
Read more

php - How to modify a menu to show sub-menus -

i have custom php menu made person, displays categories menu. however, need display subcategories well. tried adding hierarchical parameter didn't work. hoping assistance! here code: <ul> <?php /* categories wp e-commerce products */ $wpec_product_categories = get_terms( 'wpsc_product_category', 'hide_empty=0&parent=0&exclude=298'); $myarray = array(); foreach($wpec_product_categories $wpec_categories){ $myarray[$wpec_categories->term_id]['name'] = $wpec_categories->name; $myarray[$wpec_categories->term_id]['slug'] = $wpec_categories->slug; $myarray[$wpec_categories->term_id]['term_id'] = $wpec_categories->term_id; } foreach($myarray $wpec_categories1){ $wpec_term_name = $wpec_categories1['name']; $wpec_term_slug = $wpec_categories1['slug']; $wpec_term_id = $wpec_categories1['term_id']; echo '<li> <a href="'.get_page_link(19)....
Read more

python - Installing PyDev in eclipse is failed -

unable read repository @ http://pydev.sf.net/updates/content.xml. sun.security.validator.validatorexception: pkix path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable find valid certification path requested target i installing pydev in eclipse using python, doesn't install successfully. gives above error. happened? how solve error? i had error when trying install on laptop behind corporate firewall. couldn't tell setup of firewall, not sure of root cause. i got passed error installing pydev eclipse when out behind firewall. another way got around have eclipse on non-firewalled machine, install pydev eclipse, , copy whole eclipse directory non-firewalled machine firewalled machine. there better way, , way fix properly, didn't have time work out firewall doing. this on windows 7 machine (yeah know) not work *nix versions installed through package manager yum or apt.
Read more