web crawler - Scrapy python error - Missing scheme in request URL -


i'm trying pull file password protected ftp server. code i'm using:

import scrapy scrapy.contrib.spiders import xmlfeedspider scrapy.http import request crawler.items import crawleritem  class sitespider(xmlfeedspider):     name = 'site'     allowed_domains = ['ftp.site.co.uk']     itertag = 'item'      def start_requests(self):         yield request('ftp.site.co.uk/feed.xml',             meta={'ftp_user': 'test', 'ftp_password': 'test'})      def parse_node(self, response, selector):         item = crawleritem()         item['title'] = (selector.xpath('//title/text()').extract() or [''])[0]               return item

this traceback error get:

        traceback (most recent call last):                                                                         file "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run                  self.mainloop()                                                                                        file "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainloop             self.rununtilcurrent()                                                                                 file "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in rununtilc urrent                                                                                                               call.func(*call.args, **call.kw)                                                                       file "/usr/local/lib/python2.7/dist-packages/scrapy/utils/reactor.py", line 41, in __call__                return self._func(*self._a, **self._kw)                                                              --- <exception caught here> ---                                                                            file "/usr/local/lib/python2.7/dist-packages/scrapy/core/engine.py", line 112, in _next_reques t                                                                                                                    request = next(slot.start_requests)                                                                    file "/var/www/spider/crawler/spiders/site.py", line 13, in start_requests                              meta={'ftp_user': 'test', 'ftp_password': 'test'})                                           file "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 26, in __i nit__                                                                                                                self._set_url(url)                                                                                     file "/usr/local/lib/python2.7/dist-packages/scrapy/http/request/__init__.py", line 61, in _se t_url                                                                                                                raise valueerror('missing scheme in request url: %s' % self._url)                                    exceptions.valueerror: missing scheme in request url: ftp.site.co.uk/f eed.xml

you need add scheme url:

ftp://ftp.site.co.uk

the ftp url syntax defined as:

ftp://[<user>[:<password>]@]<host>[:<port>]/<url-path>

basically, this:

yield request('ftp://ftp.site.co.uk/feed.xml', ...)

read more schemas @ wikipedia: http://en.wikipedia.org/wiki/uri_scheme


Comments

Post a Comment

Popular posts from this blog

jquery - How do you format the date used in the popover widget title of FullCalendar? -

Image
in fullcalendar, how format date used in popover widget title when clicking '+n more'? code: <div class="fc-popover fc-more-popover"> <div class="fc-header fc-widget-header"> <span class="fc-close fc-icon fc-icon-x"></span> <span class="fc-title">tuesday, april 7</span> ... </div> ... </div> you use daypopoverformat that. determines date format of title of popover created eventlimitclick option. must date formatting string. default value "dddd, mmmm d" english , "ll" other languages.
Read more

Bubble Sort Manually a Linked List in Java -

Image
this first question here. trying manually sort linked list of integers in java , can not figure out wrong code. suggestions? don't error, still have output unordered. tried few different ways, nothing worked. appreciate if can me that. public class node { int data; node nextnode; public node(int data) { this.data = data; this.nextnode = null; } public int getdata() { return this.data; } } // node class public class datalinkedlist implements datainterface { private node head; private int size; public datalinkedlist(){ this.head = null; this.size = 0; } public void add(int data) { node node = new node(data); if (head == null) { head = node; } else { node currentnode = head; while(currentnode.nextnode != null) { currentnode = currentnode.nextnode; } currentnode.nextnode = node; ...
Read more

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a...
Read more