apache spark - pyspark how to load compressed snappy file -


i have compressed file using python-snappy , put in hdfs store. trying read in following traceback. can't find example of how read file in can process it. can read text file (uncompressed) version fine. should using sc.sequencefile ? thanks!

i first compressed file , pushed hdfs  python-snappy -m snappy -c gene_regions.vcf gene_regions.vcf.snappy hdfs dfs -put gene_regions.vcf.snappy /  added following spark-env.sh export spark_executor_memory=16g                                                 export hadoop_home=/usr/local/hadoop                                              export java_library_path=$java_library_path:$hadoop_home/lib/native              export ld_library_path=$ld_library_path:$hadoop_home/lib/native                  export spark_library_path=$spark_library_path:$hadoop_home/lib/native            export spark_classpath=$spark_classpath:$hadoop_home/lib/lib/snappy-java-1.1.1.8-snapshot.jar  launch spark master , slave , ipython notebook executing code below.  a_file = sc.textfile("hdfs://master:54310/gene_regions.vcf.snappy") a_file.first()

valueerror traceback (most recent call last) in () ----> 1 a_file.first()

/home/user/software/spark-1.3.0-bin-hadoop2.4/python/pyspark/rdd.pyc in first(self) 1244 if rs: 1245 return rs[0] -> 1246 raise valueerror("rdd empty") 1247 1248 def isempty(self):

valueerror: rdd empty

working code (uncompressed) text file a_file = sc.textfile("hdfs://master:54310/gene_regions.vcf") a_file.first()

output: u'##fileformat=vcfv4.1'

the issue here python-snappy not compatible hadoop's snappy codec, spark use read data when sees ".snappy" suffix. based on same underlying algorithm aren't compatible in can compress 1 , decompress another.

you can make work either writing data out in first place snappy using spark or hadoop. or having spark read data binary blobs , manually invoke python-snappy decompression (see binaryfiles here http://spark.apache.org/docs/latest/api/python/pyspark.html). binary blob approach bit more brittle because needs fit entire file in memory each input file. if data small enough work.


Comments

Post a Comment

Popular posts from this blog

jquery - How do you format the date used in the popover widget title of FullCalendar? -

Image
in fullcalendar, how format date used in popover widget title when clicking '+n more'? code: <div class="fc-popover fc-more-popover"> <div class="fc-header fc-widget-header"> <span class="fc-close fc-icon fc-icon-x"></span> <span class="fc-title">tuesday, april 7</span> ... </div> ... </div> you use daypopoverformat that. determines date format of title of popover created eventlimitclick option. must date formatting string. default value "dddd, mmmm d" english , "ll" other languages.
Read more

Bubble Sort Manually a Linked List in Java -

Image
this first question here. trying manually sort linked list of integers in java , can not figure out wrong code. suggestions? don't error, still have output unordered. tried few different ways, nothing worked. appreciate if can me that. public class node { int data; node nextnode; public node(int data) { this.data = data; this.nextnode = null; } public int getdata() { return this.data; } } // node class public class datalinkedlist implements datainterface { private node head; private int size; public datalinkedlist(){ this.head = null; this.size = 0; } public void add(int data) { node node = new node(data); if (head == null) { head = node; } else { node currentnode = head; while(currentnode.nextnode != null) { currentnode = currentnode.nextnode; } currentnode.nextnode = node; ...
Read more

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a...
Read more