spark-How can I retrieve item-pair after calculating similarity using RowMatrix -


i have encountered "all-pairs similarity" problem in recommendation system. this databricks blog, seems rowmatrix may come help.

however, rowmatrix matrix type without meaningful row indices, thereby don't know how retrieve similarity result after invoking columnsimilarities(threshold) specific item , j

below details doing:

1) data file comes movielens format this:

user::item::rating

2) build rowmatrix in each sparse vector i represents ratings of users item i

val datapath = ... val ratings: rdd[rating] = sc.textfile(datapath).map(_.split("::") match {    case array(user, item, rate) => rating(user.toint, item.toint, rate.todouble) }) val rows = ratings.map(rating=>(rating.product, (rating.user, rating.rating)))   .groupbykey()   .map(p => vectors.sparse(useramount, p._2.map(r=>(r._1-1, r._2)).toseq))  val mat = new rowmatrix(rows)  val similarities = mat.columnsimilarities(0.5)

now coordinatematrix similarities. how can similarity of specific item , j? although can used retrieve rdd[matrixentry], not sure whether row , column j correspond item , j.

i have encountered same problem , solved follows.

  1. you should note columnsimilarities() call similarity of column vectors. however, our "rows" composed of row vectors. should transpose of "rows", let's suppose "tran_rows". calculate tran_rows.columnsimilarities()

  2. thing easy then. in result of columnsimilarities(), the index , j correspond item , item j.


Comments

Post a Comment

Popular posts from this blog

jquery - How do you format the date used in the popover widget title of FullCalendar? -

Image
in fullcalendar, how format date used in popover widget title when clicking '+n more'? code: <div class="fc-popover fc-more-popover"> <div class="fc-header fc-widget-header"> <span class="fc-close fc-icon fc-icon-x"></span> <span class="fc-title">tuesday, april 7</span> ... </div> ... </div> you use daypopoverformat that. determines date format of title of popover created eventlimitclick option. must date formatting string. default value "dddd, mmmm d" english , "ll" other languages.
Read more

asp.net mvc - SSO between MVCForum and Umbraco7 -

i need integrate sso between mvcforum (forum of company) , umbraco7 (manages cms company). however struggling shared authentication work. tried several ways suggested on other threads still not running expected. here did: 1. in mvcforum (forum.mywebsite.com) 1.1. machine key in web.xml: <machinekey validationkey="the same key umbraco7" decryptionkey="the same key umbraco7" validation="sha1" decryption="aes" /> 1.2. authenticate forms in web.xml <forms name="my-sso-auth" protection="all" path="/" timeout="43200" domain="forum.mywebsite.com" loginurl="/members/logon" enablecrossappredirects="true" /> 2. in umbraco7 (news.mywebsite.com) 2.1. machine key in web.xml: <machinekey validationkey="the same key mvcforum" decryptionkey="the same key mvcforum" validation="sha1" decryption="aes" /> 2.2. a...
Read more

Python Tkinter keyboard using bind -

i have problem using bind option can click button in keyboard, , call function once key pressed. tried taking other codes have similar purpose , saw i'm doing pretty same, though still have problem must've missed. thankful if me this. here's code: # -*- coding: utf-8 -*- #imports tkinter import * pil import imagetk, image import time import tkmessagebox #===========================================================================================================================================# #tkinter class class app(frame): def __init__(self, master=none): frame.__init__(self, master) self.pack() #===========================================================================================================================================# #game pieces classes class board: def __init__(self, rows = 7, columns = 6, picture_height = none, picture_width = none): self.rows = rows self.columns = columns self.picture ...
Read more