python - Beautiful Soup Strip html caption tags, th class tags and retrieve data not in list -


so have created web scraper goes cfbstats.com/2014/player/index.html , retrieves college football teams , links of football teams. there goes each link , takes roster , players link. goes each players link , takes stats.

i having problem taking players stats. when call header of each table printed output [tackle] , when call first row of table [g]. rid of tags. have been able not have them past functions. appreciated.

import csv import sys import json import urllib import requests bs4 import beautifulsoup import xlrd import xlwt       def getcollegeandurl():        f = open('colleges.csv', 'w')      f.write("teams" + "," + "," + "url" + '\n')     originalurl = "http://www.cfbstats.com/2014/player/index.html"     base = requests.get("http://www.cfbstats.com/2014/player/index.html")     base = base.text     soup = beautifulsoup(base)         # find colleges in div conference    mydivs = soup.find_all('div',{'class': 'conference'})      ##g excel document roster    g = open('rosters.csv', 'w')    g.write("college rosters" + '\n' + '\n' + 'college' + ',' + 'playernumber' + ',' + 'player last name' + ',' +'player first name' + ',' + 'position' + ',' + 'year' + ',' + 'height' + ',' + ' weight' + ',' +'hometown' + ',' + 'state' + ',' + 'last school' + ',' + '\n')     # h excel each player stats    h = xlwt.workbook()     # loop finds writes each college line    div in mydivs:         urls= div.findall('a')       # pull college names , each of links         url in urls:               college = url.text             url = url.attrs['href']              teamurl = originalurl[:23]+url              f.write(college[:]+ ',' + ',' + teamurl[:]+'\n')               scraperosters(college, teamurl, g, h)        ############################################################################  def scraperosters(college, teamurl, g, h):      # create excel documents     # gets pages of teams     roster = requests.get(teamurl)     roster = roster.text     roster = beautifulsoup(roster)           teamname = roster.find_all('h1' , {'id': 'pagetitle'})      teamandplayers = {}     table = roster.find_all('table', {'class' : 'team-roster'})        in table:          rows = i.find_all('tr')            row in rows[1:]:              data = [str(i.gettext()) in row('td')]             link = row('td')[1]('a')                  if len(link) > 0:                 link = str(link[0]['href'])                 data = [str(link)] + data                  # unpacking data variables                  (playerurl, playernumber, playername,    playerposition,yearincollege, playerheight, playerweight, playerhometown, lastschool) = data                  # creating full player url                 playerurl = teamurl[:23] + playerurl                  # repacking data                  data = (college, playernumber, playername, playerposition,yearincollege, playerheight, playerweight, playerhometown, lastschool)                      g.write(college + ',' + playernumber + ',' + playername + ',' + playerposition + ','+ yearincollege + ',' + playerheight + ',' + playerweight + ',' + playerhometown + ',' + lastschool+ ',' + ',' + playerurl + ',' + '\n')                    playerstats(data, playerurl, h)          ############################################################################ def playerstats(data,playerurl, h):      playerurl = requests.get(playerurl)     playerurl = playerurl.text     playerurl = beautifulsoup(playerurl)        tablestats = playerurl.find_all('table', {'class' : 'player-home'})       (college, playernumber, playername, playerposition,yearincollege, playerheight, playerweight, playerhometown, lastschool) = data      #print college, playernumber, playername      print college, playername, playernumber      x in tablestats:          caption = x.find_all('caption')                rows = x.find_all('tr')  ##        caption = caption.strip          row in rows:             headers = x.find_all('th')              headers = [str(i.gettext()) in row('tr')]              stats = [str(x.gettext()) x in row('td')]              print caption, headers, stats   ############################################################################ def main():     getcollegeandurl()    main()

don't work hard, data already available in parseable form.


Comments

Post a Comment

Popular posts from this blog

shopping cart - Page redirect not working PHP -

im trying redirect user when click "empty cart". redirect working "add cart" not "empty cart". ideas great. in main .php file set current url seen $current_url = base64_encode("http://$_server[http_host]$_server[request_uri]"); then below cartaction.php file handles actions such "add cart" , "empty cart" <?php session_start(); include_once 'config.php'; //add item in shopping cart if(isset($_post["add"])) { $product_id = $_post["product_id"]; //product id $return_url = base64_decode($_post["return_url"]); //return url $qty = $_post["qty"]; $product = $mysqli->query("select product_name, unit_price, unit_quantity products product_id = '$product_id'"); $obj = $product->fetch_object(); if(!isset($_session['products'])) { $_session['products'] = array(); } if(array_key_exists($product_id...
Read more

php - How to modify a menu to show sub-menus -

i have custom php menu made person, displays categories menu. however, need display subcategories well. tried adding hierarchical parameter didn't work. hoping assistance! here code: <ul> <?php /* categories wp e-commerce products */ $wpec_product_categories = get_terms( 'wpsc_product_category', 'hide_empty=0&parent=0&exclude=298'); $myarray = array(); foreach($wpec_product_categories $wpec_categories){ $myarray[$wpec_categories->term_id]['name'] = $wpec_categories->name; $myarray[$wpec_categories->term_id]['slug'] = $wpec_categories->slug; $myarray[$wpec_categories->term_id]['term_id'] = $wpec_categories->term_id; } foreach($myarray $wpec_categories1){ $wpec_term_name = $wpec_categories1['name']; $wpec_term_slug = $wpec_categories1['slug']; $wpec_term_id = $wpec_categories1['term_id']; echo '<li> <a href="'.get_page_link(19)....
Read more

python - Installing PyDev in eclipse is failed -

unable read repository @ http://pydev.sf.net/updates/content.xml. sun.security.validator.validatorexception: pkix path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable find valid certification path requested target i installing pydev in eclipse using python, doesn't install successfully. gives above error. happened? how solve error? i had error when trying install on laptop behind corporate firewall. couldn't tell setup of firewall, not sure of root cause. i got passed error installing pydev eclipse when out behind firewall. another way got around have eclipse on non-firewalled machine, install pydev eclipse, , copy whole eclipse directory non-firewalled machine firewalled machine. there better way, , way fix properly, didn't have time work out firewall doing. this on windows 7 machine (yeah know) not work *nix versions installed through package manager yum or apt.
Read more