Using Scrapy to scrape MITOPENCOURSEWARE site for syllabi

manpreet Best Answer 3 years ago

My spider to scrape the MITOPENCOURSEWARE site for syllabi doesn't work. Will someone please help me figure out what's wrong with it? The .*'s are to get to all the courses. Is this right?

  1 from scrapy.contrib.spiders import CrawlSpider, Rule
  2 from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
  3 from scrapy.selector import HtmlXPathSelector
  4 from opensyllabi.items import OpensyllabiItem
  5 
  6 class MITSpider(CrawlSpider):
  7     name = 'mit'
  8     allowed_domains = ['ocw.mit.edu']
  9     start_urls = ['http://ocw.mit.edu/courses']
 10     rules = [Rule(SgmlLinkExtractor(allow=['/.*/.*/syllabus']), 'parse_syllabus')]
 11 
 12     def parse_syllabus(self, response):
 13         x = HtmlXPathSelector(response)
 14 
 15         syllabus = OpensyllabiItem()
 16         syllabus['url'] = response.url
 17         syllabus['body'] = x.select("//div[@id='course_inner_section']").extract()
 18         return syllabus