{"id":837,"date":"2010-09-08T15:44:15","date_gmt":"2010-09-08T23:44:15","guid":{"rendered":"http:\/\/owenmundy.com\/blog\/?p=837"},"modified":"2010-09-13T01:57:51","modified_gmt":"2010-09-13T09:57:51","slug":"setup-macports-python-and-scrapy-successfully","status":"publish","type":"post","link":"https:\/\/owenmundy.com\/blog\/2010\/09\/setup-macports-python-and-scrapy-successfully\/","title":{"rendered":"Setup Macports Python and Scrapy successfully"},"content":{"rendered":"

\"logo\"<\/a><\/p>\n

“Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.”<\/p>\n

But, it can be a little tricky to get running…<\/p>\n

Attempting to install Scrapy on my MBP with the help of this post<\/a> I kept running into errors with the libxml and libxslt libraries using the Scrapy documentation<\/a>.<\/p>\n

I wanted to try to let Macports manage all the libraries but I had trouble with it referencing the wrong installation of Python. I began with three installs:<\/p>\n

    \n
  1. The default Apple Python 2.5.1 located at: \/usr\/bin\/python<\/li>\n
  2. A previous version I had installed located: \/Library\/Frameworks\/Python.framework\/Versions\/2.7<\/li>\n
  3. And a Macport version located: \/opt\/local\/bin\/python2.6<\/li>\n<\/ol>\n

    My trouble was that:<\/p>\n

    $ python<\/code><\/p>\n

    would always default to the 2.7 when I needed it to use the Macports version. The following did not help:<\/p>\n

    $ sudo python_select python26<\/code><\/p>\n

    I even removed the 2.7 version which caused only an error.<\/p>\n

    I figured out I needed to change the default path to the Macports version using the following:<\/p>\n

    $ PATH=$PATH\\:\/opt\/local\/bin ; export PATH<\/code><\/p>\n

    And then reinitiate the ports, etc.<\/p>\n

    Finally, I was not able to reference the scrapy-ctl.py file by default through these instructions so I had to reference the scrapy-ctl.py file directly<\/p>\n

    \/opt\/local\/Library\/Frameworks\/Python.framework\/Versions\/2.6\/bin\/scrapy-ctl.py<\/code><\/p>\n

    UPDATE<\/strong><\/p>\n

    A quick addendum to this post with instructions to create the link, found on the Scrapy site<\/a> (#2 and #3).<\/p>\n

    Starting with #2, “Add Scrapy to your Python Path”<\/p>\n

    sudo ln -s \/opt\/local\/Library\/Frameworks\/Python.framework\/Versions\/2.6\/bin\/scrapy-ctl.py \/opt\/local\/Library\/Frameworks\/Python.framework\/Versions\/2.6\/lib\/python2.6\/site-packages\/scrapy<\/code><\/p>\n

    And #3, “Make the scrapy command available”<\/p>\n

    sudo ln -s \/opt\/local\/Library\/Frameworks\/Python.framework\/Versions\/2.6\/bin\/scrapy-ctl.py \/usr\/local\/bin\/scrapy<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"

    “Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.” But, it can be a little tricky to get running… Attempting to install Scrapy on […]<\/p>\n

    Read More… from Setup Macports Python and Scrapy successfully<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[42],"tags":[165,78],"class_list":["post-837","post","type-post","status-publish","format-standard","hentry","category-code","tag-code","tag-python"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2ovQg-dv","_links":{"self":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/837","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/comments?post=837"}],"version-history":[{"count":10,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/837\/revisions"}],"predecessor-version":[{"id":844,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/837\/revisions\/844"}],"wp:attachment":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/media?parent=837"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/categories?post=837"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/tags?post=837"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}