{"id":889,"date":"2010-09-17T15:39:11","date_gmt":"2010-09-17T23:39:11","guid":{"rendered":"http:\/\/owenmundy.com\/blog\/?p=889"},"modified":"2011-12-07T06:41:53","modified_gmt":"2011-12-07T14:41:53","slug":"how-to-install-scrapy-with-macports-full-version","status":"publish","type":"post","link":"https:\/\/owenmundy.com\/blog\/2010\/09\/how-to-install-scrapy-with-macports-full-version\/","title":{"rendered":"How to install Scrapy with MacPorts (full version)"},"content":{"rendered":"

\"macports-logo-top\" + <\/strong>\"logo\"<\/p>\n

Here is a step-by-step explaining how I got Scrapy<\/a> running on my MacBook Pro 10.5 using MacPorts<\/a> to install Python and all required libraries (libxml2, libxsit, etc.). The following has been tested on two separate machines with Scrapy .10.<\/p>\n

Many thanks to users here<\/a> who shared some helpful amendments to the default installation guide<\/a>. My original intention was to post this at stackoverflow, but their instructions discourage posting issues that have already been answered so here it is…<\/p>\n

1. Install Xcode<\/a> with options for command line development (a.k.a. “Unix Development”). This requires a free<\/a> registration.<\/p>\n

2. Install MacPorts<\/a><\/p>\n

3. Confirm and update MacPorts<\/p>\n

$ sudo port -v selfupdate<\/code><\/p>\n

4. “Add the following to \/opt\/local\/etc\/macports\/variants.conf to prevent downloading the entire unix library with the next commands”<\/p>\n

+bash_completion +quartz +ssl +no_x11 +no_neon +no_tkinter +universal +libyaml -scientific<\/code><\/p>\n

5. Install Python<\/p>\n

$ sudo port install python26<\/code><\/p>\n

If for any reason you forgot to add the above exceptions, then cancel the install and do a “clean” to delete all the intermediary files MacPorts created. Then edit the variants.conf file (above) and install Python.<\/p>\n

$ sudo port clean python26<\/code><\/p>\n

6. Change the reference to the new Python installation<\/p>\n

If you type the following you will see a reference to the default installation of Python on MacOS 10.5 (Python2.5).<\/p>\n

$ which python<\/code><\/p>\n

You should see this<\/p>\n

\/usr\/bin\/python<\/code><\/p>\n

To change this reference to the MacPorts installation, first install python_select<\/del><\/p>\n

$ sudo port install python_select<\/code><\/del><\/del><\/p>\n

Then use python_select to change the $ python reference to the Python version installed above.<\/del><\/p>\n

$ sudo python_select python26<\/code><\/del><\/p>\n

UPDATE 2011-12-07<\/strong>: python_select has been replaced by port select so…<\/p>\n

To see the possible pythons run<\/p>\n

port select --list python<\/code><\/p>\n

From that list choose the one you want and change to it e.g.<\/p>\n

sudo port select --set python python26<\/code><\/p>\n

Now if you type<\/p>\n

$ which python<\/code><\/p>\n

You should see<\/p>\n

\/opt\/local\/bin\/python<\/code><\/p>\n

which is a symlink to<\/p>\n

\/opt\/local\/bin\/python2.6<\/code><\/p>\n

Typing the below will now launch the Python2.6 shell editor (ctl + d to exit)<\/p>\n

$ python<\/code><\/p>\n

7. Install required libraries for Scrapy<\/p>\n

$ sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson<\/del><\/code><\/p>\n

Other posts recommended installing py26-setuptools but it kept returning with with errors, so I skipped it.<\/p>\n

8. “Test that the correct architectures are present:<\/p>\n

$ file `which python`<\/code><\/p>\n

The single quotes should be backticks, which should spit out (for intel macs running 10.5):”<\/p>\n

\/opt\/local\/bin\/python: Mach-O universal binary with 2 architectures
\n\/opt\/local\/bin\/python (for architecture i386):\tMach-O executable i386
\n\/opt\/local\/bin\/python (for architecture ppc7400):\tMach-O executable ppc<\/code><\/p>\n

9. Confirm libxml2 library is installed (those really are single quotes). If there are no errors it imported successfully.<\/p>\n

$ python -c 'import libxml2'<\/code><\/p>\n

10. Install Scrapy<\/p>\n

$ sudo \/opt\/local\/bin\/easy_install-2.6 scrapy<\/code><\/p>\n

<\/p>\n

11. Make the scrapy command available in the shell<\/p>\n

$ sudo ln -s \/opt\/local\/Library\/Frameworks\/Python.framework\/Versions\/2.6\/bin\/scrapy \/usr\/local\/bin\/scrapy<\/code><\/p>\n

One caveat for the above, on a fresh computer, you might not have a \/usr\/local\/bin directory so you will need to create it before you can run the above to create the symlink.<\/p>\n

$ sudo mkdir \/usr\/local\/bin<\/code><\/p>\n

13. Finally, type either of the following to confirm that Scrapy is indeed running on your system.<\/p>\n

$ python scrapy<\/code><\/p>\n

$ scrapy<\/code><\/p>\n

A final final bit… I also installed ipython<\/a> from Macports for use with Scrapy<\/p>\n

sudo port install py26-ipython<\/code><\/p>\n

Make a symbolic link
\nsudo ln -s \/opt\/local\/bin\/ipython-2.6 \/usr\/local\/bin\/ipython<\/code><\/p>\n

An article on ipython
\nhttp:\/\/onlamp.com\/pub\/a\/python\/2005\/01\/27\/ipython.html<\/a><\/p>\n

ipython tutorial
\nhttp:\/\/ipython.scipy.org\/doc\/manual\/html\/interactive\/tutorial.html<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

+ Here is a step-by-step explaining how I got Scrapy running on my MacBook Pro 10.5 using MacPorts to install Python and all required libraries (libxml2, libxsit, etc.). The following has been tested on two separate machines with Scrapy .10. Many thanks to users here who shared some helpful amendments to the default installation guide. […]<\/p>\n

Read More… from How to install Scrapy with MacPorts (full version)<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[42],"tags":[165,78],"class_list":["post-889","post","type-post","status-publish","format-standard","hentry","category-code","tag-code","tag-python"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p2ovQg-el","_links":{"self":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/889","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/comments?post=889"}],"version-history":[{"count":47,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/889\/revisions"}],"predecessor-version":[{"id":1894,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/posts\/889\/revisions\/1894"}],"wp:attachment":[{"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/media?parent=889"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/categories?post=889"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/owenmundy.com\/blog\/wp-json\/wp\/v2\/tags?post=889"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}