Owen Mundy

I’m happy to share the code from a Facebook app I created for a workshop earlier this year.

This is an example of a simple Facebook canvas application. This code was originally demoed during the Freedom for Our Files Facebook API workshop at the 2011 Art Meets Radical Openness festival in Linz, Austria. You can view and download the source code on github

The application is very simple; it creates a Facebook object, performs calls to get data belonging to the current user, then prints the data exactly as it is returned. It has examples of basic Facebook Graph API calls as well as an example of FQL (Facebook Query Language).

The Processing.js project is really cool. It allows you to run Processing .pde files inside (HTML5 compatible) web browsers using Javascript. You can pass data back and forth between the two programs, access the DOM with Processing, and you don’t need any plugins or Java.

Click here to run this Processing sketch in your browser

One caveat… When importing a Processing .pde file into the HTML5 canvas you must access the files on a web server or by using localhost (a server running on your computer, e.g. MAMP) because most (all modern?) web browsers don’t allow file:/// access for security reasons. Unfortunately this is not intuitive as Javascript should run in the browser regardless of file:/// access. Nor is it mentioned in any of the Processing.js Quick Start documentation. I found it by testing, and then confirmed it in their README. Darn, need to remember to read (all of) the instructions.

UPDATE: A friend pointed out that the problem accessing the .pde file could be due to the same origin policy. Though not explicitly stated on the Github page for Processing.js, they do mention that disabling same origin setting in your browser is a(n undesirable) workaround.

Some web browsers (e.g., Chrome) require secondary files to be loaded from a web server for security reasons. This means loading a web page that references a Processing.js sketch in a file via a file:/// URL vs. http:// will fail. You are particularly likely to run into this problem when you try to view your webpage directly from file, as this makes all relatively links file:/// links.

// Hello World file for Processing or Processing.js
void setup(){
  size(1000,700);
  background(255);
}
void draw(){
  stroke(random(255),random(255), random(255));
  line(random(width),random(height), random(width),random(height));
  println("Hello World!");
}

This weekend I am presenting a lecture about GIve Me My Data and conducting a two-day data-scraping workshop at Art Meets Radical Openness in Linz, Austria. Here are the details.

The Self-Indulgence of Closed Systems
May 13, 18:45 – 19:15
Part artist lecture, part historical context, Owen Mundy will discuss his Give Me My Data project within the contexts of the history of state surveillance apparatuses, digital media and dialogical art practices, and the ongoing contradiction of privacy and utility in new media.

Freedom for Our Files: Creative Reuse of Personal Data
May 13-14, 14:00 – 16:30
A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours! Register here

Art Meets Radical Openness (LiWoLi 2011),
Date: 12th – 14th May 2011
Location: Kunstuniversität Linz, Hauptplatz 8, 4020 Linz, Austria

Observing, comparing, reflecting, imitating, testing, combining

LiWoLi is an open lab and meeting spot for artists, developers and educators using and creating FLOSS (free/libre open source software) and Open Hardware in the artistic and cultural context. LiWoLi is all about sharing skills, code and knowledge within the public domain and discussing the challenges of open practice.

This “Mutual friends network graph” created with Nodebox using data I exported with Give Me My Data contains 540 “Facebook friends” and their connections to each other. When the graph renders it attempts to position people who have lots of connections closer together. With this you can see groups unfold based on your own social networks. Since I have spent more time in academia than I have at specific jobs my “clusters” are based mostly on my academic history.

You can also see that there are a lot of connections between my high school and where I did my undergraduate study, which is based on the fact they are located very close to each other, so friends from high school also chose the same university or town to live in. There are also a lot of interconnections between Indiana University where I did my undergrad, the University of California, San Diego, where I did graduate study, and Florida State University, where I teach now. This is probably due to the fact that my connections are all within a given field, in my case visual arts, and points to the often expressed notion that “the art world is actually very small.”

Last weekend I took part in Random Hacks of Kindness an international hackathon dedicated to creating useful systems to respond to critical global challenges. I met with other programmers at the Betahaus in Berlin and worked with Tim Schwartz and Mikkel Gravgaard on Google Person Finder a searchable database of missing persons that helps people find loved ones during disasters. It was used during the 2010 Haiti and Chilean Earthquakes and is developed by volunteers and employees of Google.

I launched Keyword Intervention in January 2007 and for almost four years now it has been scraping topical search terms and attracting random traffic. Today I moved the project to its own domain, keywordintervention.com and also updated the documentation on the site. Below is a sample of the last 500 search terms by users all around the world. The full list is here.

+

Here is a step-by-step explaining how I got Scrapy running on my MacBook Pro 10.5 using MacPorts to install Python and all required libraries (libxml2, libxsit, etc.). The following has been tested on two separate machines with Scrapy .10.

Many thanks to users here who shared some helpful amendments to the default installation guide. My original intention was to post this at stackoverflow, but their instructions discourage posting issues that have already been answered so here it is…

1. Install Xcode with options for command line development (a.k.a. “Unix Development”). This requires a free registration.

2. Install MacPorts

3. Confirm and update MacPorts

$ sudo port -v selfupdate

4. “Add the following to /opt/local/etc/macports/variants.conf to prevent downloading the entire unix library with the next commands”

+bash_completion +quartz +ssl +no_x11 +no_neon +no_tkinter +universal +libyaml -scientific

5. Install Python

$ sudo port install python26

If for any reason you forgot to add the above exceptions, then cancel the install and do a “clean” to delete all the intermediary files MacPorts created. Then edit the variants.conf file (above) and install Python.

$ sudo port clean python26

6. Change the reference to the new Python installation

If you type the following you will see a reference to the default installation of Python on MacOS 10.5 (Python2.5).

$ which python

You should see this

/usr/bin/python

To change this reference to the MacPorts installation, first install python_select

$ sudo port install python_select

Then use python_select to change the $ python reference to the Python version installed above.

$ sudo python_select python26

UPDATE 2011-12-07: python_select has been replaced by port select so…

To see the possible pythons run

port select --list python

From that list choose the one you want and change to it e.g.

sudo port select --set python python26

Now if you type

$ which python

You should see

/opt/local/bin/python

which is a symlink to

/opt/local/bin/python2.6

Typing the below will now launch the Python2.6 shell editor (ctl + d to exit)

$ python

7. Install required libraries for Scrapy

$ sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson

Other posts recommended installing py26-setuptools but it kept returning with with errors, so I skipped it.

8. “Test that the correct architectures are present:

$ file `which python`

The single quotes should be backticks, which should spit out (for intel macs running 10.5):”

/opt/local/bin/python: Mach-O universal binary with 2 architectures
/opt/local/bin/python (for architecture i386): Mach-O executable i386
/opt/local/bin/python (for architecture ppc7400): Mach-O executable ppc

9. Confirm libxml2 library is installed (those really are single quotes). If there are no errors it imported successfully.

$ python -c 'import libxml2'

10. Install Scrapy

$ sudo /opt/local/bin/easy_install-2.6 scrapy

11. Make the scrapy command available in the shell

$ sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy /usr/local/bin/scrapy

One caveat for the above, on a fresh computer, you might not have a /usr/local/bin directory so you will need to create it before you can run the above to create the symlink.

$ sudo mkdir /usr/local/bin

13. Finally, type either of the following to confirm that Scrapy is indeed running on your system.

$ python scrapy

$ scrapy

A final final bit... I also installed ipython from Macports for use with Scrapy

sudo port install py26-ipython

Make a symbolic link
sudo ln -s /opt/local/bin/ipython-2.6 /usr/local/bin/ipython

An article on ipython
http://onlamp.com/pub/a/python/2005/01/27/ipython.html

ipython tutorial
http://ipython.scipy.org/doc/manual/html/interactive/tutorial.html

The German newspaper, Bild published another article mentioning Give Me My Data today.

Was weiß Facebook über mich? or in English, Facebook knows what about me?

“About 500 million people worldwide use the social network Facebook to stay in touch with friends. In Germany, almost 9.8 million people are registered with Facebook. Many users are worried about their privacy. BILD.de answered the important questions…”

“Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.”

But, it can be a little tricky to get running…

Attempting to install Scrapy on my MBP with the help of this post I kept running into errors with the libxml and libxslt libraries using the Scrapy documentation.

I wanted to try to let Macports manage all the libraries but I had trouble with it referencing the wrong installation of Python. I began with three installs:

1. The default Apple Python 2.5.1 located at: /usr/bin/python
2. A previous version I had installed located: /Library/Frameworks/Python.framework/Versions/2.7
3. And a Macport version located: /opt/local/bin/python2.6

My trouble was that:

$ python

would always default to the 2.7 when I needed it to use the Macports version. The following did not help:

$ sudo python_select python26

I even removed the 2.7 version which caused only an error.

I figured out I needed to change the default path to the Macports version using the following:

$ PATH=$PATH\:/opt/local/bin ; export PATH

And then reinitiate the ports, etc.

Finally, I was not able to reference the scrapy-ctl.py file by default through these instructions so I had to reference the scrapy-ctl.py file directly

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py

UPDATE

A quick addendum to this post with instructions to create the link, found on the Scrapy site (#2 and #3).

Starting with #2, “Add Scrapy to your Python Path”

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scrapy

And #3, “Make the scrapy command available”

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /usr/local/bin/scrapy

I’ve been trying for a few hours to run a Python script from The Sunlight Foundation Labs which downloads (and updates) a campaign finance database from the Center for Responsive Politics. See their original post for more information.

In the process of getting this working I accidentally broke a working copy of MySQL and overwrote a database installed on my MBP (which I had stupidly not backed-up since last year). FYI, you can rebuild any MySQL database with the original .frm, .MYD, and .MYI files if you 1. Recreate the database in the new install of MySQL and 2. Drag the files into the mysql data folder.

I struggled quite a bit getting Python to work with MySQL via MySQLdb. I’m documenting some of the headaches and resolutions here in case they are useful. I’ve tried to include error messages for searches as well.

The Sunlight Foundation instructions require Python and MySQL, but don’t mention you have to have already wrestled with the madness involved in installing Django on your machine. Here is what I did to get it working on my MacBook Pro Intel Core 2 Duo. I’ve included their original instructions with my own (and a host of others).

Instructions

1. Install MAMP.

   While I had working installations of MySQL and Python (via installers on respective sites), I couldn’t get Python to connect to MySQL via MySQLdb. I decided to download and try MAMP for a clean start.

2. Install XCode

   Past installs are available on Apple Developer website.

3. Install setuptools

   Required for the MySQLdb driver. Remove the .sh extension from the filename (setuptools-0.6c11-py2.7.egg.sh) and in a shell:

   ~$ chmod +x setuptools-0.6c11-py2.7.egg
~$ ./setuptools-0.6c11-py2.7.egg

4. Install the MySQLdb driver

   After downloading and unzipping, from the directory:

   ~$ python setup.py build
~$ sudo python setup.py install

   Continue following the advice of this post to the end How to install Django with MySQL on Mac OS X.

   I also followed another piece of advice in Python MySQL on a Mac with MAMP to change the mysql_config.path from:

   /usr/local/mysql/bin/mysql_config

   to

   /Applications/MAMP/Library/bin/mysql_config

   Especially useful is his test script for making sure that Python is indeed accessing MySQL.

5. Create a symbolic link between Python and MySQL in MAMP

   This is required in order to use a socket to connect to the MySQL. See How to install MySQLdb on Leopard with MAMP for more information.

   ~$ sudo ln -s /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock

6. Create a directory and put the two Python files in it.
7. Modify the top of the sun_crp.py file to set certain parameters–your login credentials for the CRP download site and your MySQL database information.
8. Install pyExcelerator

   Error: ImportError: No module named pyExcelerator

   I had to install this module next.

9. Comment out multiple lines

   Error: NameError: name 'BaseCommand' is not defined

   In download.py comment out the following:

   The line: from django.core.management.base import BaseCommand, CommandError

   Everything from class CRPDownloadCommand(BaseCommand): to the end of the document.

10. From the command line, run the script by typing, from the proper directory: Python sun-crp.py.
11. It will take several hours to download and extract the data, especially the first time it’s run. But after that, you’re good to go.