Internet service just got creepy: How to set up a wireless router with Comcast internet service

I just moved back to Florida after a one year research project in Berlin and have subscribed to Comcast broadband service. The whole experience left a bad taste in my mouth, though not because the tech showed-up 2 hours after the installation appointment window. Nor was it because he held loud personal conversations on his cell phone while he was setting up the service. No, the icky feeling is more corporate and selfish, and impedes much more into my private space than “Joe the cable guy” ever could.

Comcast made me install software on my computer in order to use their broadband.

Upon his arrival, “Joe” announced he would need access to my computer to setup broadband service. Understanding that most of the people Joe deals with might not be IT whizzes, and could manage to not be able to connect their machines without his help, I decided to let him use it rather than attempt to prove I was not a member of the usual group. After half an hour of complaining about previous customers to his friend on his cellphone, waiting for an other Comcast person to flip a switch allowing him to do his job, and multiple trips to his truck, he showed me that the internet was indeed accessible on my computer.

At this point the laptop was directly connected to the cable modem via an ethernet cable. He announced I was to follow the steps on the screen and he was out the door. The web page he had left up required me to agree to some terms, create a username and then… install software? Really? I tried to access the net without the final step but nothing doing. Unless I installed this software I was stuck. So I did it, still not believing that a company had really initiated this final invasion onto every customer’s computer. After it was done I had new bookmarks everywhere, for Comcast email, security, and some branding nonsense called “XFINITY” (I thought “X” was out with the ’90’s and “X”games?)

So I thought, “OK, Comcast, you got me, hit me with your best marketing slime. Whatever, I can delete the bookmarklets you installed in my browser, just let me access the service I paid for, wirelessly, on whichever device I want.”

But this is where the relationship got really creepy. Apparently when I installed the Comcast (spyware?) on my machine, it made note of my MAC address, a unique identifier of networked machines, so that it would only allow my machine (or another machine with that MAC address) to connect to the internet. This means when I attached a wireless router to the cable modem I could connect to the wifi, but there was no internet.

So it turns-out that Comcast is not only forcing their adware on customers, it’s also making it difficult (though not impossible) for them use more than one device. Presumably Comcast is doing this in order to circumvent sharing of services among neighbors, but the end result is that you can’t share the service between more than one device, or between roommates or spouses for that matter.

An example (albeit a geeky one): between my wife and I we have 2 laptops, 2 smartphones, and a desktop computer that all might be talking to each other or accessing the net. Comcast’s so-called internet service didn’t allow for any such geekery because it only allows one device, with the correct MAC address, to connect.

So, here’s what I did, on my Mac, with some help from my sister’s boyfriend, Tom, and a lot from Google, to get my linksys wireless router to work with Comcast internet.

  1. Confirm you can access the internet with your machine connected directly to the Comcast cable modem.
  2. Open Terminal and type (without the quotes): “ifconfig en0 | grep ether”
  3. Now disconnect your computer from the modem and connect the modem ethernet cable to your wireless router. Make sure both are plugged-in.
  4. Connect to your wireless router via the airport on your machine.
  5. Go to the following link: http://192.168.1.1
  6. Under Setup, choose DHCP as the Internet Connection Type. Save Settings.
  7. Under Setup : Mac Address Clone, enter the alpha numeric characters returned from Terminal. Save Settings.
  8. Configure your wireless router like you normally would and you are up and running.
  9. Snicker at Comcast

Facebook’s God complex: No data for the “wicked”

I was quoted recently in a TechCrunch article about the ongoing battle between Google and Facebook to draw users to their respective social networking services. In the article, The Only Backdoor Left To Sneak Your Facebook Friends Into Google+ Is Yahoo, Erick Schonfeld writes:

‘Over the weekend, Facebook blocked a Google Chrome extension called the Facebook Friend Exporter. And in fact, Facebook changed its OAuth 2.0 API in such a way that it “suddenly removed email addresses from the queries without warning,” says Owen Mundy, creator of Give Me My Data. Other data can still be exported, just not your friends’ email addresses.’ [1]

I wanted to clarify something about the above because my contribution is slightly vague. I also wanted to elaborate for other developers and people interested in how to get their data, specifically the email addresses *out* of Facebook.

Getting your data our of Facebook is a hot issue right now with the emergence of Google+. It’s important to be able to preserve your data, especially in the event you want to exit Facebook, or prepare for its possible demise. I think most people don’t care which social networking software they use, as long as they can stay in touch with their friends. Many depend on Facebook to keep their friends’ contact information up to date. Their friends’ phone numbers, emails, and physical addresses may have changed, but they can still be found on Facebook.

This is one great benefit of Facebook, but I think they’ve come to take a higher-than-thou approach to user data. For example, if I use a single software on my computer to manage contact data for my friends, send them messages with pictures, etc., I am not locked-into a contract with the machine nor the software. I physically have the device that stores this data in my possession, and therefore can do whatever I like with it—assuming I can wrangle my data out of it.

One of the great benefits of the cloud is access, right? Wrong. Facebook has a god’s eye view regarding the matter of our data. The view from on high is that they can do whatever benefits Facebook, not necessarily the user. So they continue profiting from our activity while protecting it’s evidence from any company/person/software which could compromise their lead in social networking. Unfortunately this includes us, so we are ultimately at their mercy.

But Facebook is not a merciless God. They don’t charge to use their service or store my data. This is the tradeoff. We sacrifice privacy—our right to not be tracked, advertised to, or sniffed by governments—in order to play games, have stupid political debates, and post pictures of our kids. So, they have the right to allow or prevent access to this information, regardless of whether it is right.

Regarding getting email addresses of your contacts, as far as I know, there are three ways to mass export your friends’ contact data, only one of which is currently allowed by Facebook.

Hacking

The Facebook Friend Exporter, created by Mohamed Mansour, is a browser extension that works independent of the Facebook API. It scrapes your friends’ contact data from the Facebook pages you visit and exports them as CSV. First, Facebook’s rules here for reusing this type of data are not exactly clear. While they don’t explicitly ban saving the contact info of your friends, the have a catch-all statement that tries to set guidelines for doing so. One could argue by accepting your friend request a user has already consented to sharing with you.

“If you collect information from users, you will: obtain their consent, make it clear you (and not Facebook) are the one collecting their information, and post a privacy policy explaining what information you collect and how you will use it.” [2]

One thing that is clear is the method the Facebook Friend Exporter extension uses is banned according to Facebook’s Statement of Rights and Responsibilities:

“You will not collect users’ content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our permission.” [2]

And as expected, Facebook has taken measures to prevent Mansour’s app from working.

“Mansour says that Facebook removed emails from their mobile site, which were critical to the original design of his extension. He told me that the company had implemented a throttling mechanism: if you visit any friend page five times in a short period of time, the email field is removed.” [3]

This is not the first time Facebook has referenced their Statement of Rights and Responsibilities to take measures to keep user data from hackers, artists, or competitors. In 2010 they served moddr and other makers of the Web 2.0 Suicide Machine with a cease and desist letter to prevent them from providing a service for Facebook users to delete their accounts. The letter, which claimed the project was guilty of “Soliciting users’ Facebook login information; Accessing a Facebook account belonging to someone else; Collecting Facebook users’ content or information using automated means such as scripts or scrapers without Facebook’s permission” was possibly effective. Currently only users of MySpace, LinkedIn and Twitter can commit “virtual suicide.”

The Face to Facebook (see image on left) project by Paolo Cirio and Alessandro Ludovico stole one-million Facebook profiles, filtered them with face-recognition software, and then posted them on a fake dating website sorted by facial characteristics. From a statement by the makers:

“Everybody can steal personal data and re-contextualize it in a completely unexpected context. And that shows, once more, how fragile and potentially manipulable the online environment actually is.”

I agree, and I like the project. But in a public work such as this it’s hard to argue, in my opinion, that this project will help the average web user. While many may reconsider the type of information they post, more than likely they will think this project is another example of the “bad guys” (a.k.a. hackers) doing bad things. While it critiques, in a very amusing and relevant way, the issue of online privacy, it leaves the user feeling violated, not necessarily thoughtful. As a work of protest it encounters the same problem that holding a giant sign in someone’s face that says “go to hell”—It makes it hard for the other person to see your perspective when you do that.

While I appreciated the fun jab and reference to FaceMash, Facebook did not see the humor in the project. The creators received cease and desist letters and were threatened with multiple lawsuits from Facebook’s lawyers. And in a final spiteful measure, Facebook deleted their profiles [4]. Apparently the rule is, what happens on Facebook, stays on Facebook.

Using the API

While I was sure that in the past my app, Give Me My Data, was able to retrieve a user’s friend’s email address, when I tested it after the Facebook Friend Exporter news broke I found no emails. Looking around the web I can’t find any evidence that Facebook ever allowed this data to be accessed. While you can still export all other data from your Facebook profile using Give Me My Data, this tiny and important string of characters with an ‘@’ is one essential component Facebook won’t allow. Likely, the policy was spurred by spam prevention, but given the above, it has the added bonus of blocking an exodus of users from Facebook.

In any case when you run the following FQL (Facebook Query Language) against their API it doesn’t error. This means the field exists, but they have written a custom script to remove it from the results.

SELECT first_name, middle_name, last_name, email
FROM user
WHERE uid IN (SELECT uid2 FROM friend WHERE uid1 = me())

Becoming a Preferred Developer

In the article above, Schonfeld also explains how users can access their Facebook friend’s contact info by first importing it into a Yahoo! account and then exporting a CSV which can be imported into Google+ (or anything for that matter). I believe that Yahoo! belongs to the Facebook Preferred Developer Consultant Program which gives them access above and beyond regular developers:

“Facebook provides PDCs with increased access to its employees and training. PDCs are expected to abide by program expectations around policy compliance, integration quality, and high-level cooperation with Facebook.” [6]

Whatever kind of cooperation it is that Facebook is giving these preferred developers, one can be sure it includes access to data Facebook considers sensitive, like email addresses. While Yahoo! is not listed as a preferred developer on the Facebook page above, they have access to the emails so they clearly have some kind of arrangement.

Google, on the other hand, most definitely does not. This is not the first time Google and Facebook have gotten into a scuffle over sharing (or lack of) data. Late in 2010 Google stated they would no longer allow Facebook and other services access to their users’ data unless Facebook or the other service allowed data to be accessed by Google. [7]

In closing, we already know everyone wants our data. All the clicks, likes, comments, photos, and video we incur or upload are tracked, analyzed, and ultimately compiled and sold to advertisers or others in the business of molding consumer (or political) behavior. We’ve come a long way since Gutenberg, but just like when he was alive, it seems there will always be powerful groups in control of the media of the day. And, even with the utopian promise of a democratic internet, information continues to be manipulated or hidden in order to keep them powerful.

Notes

  1. Schonfeld, Erick, “The Only Backdoor Left To Sneak Your Facebook Friends Into Google+ Is Yahoo,” Jul 5, 2011, http://techcrunch.com/2011/07/05/google-facebook-friends-yahoo/
  2. “Statement of Rights and Responsibilities,” Facebook, last revised April 26, 2011, http://www.facebook.com/terms.php
  3. Protalinski, Emil, “Facebook blocks Google Chrome extension for exporting friends,” July 5, 2011 http://www.zdnet.com/blog/facebook/facebook-blocks-google-chrome-extension-for-exporting-friends/1935
  4. “Angry Victims and Eager Business Partners React to the “Face to Facebook” Art Stunt,” ARTINFO, February 11, 2011 http://www.artinfo.com/news/story/36963/status-update-angry-victims-and-eager-business-partners-react-to-the-face-to-facebook-art-stunt/
  5. Gayathri, Amrutha, “Why Facebook’s Acts of Desperation Are Not Enough to Stop Google+,” International Business Times, July 6, 2011, http://www.ibtimes.com/articles/174946/20110706/google-plus-facebook-chrome-extension-block-facebook-friends-exporter-disable-social-network-yahoo-b.htm
  6. “Facebook Preferred Developer Consultant Program FAQ,” http://developers.facebook.com/preferreddevelopers/#FAQ
  7. Oreskovic, Alexei, “Google bars data from Facebook as rivalry heats up,” Reuters, Nov 5, 2010, http://www.reuters.com/article/2010/11/05/us-google-facebook-idUSTRE6A455420101105

Give Me My Data upgrade: New API, authorization, and data formats

No one would be surprised to learn that almost all of the user-generated content websites use our personal data to sell advertisements. In fact 97% of Google’s revenue comes from advertising.[1] That’s why it’s important these sites provide as much access as possible to the real owners of our data‐us. After all, we put it there and allow them to use it in exchange for the use of their software. Seems like a fair trade if you ask me.

A year and a half ago Facebook didn’t provide any access. That’s why I created Give Me My Data, to help users reclaim and reuse their personal data they put on Facebook.

By giving more agency to users of online systems, Give Me My Data may have already impacted the nature of online application development. In November 2010, almost a year after I launched Give Me My Data, Facebook created their own service for users to export their profile from Facebook as a series of HTML pages. Unlike Give Me My Data, the Facebook service doesn’t allow you to select which data you want or to choose custom formats to export. It also doesn’t give you options for visualization like the custom network graphs that Give Me My Data offers.

I believe their motivation originates in part with my application, likely due to the popularity of Give Me My Data, and points to the potential usefulness of similar apps. While years down the road may reveal many other online systems giving users control over their data, I see this as a positive effect where the content we create, as well as the means to share and manage it, are democratized.

Meanwhile, the above also keeps me hard at work developing the Give Me My Data project. This week I rewrote the program to use Facebook’s new OAuth authorization, which also required rewriting all of the code that fetches the data. Previously it used the REST API which is being deprecated (sometime?) in the future. I also added new data types, fixed the CSV format (which had the rows and columns mixed-up), and added the possibility to export in the JSON data format.

Finally, in the data selector, I distinguished standard data and customized data types. When I say customized, I mean that I’ve written code that mashes together more than one data table and/or addresses a specific question. For example, right now users can select from two types of network graphs and corresponding formats. One describes the user’s relationship to their friends, and the other describes the user’s relationship to their friends, as well as all their friends’ relationships to each other in various graph description languages. This is how I made the network graph image below. I’m also interested in hearing other suggestions for custom queries I might add. The project will be open source on Github soon, so even code contributions will be welcome.

Anyway, please try out the new version. You may have to delete the app from your allowed applications and then re-authorize it if you’ve used it before. As usual, you can provide feedback on the application page, and you can also contact me on Twitter via @givememydata.

[1] “Google Financial Tables for Quarter ending June 30, 2009” Retrieved October 13, 2010

12 apps to track, share, and visualize personal data

When it comes to personal data everyone’s first concern is usually privacy. But a lot of us want to share our data too, with friends, colleagues, and even complete strangers. While numbers have been used for centuries to improve the way we manufacture and do business, using them to quantify our personal lives is a recent phenomenon.

I’ve been thinking about this because one of my goals in creating Give Me My Data was to inspire others to reuse their data, and respond with images and objects they created. But I’m learning if you don’t know a programming language your choices are somewhat scattered and intimidating.

In a recent email exchange with Nicholas Felton, creator of daytum.com and other quality data products, I asked him what other user data sharing and/or visualization web applications he might have encountered while working on daytum.

Included in this article are the three apps he mentioned with my research plus nine additions of my own. All of the apps I mention help users access their own data to track, share, and/or visualize it either by recording it themselves or exporting it from another software. There’s a table at the end of the article to summarize and compare each.

Give Me My Data givememydata.com free

First, to give some context, Give Me My Data is a Facebook application that helps users export their data out of Facebook for reuse in visualizations, archives, or any possible method of digital storytelling. Data can be exported in common formats like CSV, XML, and JSON as well as customized network graph formats.

Status: operational, in-development

Daytum daytum.com free/$$

And to further contextualize, I’ll also address Daytum, an online app that allows users to collect, categorize, and share personal or other data. You can add any data that can be quantified or written down and organize and display it in many forms including bar and pie charts, plain text, and lists. There’s also a mobile site for quick submissions from your device or you can use their iphone app.

Status: operational, but not currently being developed

Geckoboard geckoboard.com $$

Geckoboard is a hosted real-time status board for all sorts of business (or personal) data. You can view web analytics, CRM, support, infrastructure, project management, etc., in one interface, on your computer or smart phone. To see data from other web services in your “dashboard” you add “widgets”—choose from a large list of APIs, give permissions, configure a variety of options, and see your data in a customized graph. Note though, this service is only for presenting data that is hosted elsewhere, and only in this interface. If you like looking at numbers all day, this is for you.

Status: operational

Track-n-Graph trackngraph.com free/$$

Track, graph, and share any information you can think of: your weight, gas mileage, coffee consumption, anything. The design is a little awkward, the graphs don’t display in Chrome or Safari (Mac), and as far as I can tell there’s no API, but the site seems very useful for storing and making simple graphs of your personal data. There are also various “templates” you can reuse to keep track of data like the Workout Tracker, which has fields for gender and age in addition to minutes you worked out, all of which are important in figuring other data (e.g. calories).

Status: operational

your.flowingdata.com your.flowingdata.com/ free

your.flowingdata lets you record your personal data with Twitter. With it you can collect, interact, customize views, and determine privacy by sending private tweets to your account. This project is created by Nathan Yau who writes Flowing Data and studies statistics at UCSD.

Status: operational, in-development

mycrocosm mycro.media.mit.edu free

Mycrocosm is a web service that allows you to track and share data and statistical graphs from the minutiae of daily life. Mycrocosm was developed by Yannick Assogba of the Sociable Media Group of the MIT Media Lab.

Status: operational, but not currently being developed

ManyEyes www-958.ibm.com free

ManyEyes is a project by the IBM Research and the IBM Cognos software group. On Many Eyes you can upload your own data and create visualizations, and view, discuss, and rate other’s visualizations and data sets. It is a great concept but it hasn’t evolved much since its original launch. In fact I’m finding the visualization technology has slowly devolved, leaving only about 20% of visualizations actually displaying (Chrome 12.0 on OSX 10.5.8 if folks are reading).

Status: operational

Fitbit fitbit.com $99.95

The Fitbit is a hardware device which tracks your motions and sleep throughout each day. This data can be uploaded and visualized on their website to realize information about your daily activities like calories burned, steps taken, distance traveled and sleep quality. The Fitbit contains a 3D motion sensor like the one found in the Nintendo Wii and plugs into a base station to upload the data.

Status: operational

Personal Google Search History google.com/history free

When I first saw this application on the Google site I was immediately alarmed. The amount of data they have collected is staggering; for example, “Total Google searches: 36323.” (since Jan 2006) This is a fantastic picture into the life of a user and what they are reading, watching, responding to. It’s like another, admittedly less manicured version, of Facebook. Instead of creating a profile, I am being profiled.

The privacy implications are serious here, which is probably why you have to login again to view it. It is also why a user’s search history draws the interest of interface artist, Johannes P. Osterhof, who is in the process of exploring the line between private and public data, as well as the even further-evaporated division between surveillance and social networks, in his one-year-long search history-made public project, simply titled, Google.

But, as everyone probably already knows, these big companies are making money and providing services. Google has the resources to take your privacy seriously. Well, kind of, because it mostly doesn’t fit into their business model to not track people.

Status: operational

Google Takeout google.com/takeout free

Speaking of funding, I’m quite impressed by this project. Google Takeout is developed by an engineering team at Google called the Data Liberation Front who take their jobs very seriously. In addition to their Google Takeout project, which allows you to export some of your data from Google, they have a really great website with current information about getting access to the data you store with Google.

Status: operational, in-development

gottaFeeling gottafeeling.com free/$$

gottaFeeling is an iphone application that allows you to track and share your feelings. It’s a simple concept, and while loaded down with a lot of rhetoric, reminds me of the amazing, “We Feel Fine.”

Status: in-development

BuzzData buzzdata.com unsure

Finally, I’ll end with BuzzData, a data-publishing platform that encourages the growth of communities around data. Not yet public, I’ve received a private taste of what this app will do, and it looks like it will be pretty cool. Think a mashup between Github and ManyEyes.

Status: still in-development, not public

So I’ll end with the table I created in my research. There are obviously many more types of ways to keep and manage data that I haven’t addressed here, but this is a good start. For further reading check out the Quantified Self blog/user community/conference created by Gary Wolf, who also authored, The Data-Driven Life, the New York Times article linked above.

track/ upload custom data types visualize publish privacy export mobile upload API price limits

Give Me
My Data

yes yes n/a no yes yes n/a no free none

Daytum

yes yes yes yes $$ yes mobile site and iphone app no free / $4/ month free account limited by amount

Geckoboard

no yes yes no yes no n/a only for viewing $9-$200
/ month
number of users

Track-n-Graph

yes yes yes yes yes no web-based no free / $25 per year free account limited by amount

your. flowingdata

yes yes yes yes yes yes via twitter via twitter free none

mycrocosm

yes yes yes yes yes no web-based email-based free none

ManyEyes

yes yes yes yes no yes no no free none

Buzzdata

yes yes yes yes yes yes email ??? ??? ???

Google Web History

yes n/a yes yes yes yes yes no free none

Google Takeout

yes yes n/a n/a yes yes n/a no free none

Fitbit

yes yes yes yes yes yes yes yes $100 / website is free free web account limited by amount

gottaFeeling

yes no no yes yes no iphone no free none
track/upload Can you track or upload your own data?
custom data types Does the a support custom data types?
visualize Can you create visualizations with the app?
publish Can you publish your data with the software?
privacy Are there options for keeping your data private while using the app?
export Can you export the data back out?
mobile upload Are there options to track or upload data from a device?
API Is there an Application Program Interface that allows you to write code to manage data?
price Is there a free version?
limits What limits are imposed on the free version?

Update: Check out Google Guages and other Google Charts.

Semantic network of hierarchical tags from Camp La Jolla Military Park

A semantic network visualization and detail using tags generated from the hierarchical tagging system I created with Thomas Evan Lecklider as part of my Camp La Jolla Military Park project.

For example this item, Defense Contractors recruit at UCSD Job Fair, in the park is filed under: business » arms industry » spending » recruiting

Give Me My Data 2.0 advances to 2nd round of Rhizome Commission voting

I just learned the Give Me My Data 2.0 proposal I put together with Dr. Phil Steinberg has advanced to the second stage of 2011 Rhizome Commission voting. Give Me My Data 2.0 is an open source web application that exports your user data from any crowdsourcing website in multiple, reusable formats.

I created Give Me My Data while working on Automata, a project which visualizes powerful relationships and which also made it to the second round of the Rhizome Commissions two years ago.

Please VOTE for Give Me Me Data if you are a Rhizome member!

Art Meets Radical Openness + Give Me My Data: A Short History

I’ve just returned from Linz, Austria, where I attended the art and technology festival/conference/exhibition called Art Meets Radical Openness (LiWoLi 2011). It is put on and hosted by Ushi Reiter and others of servus.at, and Martin Kaltenbrunner (creator of Reactable) and others at the Kunst Universität Linz.

While I was here I met a lot of friendly and engaging people, and saw some great works, talks, and performances. I also gave a paper about Give Me My Data, titled, “The Self-Indulgence of Closed Systems.” The presentation was about my current research into historic and contemporary modes of surveillance, ensuing privacy issues, and examples of resistance to these modes of data gathering. It was part the Naked on Pluto lecture series, organized by the cheery Marloes de Valk.

I also conducted a 2-day workshop, “Freedom for Our Files: Creative Reuse of Personal Data,” where I shared information and examples on how to go about data scraping and spidering, reuse and visualization, and using the Facebook API for all of the same, for creative or experimental purposes. I’ve posted some of the code and content from FFOF already.

One important motivation for my attendance was to meet programmers who might be interested in working with me on the more technical aspects of upgrading Give Me My Data to the new Facebook Graph API and finding a way around undocumented data limits imposed by Facebook that my users continue encountering.

Give Me My Data: A Short History

I thought I would give a little bit of a background here, about Give Me My Data, for those interested in its development and other related projects.

In 2008 I finished my Masters of Fine Arts at the University of California, San Diego with a thesis project titled Camp La Jolla Military Park. It was a fictional national park which documented the history and ongoing collaboration between the university with the defense industry. I created a custom content management system (PHP, MySQL, Google Maps API) and formed a group of collaborators to enter data in the system documenting relationships. The project exists now as a series of brochures and website which mimics the national park website and describes the research we conducted. I also gave tours of the “park” (campus) where technology was being developed for defense contractors and other sites with related interest.

The following year, while working during my first year as an assistant professor at Florida State University, I described and began developing aspects of a project I temporarily titled Automata. The project, which was a finalist in 2009 for the Rhizome Commission, proposed to use technology to visualize relationships in which there might exist ethical contradictions, but are difficult to discern for all the actors involved. An example of an ethical contradiction might be collaborations between the defense industry and educational institutions which encourage students to develop technology used to harm other human beings. I proposed to do this through various automated web spiders, semantic analysis, and finally visualizations and other interfaces to the data.

While working on Automata, which was named to reference the automation of the documentation Camp La Jolla performed, I was invited to MIT by Chris Csikszentmihályi, Research Scientist at the MediaLab and Director of Center for Future Civic Media at MIT, to discuss the project with members of his Center for Future Civic Media.

Later in 2009, while planning how I might visualize the network data from Automata, rather than creating the content from scratch I decided to use my Facebook contacts as a test subject. Since there was no application which exported data in various reusable formats, especially one which arranged the contacts into a format which described relationships, I created my own.

This application, live since October 2009, would soon became very popular in its own right. In late April 2010, Facebook made a change to the way they allowed users to present information about themselves. Previously entered user data was suddenly aggregated without permission in order to encourage strangers with common interests to form groups. This slight change in policy caused an uproar within the Facebook community because it attempted to force people together without regard to individual privacy.

Many Facebook users were originally attracted to the site because of its levels of privacy and therefore found this change to be manipulative and deeply disturbing. Further, users were no longer able to access or change this information because it had been incorporated into a larger structure. In their article, Facebook App Brings Back Data, May 1, 2010, The New York Times documented how my application allowed users to regain access to this and other information hidden behind the Facebook interface.

At this time, Facebook (still) did not feature a method to export your data for any purpose. In November 2010, likely influenced by Give Me My Data, Facebook finally added a way for users to export their profile data as a series of HTML pages. I believe their motivation originates in part with my application, and points to the effectiveness of such forms of public art / creative resistance to reach out beyond the art world and contribute to society in lasting ways.

What Now?

So in conclusion, I created Give Me My Data while working on projects related in concept and execution. While I have paused my work on Automata you can see some sitemap visualizations from the project in a series called Firing Blind.

As for the status of Give Me My Data, I developed it using the Facebook REST API, which will be deprecated soon. It uses both REST and FQL (Facebook Query Language) calls to help users access a variety of types and formats of their data. I’m working now to put the project on Github, thereby making the code open source, and also to invite others to help to upgrade the data retrieval method to the Facebook Graph API, and to improve existing code that reformats the data from JSON to XML, CSV, GraphML, Dot, and other popular data formats. Now that I have a handful of business cards and new acquaintances I’m looking forward to moving forward. Please get in touch if you are interested in contributing.

Freedom for Our Files: Code and Slides

A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours!

Here are the slides and some of the code samples from the Freedom for Our Files (FFOF) workshop I just did in Linz at Art Meets Radical Openness (LiWoLi 2011).

The first one is a basic scraping demo that uses “find-replace” parsing to change specific words (I’m including examples below the code)

<?php

/*	Basic scraping demo with "find-replace" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$contents = file_get_contents($url);	// 1. get contents of page in a string

					// 2. search and replace contents
$contents = str_replace(		// str_replace(search, replace, string)
			"News",					
			"<b style='background:yellow; color:#000; padding:2px'>LIES</b>",
			$contents);

print $contents;			// 3. print result

?>

Basic scraping demo with “foreach” parsing

<?php

/*	Basic scraping demo with "foreach" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$lines = file($url);			// 1. get contents of url in an array

foreach ($lines as $line_num => $line) 	// 2. loop through each line in page
{		
					// 3. if opening string is found
	if(strpos($line, '<h2 class="top-story-header ">')) 	
	{
		$get_content = true;	// 4. we can start getting content
	}
	
	if($get_content == true)
	{
		$data .= $line . "\n";	// 5. then store content until closing string appears
	}

	if(strpos($line, "</h2>")) 	// 6. if closing HTML element found
	{
		$get_content = false;	// 7. stop getting content
	}
}

print $data;				// 8. print result

?>

Basic scraping demo with “regex” parsing

<?php

/*	Basic scraping demo with "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";		// 0. url to start with

$contents = file_get_contents($url);		// 1. get contents of url in a string
											
						// 2. match title
preg_match('/<title>(.*)<\/title>/i', $contents, $title);

print $title[1];				// 3. print result

?>

Basic scraping demo with “foreach” and “regex” parsing

<?php

/*	Basic scraping demo with "foreach" and "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

// url to start
$url = "http://www.bbc.co.uk/news/";

// get contents of url in an array
$lines = file($url);

// look for the string
foreach ($lines as $line_num => $line) 
{	
	// find opening string
	if(strpos($line, '<h2 class="top-story-header ">')) 
	{
		$get_content = true;
	}
	
	// if opening string is found 
	// then print content until closing string appears
	if($get_content == true) 
	{
		$data .= $line . "\n";
	}

	// closing string
	if(strpos($line, "</h2>")) 
	{
		$get_content = false;
	}
}

// use regular expressions to extract only what we need...

// png, jpg, or gif inside a src="..." or src='...' 
$pattern = "/src=[\"']?([^\"']?.*(png|jpg|gif))[\"']?/i";
preg_match_all($pattern, $data, $images);

// text from link
$pattern = "/(<a.*>)(\w.*)(<.*>)/ismU";
preg_match_all($pattern, $data, $text);

// link
$pattern = "/(href=[\"'])(.*?)([\"'])/i";
preg_match_all($pattern, $data, $link);

/* 
// test if you like
print "<pre>";
print_r($images);
print_r($text);
print_r($link);
print "</pre>";
*/

?>

<html>
<head>
<style> 
body { margin:0; } 
.textblock { position:absolute; top:600px; left:0px; }
span { font:5.0em/1.0em Arial, Helvetica, sans-serif; line-height:normal; 
background:url(trans.png); color:#fff; font-weight:bold; padding:5px } 
a { text-decoration:none; color:#900 }
</style>
</head>
<body>
<img src="<?php print $images[1][0] ?>" height="100%"> </div>
<div class="textblock"><span><a href="<?php print "http://www.bbc.co.uk".$link[2][0] ?>"><?php print $text[2][0] ?></a></span><br>
</div>
</body>
</html>

And the example, which presents the same information in a new way…

Advanced scraping demo with “regex” parsing. Retrieves current weather in any city and colors the background accordingly. The math below for normalization could use some work.

<?php

/*	Advanced scraping demo with "regex" parsing. Retrieves current 
 * 	weather in any city and colors the background accordingly. 
 *	The math below for normalization could use some work.
 *	Owen Mundy Copyright 2011 GNU/GPL */

?>

<html>
<head>
<style> 
body { margin:20; font:1.0em/1.4em Arial, Helvetica, sans-serif; } 
.text { font:10.0em/1.0em Arial, Helvetica, sans-serif; color:#000; font-weight:bold; } 
.navlist { list-style:none; margin:0; position:absolute; top:20px; left:200px }
.navlist li { float:left; margin-right:10px; }
</style>
</head>

<body onLoad="document.f.q.focus();">

<form method="GET" action="<?php print $_SERVER['PHP_SELF']; ?>" name="f">

	<input type="text" name="q" value="<?php print $_GET['q'] ?>" />
	<input type="submit" />

</form>

<ul class="navlist">
	<li><a href="?q=anchorage+alaska">anchorage</a></li>
	<li><a href="?q=toronto+canada">toronto</a></li>
	<li><a href="?q=new+york+ny">nyc</a></li>
	<li><a href="?q=london+uk">london</a></li>
	<li><a href="?q=houston+texas">houston</a></li>
	<li><a href="?q=linz+austria">linz</a></li>
	<li><a href="?q=rome+italy">rome</a></li>
	<li><a href="?q=cairo+egypt">cairo</a></li>
	<li><a href="?q=new+delhi+india">new delhi</a></li>
	<li><a href="?q=mars">mars</a></li>
</ul>

<?php

// make sure the form has been sent
if (isset($_GET['q']))
{
	// get contents of url in an array
	if ($str = file_get_contents('http://www.google.com/search?q=weather+in+'
						. str_replace(" ","+",$_GET['q'])))
	{
		
		// use regular expressions to extract only what we need...
		
		// 1, 2, or 3 digits followed by any version of the degree symbol 
		$pattern = "/[0-9]{1,3}[º°]C/";
		// match the pattern with a C or with an F
		if (preg_match_all($pattern, $str, $data) > 0)
		{
			$scale = "C";
		}
		else
		{
			$pattern = "/[0-9]{1,3}[º°]F/";
			if (preg_match_all($pattern, $str, $data) > 0)
			{
				$scale = "F";
			}
		}
		
		// remove html
		$temp_str = strip_tags($data[0][0]);
		// remove everything except numbers and points
		$temp = ereg_replace("[^0-9..]", "", $temp_str);
		
		if ($temp)
		{
			
			// what is the scale?
			if ($scale == "C"){
				// convert ºC to ºF
				$tempc = $temp;
				$tempf = ($temp*1.8)+32;
			}
			else if ($scale == "F")
			{
				// convert ºF to ºC
				$tempc = ($temp-32)/1.8;
				$tempf = $temp;
			}
			// normalize the number
			$color = round($tempf/140,1)*10;
			// cool -> warm
			// scale -20 to: 120
			$color_scale = array(
					'0,  0,255',
					'0,128,255',
					'0,255,255',
					'0,255,128',
					'0,255,0',
					'128,255,0',
					'255,255,0',
					'255,128,0',
					'255,  0,0'
					);	
		
?>

<style> body { background:rgb(<?php print $color_scale[$color] ?>) }</style>
<div class="text"><?php print round($tempc,1) ."&deg;C " ?></div>
<?php print round($tempf,1) ?>&deg;F

<?php
		
		}
		else 
		{
			print "city not found";	
		}
	}
}
?>

</body>
</html>




For an xpath tutorial check this page.

For the next part of the workshop we used Give Me My Data to export our information from Facebook in order to revisualize it with Nodebox 1.0, a Python IDE similar to Processing.org. Here’s an example:

Update: Some user images from the workshop. Thanks all who joined!

Mutual friends (using Give Me My Data and Graphviz) by Rob Canning

identi.ca network output (starting from my username (claude) with depth 5, rendered to svg with ‘sfdp’ from graphviz) by Claude Heiland-Allen

-->