Tag: networks

Stasi / Facebook / Big Data DAAD Day 13 – The Stasi’s Electronic Databases

The number of cards in the Stasi indexes grew to an incredible 39,000,000 by 1989. In order to provide constant access to this massive material data system they created various special storage units and containers as it expanded. The current mechanical powered card indexes in place at the BTsU were found after the end of the GDR and are still in use in order to meet requests of citizens who wish to view the data the Stasi kept on them.

Stasi / Facebook / Big Data DAAD Day 13 – The Stasi’s Electronic Databases


Mechanical index card file organization system used by the Stasi.

Beginning in the 1960s, the MfS also used various electronic databases to track citizens.* These databases were never fully implemented as a primary source of storage because computing and software available in the GDR at the time, even machines stolen or purchased in the west, were unreliable. For six years the Stasi mirrored essential data from the F16 personal card into an electronic format. In 1975, Department XII, responsible for central data collection, information and storage, introduced the “System of automatic pre-selection” (SAVO) which allowed workers to verify that a person was “not yet recorded” in the system. This considerably reduced the time it took for verification, however it still required a substantial 12–21 days for a process to complete.

By 1986 SAVO 2.0 was introduced providing printing functionality, increased capacity—now containing all data from the F16 card—and reducing search times to 1–6 days. It also provided teletype connections eliminating the need for couriers and allowing multiple agencies (passport control, tourism, etc.) to use the same data source simultaneously. (The Key to Power, 6) While this greatly modernized the surveillance workflow of the Stasi, it continued to act only as a duplicate of the F16 card, which was still faster for priority inquiries, and less susceptible to errors.

Stasi / Facebook / Big Data DAAD Day 13 – The Stasi’s Electronic Databases


A computer used by the Stasi. Source: BBC

The logic for these implementations is echoed in modern database planning and use. While redundancy is employed for backups and increasing access speed across global networks, there is still a “master” copy of all data, or in the case of distributed system, automated processes (software) that merge and maintain distributed data into a master. Like the Stasi’s decision to completely mirror portions of the F16 system, modern systems use whichever implementation is most reliable for the master, while summarizations of that data may be “pushed” to other indexes for faster retrieval.

The Stasi’s use of multiple systems, as well as the F16 and F22 card also preempt the concept of “joins”, which is a powerful feature in relational database management systems (RDBMS), to allow various pieces of data to be spread out across multiple indexes. Like the registration number on the F16/22 cards, records from multiple discrete RDBMS can be “joined” by matching a single identification field in order to supply additional, or limit access to other data, when performing queries.

Multiple other computerized database systems were developed or used by the Stasi, including SOUD, a collection of information on the “enemy” shared between the GDR, Bulgaria, Cuba, Poland, the USSR, Czechoslovakia, Hungary, and Vietnam. The most expansive of all their electronic indexes, was the central database of personal data (ZPDB). Introduced in 1981, by the end it contained records on over 1,320,000 people, 417,000 situations, 558,000 objects, and numerous related information.

The desire to track not just people, but occurrences and objects is directly visible in the largest person tracking organization today. In Facebook’s “Social Graph”, introduced in 2007, any person, place, thing, event, anything that can be named, is recorded in a massive attempt to commodify everything and our connections to it. Like the Stasi ZPDB, everything in the Facebook Social Graph is indexed by a unique identification string as single record, which Facebook considers a “node,” with links or “edges” that connect them. These definitions are based on Facebook’s desire to do data science on the their information, applying statistics and other mathematical operations from graph theory in order to understand their data and predict (and manipulate) the behavior of users.

*Regrettably, all electronic databases and their magnetic storage was destroyed not long after the end of the GDR. Meaning, BStU staff now use the original paper F16 card system to provide services to the public.

  1. The key to power: Card indexes and other finding aids practices of State Security. Archivar 64. Jahrgang Heft 04 November 2011.

Art and the Internet book published by Black Dog Publishing

Here are some shots from the recently released Art and the Internet (Black Dog Publishing, London) with contributions from Joanne McNeil, Domenico Quaranta, and Nick Lambert. The book is a welcome update to writing on the subject and contains many well known works by artists I’ve admired for years. Nice to be included.

Art and the Internet, Black Dog Publishing

Art and the Internet, Black Dog Publishing


Trust (Evidence Locker) (2004) Jill Magid. Essay by Joanne McNeil.

Art and the Internet, Black Dog Publishing


Simple Net Art Diagram by MTAA (1997). Essay by Domenico Quaranta

Art and the Internet, Black Dog Publishing


Colorflip.com (2008) by Rafaël Rozendaal

Art and the Internet, Black Dog Publishing


Every Icon (1997) by John F Simon Jr

Art and the Internet, Black Dog Publishing


Tommy Chat Just Emailed Me (2006) by Ryan Trecartin

Art and the Internet, Black Dog Publishing


1:1 (1999-2002) by Lisa Jevbratt

Art and the Internet, Black Dog Publishing


They Rule (2001) by Josh On

Art and the Internet, Black Dog Publishing


My Generation (2010) by Eva and Franco Mattes

Art and the Internet, Black Dog Publishing


Artist’s Statement No 45, 730,944: The Perfect Artistic Website (2000) Young-Hae Chang Heavy Industries

Art and the Internet, Black Dog Publishing


I am Unable to Fulfill Your Wish (2012) by Owen Mundy

Art and the Internet, Black Dog Publishing

No-one Ever Cried At A Website (Speed Show) Friday, November 22

No-one Ever Cried At A Website (Speed Show) Friday, November 22

No-one Ever Cried At A Website (Speed Show)

November 22, 5:00–6:30 pm EST, Computer training room, Facility for Arts Research, 3216 Sessions Road, Tallahassee FL

With FSU students: Monique Boileau, Alexis Cooper, Jonathan Davito, Danielle English, Justin Greenstein, Antoinette Janus, Scotty Johnson, Melissa Lidsky, Michelle Medrano, Denise Morrow, Lena Weissbrot, Meghan “Red” Yancey. Curated by: Owen Mundy

Students from the Fall 2013 Network Art and Typography classes in the Department of Art at Florida State are staging an exhibition titled No-one Ever Cried At A Website (Speed Show) on November 22 5:00–6:30 pm EST at the computer training room in the Facility for Arts Research, Tallahassee FL.

The exhibition title is modified from an article called, “No-one Ever Cried At A Website,” written by artist/coder Matt Pearson. The document examines how emotion is often forgotten when analyzing technologically-sophisticated works of art such as those which exist on the internet. It reminds readers that painting was once a technology, and asks how beauty, empathy, and interaction can all be triggers for emotional response regardless of the medium for delivery. The prompt for the works in this show, created mostly collaboratively, over the course of 10 days, and specifically for this exhibition, is to address how emotion can be used to engage online audiences to look, listen, and be moved by internet-based art.

Speed Show exhibition, popularized by artist, Aram Bartholl, are arranged as following: “Hit an Internet-cafe (or computer classroom), rent all computers they have and run a show on them for one night. All art works of the participating artists need to be on-line and are shown in a typical browser with standard plug-ins.”

Poster: Print resolution and scaled version

MutaMorphosis: Examining Networks of Power stream today

I am an “attractor” for a stream of interest at a conference that is happening this weekend in Prague. There are thought-provoking presentations lined-up. Check it out if you are nearby.

MutaMorphosis: Examining Networks of Power stream today

Examining Networks of Power
Attractor: Owen Mundy (US)

Addressing the Future: The Tactics of Uncertainty
Attractor: Claudia Westermann (AT)

MutaMorphosis: Tribute To Uncertainty
Saturday, December 8, 2012
13:30-15:30 | Academy 2

Marc Garrett (UK) | Revisiting the Curious World of Art & Hacktivism
Diane Derr (QA) | Morphological Mapping of the Arab Spring through an Ecological Lens
Daphne Dragona (GR) The power of counterpower
Katerina Karoussos (UK) | LUMEN HILARE – The (bio)photonic emission of a church and its metabolism in human consciousness

Owen Mundy: This thread intends to examine representations of contemporary control structures and discuss whether or not an image, particularly network graphs depicting power relationships, information flows, economic activity, etc. have the ability to reverse trends of concentrated power. Does depicting power lead to greater democratization of said power or are we merely creating beautiful images that detract from their original intent—to reveal, examine, and act. To what effect does mapping complex data depicting influential actors, institutions, moments in time, finance systems, mapped onto a two-dimensional surface unravel the power they represent by distributing that power? Methodology: We’ll begin the discussion by looking at historical examples of communication that intend to affect power structures. We’ll examine works by artists and cultural practitioners such as Hans Haacke, Mark Lombardi, Josh On (theyrule.net), Bureau d’études, and more. We’ll consider Manual Castells‘ “Network Theory of Power” within these context(s). Then we’ll discuss visual components of network graphs, asking in particular; what data and
representational forms help make a graph speak to as many people as possible. Finally we’ll work in groups to extend the discussion into possible solutions and various statements and recommendations for such representations.

Claudia Westermann: More than 30 years ago, Michel de Certeau suggested that a shift in methodologies was necessary from strategies that operate on the basis of planning and prediction to tactics that operate on the basis of appropriation. Within this context, competition for the best prediction is substituted for a search for authorities that create the places that give space. This stream of interest with the title ‘Addressing the Future – The Tactics of Uncertainty’ suggests that even though Utopia has not been valued recently, it is nevertheless the actual turn towards the future – this radical uncertainty – that allows for a profound re- questioning of our methodologies. How do the sciences and the arts address the future? Can we speak to it, and in doing so, can we create openings in the present? Methodology: The main intention of this suggested stream of interest is to provide for a ‘good’ start question from which the theme of uncertainty could be addressed in a rather radical way. I am interested in viewpoints from all areas of the sciences and arts. In addition to presentations, there could be conversational panels and workshops. The latter could also be contextualized within the city of Prague. Which inclination this theme takes also depends on the other streams of interest and should be discussed.

PRAGUE | CZECH REPUBLIC | DECEMBER 6-8, 2012 | WWW.MUTAMORPHOSIS.ORG | ORGANIZED BY CIANT | WWW.CIANT.ORG

Project Presentation and Mini-Seminar: Live Project Launch, Workshop Outcomes and Talks on [Social] Media Hacking

Project presentation at WORM

Tonight Tim Schwartz, Walter Langelaar, Birgit Bachler, and I share our new project. The concept has taken a turn from the original plan, but will be exciting nonetheless.

Project Presentation and Mini-Seminar: Live Project Launch, Workshop Outcomes and Talks on [Social] Media Hacking
Friday, July 27, 2012

Tonight we launch the new version of “Give Me My Data!”, a project that together with Artists In Residence Tim C. Schwartz and Owen Mundy was revised and revived in WORM’s moddr_lab.

Give Me My Data offers functionality to its users with which they can retrieve and backup data and files from several key social networks; partly designed as a backup tool moreover targeted at networks that completely lack these functions.

Besides launching the project and am in-depth presentation by the artists, Walter Langelaar of WORM will give an introduction and overview of similar (art)works and earlier projects that came out of WORMs studios like the “Web2.0 Suicide Machine”.

Further more we’ll have presentations of current and ongoing projects from the lab, like Birgit Bachler’s ‘online social gardening’ platform “Talk To The Plant”, the Force of Freedom (Roel Roscam Abbing and Micha Prinsen) present “partsba.se”, and Geert Lovink talks about the “Unlike Us” initiative. The last addition to tonights programme is a presentation by Greenhost.nl on their very excellent RePress project; a WordPress plugin that automagically converts your site to a proxy-server countering censorship on the internet!

In conclusion there will be an open Q&A and panel discussion moderated by Florian Cramer of Creating010.

Projects & Speakers

Florian Cramer – Creating010
Florian Cramer, is a reader and programme director at the applied research center Creating 010 at Hogeschool Rotterdam, The Netherlands. he is a critical writer on arts, culture and information technology. Recent publications include: Exe.cut(up)able statements: Poetische Kalküle und Phantasmen des selbstausführenden Texts, Wilhelm Fink, 201.

Unlike Us / Geert Lovink
The aim of Unlike Us is to establish a research network of artists, designers, scholars, activists and programmers who work on ‘alternatives in social media’. Through workshops, conferences, online dialogues and publications, Unlike Us intends to both analyze the economic and cultural aspects of dominant social media platforms and to propagate the further development and proliferation of alternative, decentralized social media software.

Tim C. Schwartz – moddr_/WORM Artist in Residence
Tim Schwartz grew up in St. Louis, MO. He received a BA in Physics from Wesleyan University and an MFA in Visual Arts from the University of California, San Diego. In January 2010, he developed a technology to help reunited missing people affected by the earthquake in Haiti and now co-runs an organization dealing with family reunification. Last year Schwartz spent four months traveling the country in a mobile research laboratory investigating what is lost as archives become digital.

Birgit Bachler – moddr_/WORM
Birgit is an Austrian artist living and working in Rotterdam/NL. She graduated as BA in Information Design / Media & Interactiondesign at the University of Applied Sciences in Graz/AT and is a recent graduate of the MA Networked Media at Piet Zwart Institute Rotterdam. She has a background in interactive, audiovisual media and programming. Her interests focus on the influence of new media on our everyday lives and the similarities and differences between human and computational behavior.

RePress / Greenhost.nl
“This plugin was made in response to the ongoing limitation of the Open Web. In the dawn of 2012 we found ourselves confronted with a court-ruling blocking the Piratebay.org in the Netherlands. On the other side of the ocean new laws are being discussed to curtail web-freedom even further.”

We zijn pioneer in groene hosting. We ontwikkelden een innovatief energiebesparend hostingplatform waardoor we 70% minder energie gebruiken dan andere hosters. Onze servers staat bij Evoswitch, het meest duurzame datacenter van Nederland.

Owen Mundy – moddr_/WORM Artist in Residence
Owen Mundy is an artist, designer, and programmer who investigates public space and its relationship to data. His artwork highlights inconspicuous trends and offers tools to make hackers out of everyday users. He has an MFA in Visual Art from the University of California, San Diego and is an Assistant Professor of Art at Florida State University.

partsba.se / Force Of Freedom
At partsba.se you can upload, share and download digital designs for real physical products. Partsba.se allows you to share designs of any nature, whether these designs are copyrighted or dangerous. Unlike other websites partsba.se does not claim any rights of your designs once you upload them.In the near future partsba.se will run on a fully secure and anonymous server.

We believe that users should be free to reverse engineer any everyday objects that surround them. Either to improve these objects, customize them, repair them or just to understand them.

The Force Of Freedom is a Rotterdam based collective founded by Micha Prinsen and Roel Roscam Abbing in 2009. Researching ways in which we can relate to things that happen on-line.

Kaarten
De volgende kaarten zijn beschikbaar:
Normaal Voorverkoop: € 5.00 Deurverkoop: € 5.00

Locatie
WORM
Boomgaardsstraat 71
3012 XA Rotterdam

Call for Abstracts: MutaMorphosis: Tribute to Uncertainty

MutaMorphosis: Tribute to Uncertainty conference
December 6-8, 2012, Prague, Czech Republic.

Call for Abstracts
Deadline: July 1, 2012

Detail of I am Unable to Fulfill Your Wish, 33 x 33 in.

Examining Networks of Power
By Owen Mundy

This thread intends to examine representations of contemporary control structures and discuss whether or not an image, particularly network graphs depicting power relationships, information flows, economic activity, etc. have the ability to reverse trends of concentrated power. Does depicting power lead to greater democratization of said power or are we merely creating beautiful images that detract from their original intent—to reveal, examine, and act. To what effect does mapping complex data depicting influential actors, institutions, moments in time, finance systems, mapped onto a two-dimensional surface unravel the power they represent by distributing that power?

Methodology: We’ll begin the discussion by looking at historical examples of communication that intend to affect power structures. We’ll examine works by artists and cultural practitioners such as Hans Haacke, Mark Lombardi, Josh On (theyrule.net), Bureau d’études, and more. We’ll consider Manual Castells‘ “Network Theory of Power” within these context(s). Then we’ll discuss visual components of network graphs, asking in particular; what data and representational forms help make a graph speak to as many people as possible. Finally we’ll work in groups to extend the discussion into possible solutions and various statements and recommendations for such representations.

Keywords: Networks; Power; Data; Politics

We invite you to respond to the Tribute to Uncertainty theme and/or to address one of the 21 streams of interest led by 28 conference Attractors.

Do you have something original to say about our world that is increasingly fuzzy, unstable and chaotic? Are you interested in how crisis, uncertainty and complexity can come together in order to question the known as well as predict and/or model yet unknown? Do you want to share projects intrinsically linking domains of scientific, artistic and technological research and creativity that can be introduced as relevant tools for better understanding of our common future?

The aim of this Call for Abstracts is to provide opportunity for community of interdisciplinary practitioners to get engaged in a transformative conversation on mutant futures.

Feel free to address conference Attractors and respond to a specific stream of interest in order to get involved with other interested individuals and collectives in structured collaborative efforts before, during, and hopefully also after the conference. Research groups formed by the Attractors -who serve as the Reviewers- will be expected to elaborate on the general theme Tribute to Uncertainty while turning attention to concrete research objectives as defined by the streams.

The results should be curated panels, roundtables, workshops as well as publications.

For more information about MutaMorphosis conference (planned publications, preparatory committee, partnerships) please visit mutamorphosis.org.

Freedom for Our Files: Canvas starter (Facebook) app

I’m happy to share the code from a Facebook app I created for a workshop earlier this year.

This simple Facebook canvas application was originally demoed during the Freedom for Our Files workshop at the 2011 Art Meets Radical Openness festival in Linz, Austria. You can view and download the source code on github

The application is very simple; it creates a Facebook object, performs calls to get data belonging to the current user, then prints the data exactly as it is returned. It has examples of basic Facebook Graph API calls as well as an example of FQL (Facebook Query Language).

Freedom for Our Files: Canvas starter (Facebook) app

Internet service just got creepy: How to set up a wireless router with Comcast internet service

I just moved back to Florida after a one year research project in Berlin and have subscribed to Comcast broadband service. The whole experience left a bad taste in my mouth, though not because the tech showed-up 2 hours after the installation appointment window. Nor was it because he held loud personal conversations on his cell phone while he was setting up the service. No, the icky feeling is more corporate and selfish, and impedes much more into my private space than “Joe the cable guy” ever could.

Comcast made me install software on my computer in order to use their broadband.

Upon his arrival, “Joe” announced he would need access to my computer to setup broadband service. Understanding that most of the people Joe deals with might not be IT whizzes, and could manage to not be able to connect their machines without his help, I decided to let him use it rather than attempt to prove I was not a member of the usual group. After half an hour of complaining about previous customers to his friend on his cellphone, waiting for an other Comcast person to flip a switch allowing him to do his job, and multiple trips to his truck, he showed me that the internet was indeed accessible on my computer.

At this point the laptop was directly connected to the cable modem via an ethernet cable. He announced I was to follow the steps on the screen and he was out the door. The web page he had left up required me to agree to some terms, create a username and then… install software? Really? I tried to access the net without the final step but nothing doing. Unless I installed this software I was stuck. So I did it, still not believing that a company had really initiated this final invasion onto every customer’s computer. After it was done I had new bookmarks everywhere, for Comcast email, security, and some branding nonsense called “XFINITY” (I thought “X” was out with the ’90’s and “X”games?)

Comcast icons

So I thought, “OK, Comcast, you got me, hit me with your best marketing slime. Whatever, I can delete the bookmarklets you installed in my browser, just let me access the service I paid for, wirelessly, on whichever device I want.”

But this is where the relationship got really creepy. Apparently when I installed the Comcast (spyware?) on my machine, it made note of my MAC address, a unique identifier of networked machines, so that it would only allow my machine (or another machine with that MAC address) to connect to the internet. This means when I attached a wireless router to the cable modem I could connect to the wifi, but there was no internet.

Comcast welcome page

So it turns-out that Comcast is not only forcing their adware on customers, it’s also making it difficult (though not impossible) for them use more than one device. Presumably Comcast is doing this in order to circumvent sharing of services among neighbors, but the end result is that you can’t share the service between more than one device, or between roommates or spouses for that matter.

An example (albeit a geeky one): between my wife and I we have 2 laptops, 2 smartphones, and a desktop computer that all might be talking to each other or accessing the net. Comcast’s so-called internet service didn’t allow for any such geekery because it only allows one device, with the correct MAC address, to connect.

So, here’s what I did, on my Mac, with some help from my sister’s boyfriend, Tom, and a lot from Google, to get my linksys wireless router to work with Comcast internet.

  1. Confirm you can access the internet with your machine connected directly to the Comcast cable modem.
  2. Open Terminal and type (without the quotes): “ifconfig en0 | grep ether”
  3. Now disconnect your computer from the modem and connect the modem ethernet cable to your wireless router. Make sure both are plugged-in.
  4. Connect to your wireless router via the airport on your machine.
  5. Go to the following link: http://192.168.1.1
  6. Under Setup, choose DHCP as the Internet Connection Type. Save Settings.
  7. Under Setup : Mac Address Clone, enter the alpha numeric characters returned from Terminal. Save Settings.
  8. Configure your wireless router like you normally would and you are up and running.
  9. Snicker at Comcast

Linksys UI

Give Me My Data upgrade: New API, authorization, and data formats

Picture 28

No one would be surprised to learn that almost all of the user-generated content websites use our personal data to sell advertisements. In fact 97% of Google’s revenue comes from advertising.[1] That’s why it’s important these sites provide as much access as possible to the real owners of our data‐us. After all, we put it there and allow them to use it in exchange for the use of their software. Seems like a fair trade if you ask me.

A year and a half ago Facebook didn’t provide any access. That’s why I created Give Me My Data, to help users reclaim and reuse their personal data they put on Facebook.

By giving more agency to users of online systems, Give Me My Data may have already impacted the nature of online application development. In November 2010, almost a year after I launched Give Me My Data, Facebook created their own service for users to export their profile from Facebook as a series of HTML pages. Unlike Give Me My Data, the Facebook service doesn’t allow you to select which data you want or to choose custom formats to export. It also doesn’t give you options for visualization like the custom network graphs that Give Me My Data offers.

I believe their motivation originates in part with my application, likely due to the popularity of Give Me My Data, and points to the potential usefulness of similar apps. While years down the road may reveal many other online systems giving users control over their data, I see this as a positive effect where the content we create, as well as the means to share and manage it, are democratized.

Meanwhile, the above also keeps me hard at work developing the Give Me My Data project. This week I rewrote the program to use Facebook’s new OAuth authorization, which also required rewriting all of the code that fetches the data. Previously it used the REST API which is being deprecated (sometime?) in the future. I also added new data types, fixed the CSV format (which had the rows and columns mixed-up), and added the possibility to export in the JSON data format.

Finally, in the data selector, I distinguished standard data and customized data types. When I say customized, I mean that I’ve written code that mashes together more than one data table and/or addresses a specific question. For example, right now users can select from two types of network graphs and corresponding formats. One describes the user’s relationship to their friends, and the other describes the user’s relationship to their friends, as well as all their friends’ relationships to each other in various graph description languages. This is how I made the network graph image below. I’m also interested in hearing other suggestions for custom queries I might add. The project will be open source on Github soon, so even code contributions will be welcome.

Anyway, please try out the new version. You may have to delete the app from your allowed applications and then re-authorize it if you’ve used it before. As usual, you can provide feedback on the application page, and you can also contact me on Twitter via @givememydata.

Facebook mutual friends network graph

[1] “Google Financial Tables for Quarter ending June 30, 2009” Retrieved October 13, 2010

Freedom for Our Files: Code and Slides

A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours!

Here are the slides and some of the code samples from the Freedom for Our Files (FFOF) workshop I just did in Linz at Art Meets Radical Openness (LiWoLi 2011).


Download the slides (PDF)

The first one is a basic scraping demo that uses “find-replace” parsing to change specific words (I’m including examples below the code)

<?php

/*	Basic scraping demo with "find-replace" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$contents = file_get_contents($url);	// 1. get contents of page in a string

					// 2. search and replace contents
$contents = str_replace(		// str_replace(search, replace, string)
			"News",					
			"<b style='background:yellow; color:#000; padding:2px'>LIES</b>",
			$contents);

print $contents;			// 3. print result

?>

Freedom for Our Files: Code and Slides

Web scraper code for the BBC website using find and replace

Basic scraping demo with “foreach” parsing

<?php

/*	Basic scraping demo with "foreach" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$lines = file($url);			// 1. get contents of url in an array

foreach ($lines as $line_num => $line) 	// 2. loop through each line in page
{		
					// 3. if opening string is found
	if(strpos($line, '<h2 class="top-story-header ">')) 	
	{
		$get_content = true;	// 4. we can start getting content
	}
	
	if($get_content == true)
	{
		$data .= $line . "\n";	// 5. then store content until closing string appears
	}

	if(strpos($line, "</h2>")) 	// 6. if closing HTML element found
	{
		$get_content = false;	// 7. stop getting content
	}
}

print $data;				// 8. print result

?>

Web scraper code for the BBC website using a foreach loop

Basic scraping demo with “regex” parsing

<?php

/*	Basic scraping demo with "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";		// 0. url to start with

$contents = file_get_contents($url);		// 1. get contents of url in a string
											
						// 2. match title
preg_match('/<title>(.*)<\/title>/i', $contents, $title);

print $title[1];				// 3. print result

?>

Web scraper code for the BBC website using a regular expression to find the title

Basic scraping demo with “foreach” and “regex” parsing

<?php

/*	Basic scraping demo with "foreach" and "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

// url to start
$url = "http://www.bbc.co.uk/news/";

// get contents of url in an array
$lines = file($url);

// look for the string
foreach ($lines as $line_num => $line) 
{	
	// find opening string
	if(strpos($line, '<h2 class="top-story-header ">')) 
	{
		$get_content = true;
	}
	
	// if opening string is found 
	// then print content until closing string appears
	if($get_content == true) 
	{
		$data .= $line . "\n";
	}

	// closing string
	if(strpos($line, "</h2>")) 
	{
		$get_content = false;
	}
}

// use regular expressions to extract only what we need...

// png, jpg, or gif inside a src="..." or src='...' 
$pattern = "/src=[\"']?([^\"']?.*(png|jpg|gif))[\"']?/i";
preg_match_all($pattern, $data, $images);

// text from link
$pattern = "/(<a.*>)(\w.*)(<.*>)/ismU";
preg_match_all($pattern, $data, $text);

// link
$pattern = "/(href=[\"'])(.*?)([\"'])/i";
preg_match_all($pattern, $data, $link);

/* 
// test if you like
print "<pre>";
print_r($images);
print_r($text);
print_r($link);
print "</pre>";
*/

?>

<html>
<head>
<style> 
body { margin:0; } 
.textblock { position:absolute; top:600px; left:0px; }
span { font:5.0em/1.0em Arial, Helvetica, sans-serif; line-height:normal; 
background:url(trans.png); color:#fff; font-weight:bold; padding:5px } 
a { text-decoration:none; color:#900 }
</style>
</head>
<body>
<img src="<?php print $images[1][0] ?>" height="100%"> </div>
<div class="textblock"><span><a href="<?php print "http://www.bbc.co.uk".$link[2][0] ?>"><?php print $text[2][0] ?></a></span><br>
</div>
</body>
</html>

And the example, which presents the same information in a new way…

Web scraper code for the BBC website using a regular expression

Advanced scraping demo with “regex” parsing. Retrieves current weather in any city and colors the background accordingly. The math below for normalization could use some work.

<?php

/*	Advanced scraping demo with "regex" parsing. Retrieves current 
 * 	weather in any city and colors the background accordingly. 
 *	The math below for normalization could use some work.
 *	Owen Mundy Copyright 2011 GNU/GPL */

?>

<html>
<head>
<style> 
body { margin:20; font:1.0em/1.4em Arial, Helvetica, sans-serif; } 
.text { font:10.0em/1.0em Arial, Helvetica, sans-serif; color:#000; font-weight:bold; } 
.navlist { list-style:none; margin:0; position:absolute; top:20px; left:200px }
.navlist li { float:left; margin-right:10px; }
</style>
</head>

<body onLoad="document.f.q.focus();">

<form method="GET" action="<?php print $_SERVER['PHP_SELF']; ?>" name="f">

	<input type="text" name="q" value="<?php print $_GET['q'] ?>" />
	<input type="submit" />

</form>

<ul class="navlist">
	<li><a href="?q=anchorage+alaska">anchorage</a></li>
	<li><a href="?q=toronto+canada">toronto</a></li>
	<li><a href="?q=new+york+ny">nyc</a></li>
	<li><a href="?q=london+uk">london</a></li>
	<li><a href="?q=houston+texas">houston</a></li>
	<li><a href="?q=linz+austria">linz</a></li>
	<li><a href="?q=rome+italy">rome</a></li>
	<li><a href="?q=cairo+egypt">cairo</a></li>
	<li><a href="?q=new+delhi+india">new delhi</a></li>
	<li><a href="?q=mars">mars</a></li>
</ul>

<?php

// make sure the form has been sent
if (isset($_GET['q']))
{
	// get contents of url in an array
	if ($str = file_get_contents('http://www.google.com/search?q=weather+in+'
						. str_replace(" ","+",$_GET['q'])))
	{
		
		// use regular expressions to extract only what we need...
		
		// 1, 2, or 3 digits followed by any version of the degree symbol 
		$pattern = "/[0-9]{1,3}[º°]C/";
		// match the pattern with a C or with an F
		if (preg_match_all($pattern, $str, $data) > 0)
		{
			$scale = "C";
		}
		else
		{
			$pattern = "/[0-9]{1,3}[º°]F/";
			if (preg_match_all($pattern, $str, $data) > 0)
			{
				$scale = "F";
			}
		}
		
		// remove html
		$temp_str = strip_tags($data[0][0]);
		// remove everything except numbers and points
		$temp = ereg_replace("[^0-9..]", "", $temp_str);
		
		if ($temp)
		{
			
			// what is the scale?
			if ($scale == "C"){
				// convert ºC to ºF
				$tempc = $temp;
				$tempf = ($temp*1.8)+32;
			}
			else if ($scale == "F")
			{
				// convert ºF to ºC
				$tempc = ($temp-32)/1.8;
				$tempf = $temp;
			}
			// normalize the number
			$color = round($tempf/140,1)*10;
			// cool -> warm
			// scale -20 to: 120
			$color_scale = array(
					'0,  0,255',
					'0,128,255',
					'0,255,255',
					'0,255,128',
					'0,255,0',
					'128,255,0',
					'255,255,0',
					'255,128,0',
					'255,  0,0'
					);	
		
?>

<style> body { background:rgb(<?php print $color_scale[$color] ?>) }</style>
<div class="text"><?php print round($tempc,1) ."&deg;C " ?></div>
<?php print round($tempf,1) ?>&deg;F

<?php
		
		}
		else 
		{
			print "city not found";	
		}
	}
}
?>

</body>
</html>

Freedom for Our Files: Code and Slides

Freedom for Our Files: Code and Slides

Freedom for Our Files: Code and Slides

Freedom for Our Files: Code and Slides

For an xpath tutorial check this page.

For the next part of the workshop we used Give Me My Data to export our information from Facebook in order to revisualize it with Nodebox 1.0, a Python IDE similar to Processing.org. Here’s an example:

NodeBox

Update: Some user images from the workshop. Thanks all who joined!

Mutual friends network graph, cropped

Mutual friends network graph

Mutual friends (using Give Me My Data and Graphviz) by Rob Canning

Identica network graph drawn with Graphviz sfdp

identi.ca network output (starting from my username (claude) with depth 5, rendered to svg with ‘sfdp’ from graphviz) by Claude Heiland-Allen

Freedom for Our Files workshop

Freedom for Our Files workshop

Network graph grouping: A small art world

This “Mutual friends network graph” created with Nodebox using data I exported with Give Me My Data contains 540 “Facebook friends” and their connections to each other. When the graph renders it attempts to position people who have lots of connections closer together. With this you can see groups unfold based on your own social networks. Since I have spent more time in academia than I have at specific jobs my “clusters” are based mostly on my academic history.

You can also see that there are a lot of connections between my high school and where I did my undergraduate study, which is based on the fact they are located very close to each other, so friends from high school also chose the same university or town to live in. There are also a lot of interconnections between Indiana University where I did my undergrad, the University of California, San Diego, where I did graduate study, and Florida State University, where I teach now. This is probably due to the fact that my connections are all within a given field, in my case visual arts, and points to the often expressed notion that “the art world is actually very small.”

Facebook mutual friends network graph, grouped into clusters

Reading list for August 2010

About to embark on some new projects here in Berlin. Here’s my reading list at the moment…

Book cover of Free

Free: The Future of a Radical Price
by Chris Anderson
July 7th 2009 by Hyperion

Traditional economics operates under fundamental assumptions of scarcity–there’s only so much oil, iron, and gold in the world. But the online economy is built upon three cornerstones: processing power, hard drive storage, and bandwidth–and the costs of all these elements are trending toward zero at an incredible rate.

The Exploit

The Exploit: A Theory of Networks
by Alexander R. Galloway, Eugene Thacker
October 1st 2007 by Univ Of Minnesota Press

“The Exploit is that rare thing: a book with a clear grasp of how networks operate that also understands the political implications of this emerging form of power. It cuts through the nonsense about how ‘free’ and ‘democratic’ networks supposedly are, and it offers a rich analysis of how network protocols create a new kind of control. Essential reading for all theorists, artists, activists, techheads, and hackers of the Net.” —McKenzie Wark, author of A Hacker Manifesto

Group Work book cover

Group Work
by Temporary Services
New York, NY: Printed Matter. 2007

Based on a pamphlet published by Temporary Services in 2002 titled Group Work: A Compilation of Quotes About Collaboration from a Variety of Sources and Practices, this publication provides a multitude of perspectives on the theme of Group Work by practitioners of artistic group practice from 1960s to the present.

Facebook Death Star

Here are two images I created with NodeBox (Python) and PHP (data via facebook API).

Facebook network graph drawn as a large circle


All friends, one common link.

Facebook mutual friends network graph


All the mutual connections I could gather before the script timed-out. Need to make the gathering more efficient.