Tag: PHP

Convert a PHP/MySQL site to Jamstack and host on Github Pages

One of the first items on my sabbatical to do list last year was to convert the Camp La Jolla Military Park website to a static Jamstack site hosted at Github Pages. This post documents that process.

I originally created the site as a research tool using PHP and a MySQL database for data entry, to record, organize, and ultimately publish research about the military history of UCSD. Moving the project to Github Pages would simplify maintenance and hosting costs, but was no small task. Technically it didn’t cost me extra to host, but the domain was $20/year, which adds up. Further, maintaining a PHP/MySQL website with a custom admin tool requires a bit of overhead for security and updates. Additionally, I wanted to continue having the benefit of using template engines or some way to include files (like PHP’s require()) to save time coding the site.

There are a plethora of tools available to publish a database-driven site on a static host. They mostly can be categorized as follows (ordered by most to least dynamic data sources):

  • Move the database to a remote service like Firebase, Atlas, et al. — ⚠️ Ultimately creates vendor lock-in, potential hidden costs, and probably more maintenance issues in the future.
  • Create a static version of the database by converting MySQL to flat files (JSON, CSV, etc.), which still allows for querying using one of various tools — 🤔 This method is interesting, because it still decouples the data from the presentation and allows for potentially more options later, including a remote database. I am already using this in various forms but worry again about maintenance, or issues with the client’s ability to manipulate the data as needed.
  • Generate a static version of the site, including all its pages which contain the data formerly dynamically inserted from MySQL. — 🤔 I like this concept, and have been using it to auto-publish my class lectures and tutorials using Markdown and Grunt with Marp. Plus, there are many site generators to choose from, which I explore below.
  • Scrape the site and save pages as static files. After all, everything is static once the HTML is rendered on the frontend. — ⚠️ While fast, this requires the site be built first and would make updates difficult.

Essential Concepts

Here I outline some concepts on rendering. See SSR vs CSR vs ISR vs SSG for a more in-depth look:

  • Client-Side Rendering (CSR) (e.g. Single-Page Application (SPA)) – JS renders pages in client’s browser with empty HTML files + raw data from the server. (e.g. React, Vue, Angular)
  • Server-Side Rendering (SSR) (e.g. Fullstack) – Generates pages on server with dynamic data before delivering to client. Improves SEO and loading, decreases server performance. (e.g. EJS, PHP, or Next.js)
  • Static Site Generation (SSG) (e.g. JAMStack) – Pre-renders all pages as static HTML during build process. Fast loading times, enhanced security, and dynamic content can still be achieved via “islands”.
  • Incremental Static Regeneration (ISR) – Combines SSR and SSG to pre-render static pages during build time while periodically regenerating specific pages with updated data.
  • Edge-Side Rendering (ESR) – Rendering process is pushed to the CDN’s edge servers which handles requests and generates HTML. Requires Nitro engine.

Requirements

Requirements for most of the projects I mention above.

  1. Javascript-based
  2. The data will be static, but editable using Git
  3. The data should be searchable using a tool like chosen
  4. The site design will be simple, but I should have creative control
  5. The site will be public, and be presented mostly using the same design as the original site
  6. HTML/CSS validation + Accessibility checks
  7. Convert from Google Maps to Leaflet
  8. Host the site on Github Pages

While planning the project I also wanted to select a method I could use for other potential static (or nearly static) sites like:

  1. A new site documenting the hundreds of examples and overlapping categories of Critical Web Design and internet art I’ve been collecting for an upcoming book. Update: I built it with SvelteKit
  2. Converting tallysavestheinternet.com to a static site (currently SSR) and unlinking the extension from the server to use local data only
  3. Moving away from WordPress more generally (and its security problems) and blogging with Markdown instead, starting with this blog (NO MORE TYPING IN A WEB FORM!!)

It’s nice to learn new tools. I already have some experiences with React, Next.js, and Vue.js, but wouldn’t say I’m confident in any. I found that publishing apps using React Native is very frustrating and unreliable, and my general distrust of Facebook makes me lean towards Vue. Still, I went in with with an open attitude.

Tool Comparison

A breakdown of some tools I considered. Some of the icons are clickable…

Framework SPA SSR SSG MD Themes Search Cons
Jekyll ❌ ❌ ✅ ✅ ✅ ? ❌ Ruby
Hugo ❌ ❌ ✅ ✅ ✅ ? ❌ Go
Astro ✅ ✅ ✅ ✅ ✅ ✅ React, Vue, Svelte in dynamic islands, simple routing
Vue.js ✅ ✅ ✅ ✅ ✅ ? ✅ vuepress
Nuxt.js ✅ ✅ ✅ ✅ ✅ ? Built on Vue
SvelteKit ✅ ✅ ✅ ✅ ✅ ✅ MD rendering not native but supported via MDX
Next.js ✅ ✅ ✅ ✅ ✅ ✅ MD rendering not native but supported via MDX

Next

My experience with Next.js SSG was painful.

  • Constantly running into subtle changes in syntax across versions
  • Adding an image is ridiculously difficult.
  • Exporting the SSG ultimately created dead links, packaged up in Next code that was hard to understand.
  • So much automation to make a static site in the end you can’t do anything unless you have the exact right code, in the right version, in the right paths.
  • Error messages are rarely helpful or pinpoint the line number causing the issue in the stack.
  • I followed so many tutorials that ultimately ended in dead-ends, incorrect versions, or that completely didn’t work.

Astro

I ended up choosing Astro for the final project, which can be seen here https://omundy.github.io/camplajolla/. While they appear to have many similarities, I found Astro soooooo much easier than Next.js. It’s fairly simple to install, even using a starter theme, and it didn’t take long torip out tailwind to add bootstrap.

Term vs. Term for Digital Public Library of America hackathon

I made a small app to compare the number of search results for two phrases from the Digital Public Library of America for a hackathon / workshop here at Florida State next week.

http://owenmundy.com/work/term-vs-term

dpla term vs term

Digital Humanities Hackathon II – Digital Public Library of America

Monday, April 21, 2:00-3:30 p.m.
Strozier Library, Scholars Commons Instructional Classroom [MAP]

The Digital Scholars Reading and Discussion Group will simulate its second “hackathon” on April 21, allowing participants to learn more about the back-end structure of the Digital Public Library of America. With its April 2013 launch, the DPLA became the first all-digital library that aggregates metadata from collections across the country, making them available from a single point of access. The DPLA describes itself as a freely available, web-based platform for digitized cultural heritage projects as well as a portal that connects students, teachers, scholars, and the public to library resources occurring on other platforms.

From a critical point of view, the DPLA simultaneously relies on and disrupts the principles of location and containment, making its infrastructure somewhat interesting to observe.

In this session, we will visit the DPLA’s Application Programming Interface (API) codex to observe some of the standards that contributed to its construction. We will consider how APIs function, how and why to use them, and who might access their metadata and for what purposes. For those completely unfamiliar with APIs, this session will serve as a useful introduction, as well as a demonstration of why a digital library might also want to serve as an online portal. For those more familiar with APIs, this session will serve as an opportunity to try on different tasks using the metadata that the DPLA aggregates from collections across the country.

At this particular session, we are pleased to be joined by Owen Mundy from FSU Department of Art and Richard Urban from FSU College of Communication and Information, who have considered different aspects of working with APIs for projects such as the DPLA, including visualization and graphics scripting, and developing collections dashboards.

As before, the session is designed with a low barrier of entry in mind, so participants should not worry if they do not have programming expertise or are still learning the vocabulary associated with open-source projects. We come together to learn together, and all levels of skill are accommodated, as are all attitudes and leanings.

Participants are encouraged to explore the Digital Public Library of America site prior to our meeting and to familiarize themselves with the history of the project. Laptops will be available for checkout, but attendees are encouraged to bring their own.

Packet Switching project: ColladaFragmenter software, Kassel, Germany and University of Florida Public Commission

Joelle Dietrick and I embarked on a new body of work this summer called “Packet Switching.” Inspired by her Sherwin Series images and wall paintings, and my work deconstructing and re-visualizing source code and other data, we’ve created two new software projects, as well as a series of limited edition prints, large photo installations, wall-sized paintings, and animations.

The full statement explains our process and intent clearly:

Packet Switching is an ongoing body of work by Joelle Dietrick and Owen Mundy that visualizes architecture as fragments affected by economic and communications systems.

The title of the series references how contemporary communications systems break digital files into smaller manageable blocks of data called packets. Each packet is then sent through a network, taking the quickest route possible, and reassembled once they reach their destination. One JPG image, for example, might be broken into several packets, each of which may travel a different path through the net, even through different cities, before being recompiled into a copy of the original file.

To reference this common process used in networked systems, we wrote custom software that deconstructs a 3D model’s source code and produces unique fragments. We further remixed these fragments using an original application created in Processing. The resulting images become limited edition prints, large photo installations, wall-sized paintings, and animations.

Our process underscores how incidental fragmentation and automation can streamline markets, but also make them vulnerable to systems failure. The use of architecture specifically points to recent real estate market volatility and considers how communication technology-enabled pursuits of profit margins alters our most basic needs.

The first software, that “deconstructs a 3D model’s source code and produces unique fragments,” is open source and available on Github. Essentially, the PHP software, parses a 3D COLLADA file and exports a set number of geometries, that can then be further broken down and used in an artwork or design.

The second software, which we will release soon, remixes these fragments using Processing. The video below shows an example of the whole process.

Wall painting at “Temporary Home” in Kassel, Germany

While artists-in-residence at Temporary Home, in Kassel, Germany, which coincided with Documenta13, Joelle Dietrick and I completed a wall-sized temporary painting based on the architecture from the Bauhaus School at Dessau and 2012 American color forecasts.

Commission at Weimer Hall at the University of Florida

Joelle and I have also received a commission to complete Packet Switching (Weimer Hall) at the University of Florida College of Journalism and Communications this fall. This will be inkjet on adhesive polyester on a large wall (approx. 177.5 ft. x 20.2 ft.). More details soon.

Give Me My Data API Hacking Masterclass

Give Me My Data API Hacking Masterclass
Thursday, July 27, 2012

A ‘hands-on’ workshop with technical and theoretical overview of contemporary ‘Application Programming Interfaces’ (API’s) of large social networks, en how to use these for your own project or application.

We will cover the ins and outs of creating Facebook apps, web applications, and how to play with the Twitter, Foursquare, Flickr and Instagram APIs.

With WORM’s current ‘Artists In Residence’ Tim C. Schwartz and Owen Mundy you’ll learn from a programmers perspective how to approach your target network, to subsequently make it do what you need it to…

As an example you can think of Owen’s project “Give Me My Data” or WORM’s previous release of the “Web2.0 Suicide Machine”

No specific technical skills required, but a curious mind towards the ins&outs of social networking is a must!

[EN] buy your ticket online or send your resrvation tomoddr@worm.org,

Participation is limited so act fast!
– reduced/discount tickets available for WORM volunteers & students (with ID)

Normaal 
Voorverkoop: € 15.00
Deurverkoop: € 15.00
Normaal reductie
Voorverkoop: € 10.00
Deurverkoop: € 10.00

Location: WORM, Boomgaardsstraat 71, 3012 XA Rotterdam

Give Me My Data upgrade: New API, authorization, and data formats

No one would be surprised to learn that almost all of the user-generated content websites use our personal data to sell advertisements. In fact 97% of Google’s revenue comes from advertising.[1] That’s why it’s important these sites provide as much access as possible to the real owners of our data‐us. After all, we put it there and allow them to use it in exchange for the use of their software. Seems like a fair trade if you ask me.

A year and a half ago Facebook didn’t provide any access. That’s why I created Give Me My Data, to help users reclaim and reuse their personal data they put on Facebook.

By giving more agency to users of online systems, Give Me My Data may have already impacted the nature of online application development. In November 2010, almost a year after I launched Give Me My Data, Facebook created their own service for users to export their profile from Facebook as a series of HTML pages. Unlike Give Me My Data, the Facebook service doesn’t allow you to select which data you want or to choose custom formats to export. It also doesn’t give you options for visualization like the custom network graphs that Give Me My Data offers.

I believe their motivation originates in part with my application, likely due to the popularity of Give Me My Data, and points to the potential usefulness of similar apps. While years down the road may reveal many other online systems giving users control over their data, I see this as a positive effect where the content we create, as well as the means to share and manage it, are democratized.

Meanwhile, the above also keeps me hard at work developing the Give Me My Data project. This week I rewrote the program to use Facebook’s new OAuth authorization, which also required rewriting all of the code that fetches the data. Previously it used the REST API which is being deprecated (sometime?) in the future. I also added new data types, fixed the CSV format (which had the rows and columns mixed-up), and added the possibility to export in the JSON data format.

Finally, in the data selector, I distinguished standard data and customized data types. When I say customized, I mean that I’ve written code that mashes together more than one data table and/or addresses a specific question. For example, right now users can select from two types of network graphs and corresponding formats. One describes the user’s relationship to their friends, and the other describes the user’s relationship to their friends, as well as all their friends’ relationships to each other in various graph description languages. This is how I made the network graph image below. I’m also interested in hearing other suggestions for custom queries I might add. The project will be open source on Github soon, so even code contributions will be welcome.

Anyway, please try out the new version. You may have to delete the app from your allowed applications and then re-authorize it if you’ve used it before. As usual, you can provide feedback on the application page, and you can also contact me on Twitter via @givememydata.

[1] “Google Financial Tables for Quarter ending June 30, 2009” Retrieved October 13, 2010

Freedom for Our Files: Code and Slides

A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours!

Here are the slides and some of the code samples from the Freedom for Our Files (FFOF) workshop I just did in Linz at Art Meets Radical Openness (LiWoLi 2011).

The first one is a basic scraping demo that uses “find-replace” parsing to change specific words (I’m including examples below the code)

<?php

/*	Basic scraping demo with "find-replace" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$contents = file_get_contents($url);	// 1. get contents of page in a string

					// 2. search and replace contents
$contents = str_replace(		// str_replace(search, replace, string)
			"News",					
			"<b style='background:yellow; color:#000; padding:2px'>LIES</b>",
			$contents);

print $contents;			// 3. print result

?>

Basic scraping demo with “foreach” parsing

<?php

/*	Basic scraping demo with "foreach" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$lines = file($url);			// 1. get contents of url in an array

foreach ($lines as $line_num => $line) 	// 2. loop through each line in page
{		
					// 3. if opening string is found
	if(strpos($line, '<h2 class="top-story-header ">')) 	
	{
		$get_content = true;	// 4. we can start getting content
	}
	
	if($get_content == true)
	{
		$data .= $line . "\n";	// 5. then store content until closing string appears
	}

	if(strpos($line, "</h2>")) 	// 6. if closing HTML element found
	{
		$get_content = false;	// 7. stop getting content
	}
}

print $data;				// 8. print result

?>

Basic scraping demo with “regex” parsing

<?php

/*	Basic scraping demo with "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";		// 0. url to start with

$contents = file_get_contents($url);		// 1. get contents of url in a string
											
						// 2. match title
preg_match('/<title>(.*)<\/title>/i', $contents, $title);

print $title[1];				// 3. print result

?>

Basic scraping demo with “foreach” and “regex” parsing

<?php

/*	Basic scraping demo with "foreach" and "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

// url to start
$url = "http://www.bbc.co.uk/news/";

// get contents of url in an array
$lines = file($url);

// look for the string
foreach ($lines as $line_num => $line) 
{	
	// find opening string
	if(strpos($line, '<h2 class="top-story-header ">')) 
	{
		$get_content = true;
	}
	
	// if opening string is found 
	// then print content until closing string appears
	if($get_content == true) 
	{
		$data .= $line . "\n";
	}

	// closing string
	if(strpos($line, "</h2>")) 
	{
		$get_content = false;
	}
}

// use regular expressions to extract only what we need...

// png, jpg, or gif inside a src="..." or src='...' 
$pattern = "/src=[\"']?([^\"']?.*(png|jpg|gif))[\"']?/i";
preg_match_all($pattern, $data, $images);

// text from link
$pattern = "/(<a.*>)(\w.*)(<.*>)/ismU";
preg_match_all($pattern, $data, $text);

// link
$pattern = "/(href=[\"'])(.*?)([\"'])/i";
preg_match_all($pattern, $data, $link);

/* 
// test if you like
print "<pre>";
print_r($images);
print_r($text);
print_r($link);
print "</pre>";
*/

?>

<html>
<head>
<style> 
body { margin:0; } 
.textblock { position:absolute; top:600px; left:0px; }
span { font:5.0em/1.0em Arial, Helvetica, sans-serif; line-height:normal; 
background:url(trans.png); color:#fff; font-weight:bold; padding:5px } 
a { text-decoration:none; color:#900 }
</style>
</head>
<body>
<img src="<?php print $images[1][0] ?>" height="100%"> </div>
<div class="textblock"><span><a href="<?php print "http://www.bbc.co.uk".$link[2][0] ?>"><?php print $text[2][0] ?></a></span><br>
</div>
</body>
</html>

And the example, which presents the same information in a new way…

Advanced scraping demo with “regex” parsing. Retrieves current weather in any city and colors the background accordingly. The math below for normalization could use some work.

<?php

/*	Advanced scraping demo with "regex" parsing. Retrieves current 
 * 	weather in any city and colors the background accordingly. 
 *	The math below for normalization could use some work.
 *	Owen Mundy Copyright 2011 GNU/GPL */

?>

<html>
<head>
<style> 
body { margin:20; font:1.0em/1.4em Arial, Helvetica, sans-serif; } 
.text { font:10.0em/1.0em Arial, Helvetica, sans-serif; color:#000; font-weight:bold; } 
.navlist { list-style:none; margin:0; position:absolute; top:20px; left:200px }
.navlist li { float:left; margin-right:10px; }
</style>
</head>

<body onLoad="document.f.q.focus();">

<form method="GET" action="<?php print $_SERVER['PHP_SELF']; ?>" name="f">

	<input type="text" name="q" value="<?php print $_GET['q'] ?>" />
	<input type="submit" />

</form>

<ul class="navlist">
	<li><a href="?q=anchorage+alaska">anchorage</a></li>
	<li><a href="?q=toronto+canada">toronto</a></li>
	<li><a href="?q=new+york+ny">nyc</a></li>
	<li><a href="?q=london+uk">london</a></li>
	<li><a href="?q=houston+texas">houston</a></li>
	<li><a href="?q=linz+austria">linz</a></li>
	<li><a href="?q=rome+italy">rome</a></li>
	<li><a href="?q=cairo+egypt">cairo</a></li>
	<li><a href="?q=new+delhi+india">new delhi</a></li>
	<li><a href="?q=mars">mars</a></li>
</ul>

<?php

// make sure the form has been sent
if (isset($_GET['q']))
{
	// get contents of url in an array
	if ($str = file_get_contents('http://www.google.com/search?q=weather+in+'
						. str_replace(" ","+",$_GET['q'])))
	{
		
		// use regular expressions to extract only what we need...
		
		// 1, 2, or 3 digits followed by any version of the degree symbol 
		$pattern = "/[0-9]{1,3}[º°]C/";
		// match the pattern with a C or with an F
		if (preg_match_all($pattern, $str, $data) > 0)
		{
			$scale = "C";
		}
		else
		{
			$pattern = "/[0-9]{1,3}[º°]F/";
			if (preg_match_all($pattern, $str, $data) > 0)
			{
				$scale = "F";
			}
		}
		
		// remove html
		$temp_str = strip_tags($data[0][0]);
		// remove everything except numbers and points
		$temp = ereg_replace("[^0-9..]", "", $temp_str);
		
		if ($temp)
		{
			
			// what is the scale?
			if ($scale == "C"){
				// convert ºC to ºF
				$tempc = $temp;
				$tempf = ($temp*1.8)+32;
			}
			else if ($scale == "F")
			{
				// convert ºF to ºC
				$tempc = ($temp-32)/1.8;
				$tempf = $temp;
			}
			// normalize the number
			$color = round($tempf/140,1)*10;
			// cool -> warm
			// scale -20 to: 120
			$color_scale = array(
					'0,  0,255',
					'0,128,255',
					'0,255,255',
					'0,255,128',
					'0,255,0',
					'128,255,0',
					'255,255,0',
					'255,128,0',
					'255,  0,0'
					);	
		
?>

<style> body { background:rgb(<?php print $color_scale[$color] ?>) }</style>
<div class="text"><?php print round($tempc,1) ."&deg;C " ?></div>
<?php print round($tempf,1) ?>&deg;F

<?php
		
		}
		else 
		{
			print "city not found";	
		}
	}
}
?>

</body>
</html>




For an xpath tutorial check this page.

For the next part of the workshop we used Give Me My Data to export our information from Facebook in order to revisualize it with Nodebox 1.0, a Python IDE similar to Processing.org. Here’s an example:

Update: Some user images from the workshop. Thanks all who joined!

Mutual friends (using Give Me My Data and Graphviz) by Rob Canning

identi.ca network output (starting from my username (claude) with depth 5, rendered to svg with ‘sfdp’ from graphviz) by Claude Heiland-Allen

Freedom for Our Files: Creative Reuse of Personal Data Workshop at Art Meets Radical Openness in Linz, Austria

This weekend I am presenting a lecture about GIve Me My Data and conducting a two-day data-scraping workshop at Art Meets Radical Openness in Linz, Austria. Here are the details.

The Self-Indulgence of Closed Systems
May 13, 18:45 – 19:15
Part artist lecture, part historical context, Owen Mundy will discuss his Give Me My Data project within the contexts of the history of state surveillance apparatuses, digital media and dialogical art practices, and the ongoing contradiction of privacy and utility in new media.

Freedom for Our Files: Creative Reuse of Personal Data
May 13-14, 14:00 – 16:30
A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours! Register here

Art Meets Radical Openness (LiWoLi 2011),
Date: 12th – 14th May 2011
Location: Kunstuniversität Linz, Hauptplatz 8, 4020 Linz, Austria

Observing, comparing, reflecting, imitating, testing, combining

LiWoLi is an open lab and meeting spot for artists, developers and educators using and creating FLOSS (free/libre open source software) and Open Hardware in the artistic and cultural context. LiWoLi is all about sharing skills, code and knowledge within the public domain and discussing the challenges of open practice.

Automata: Counter-Surveillance in Public Space paper on the Public Interventions panel at ISEA2010

isea2010_logo_klein

ISEA2010 RUHR Conference in Dortmund, Germany

P26 Public Interventions
Tue 24 August 2010
15:00–16:30h
Volkshochschule Dortmund, S 137a
Moderated by Georg Dietzler (de)

  • 15:00h | Owen Mundy (us): Automata: Counter-Surveillance in Public Space
  • 15:20h | Christoph Brunner (ch/ca), Jonas Fritsch (dk): Balloons, Sweat and Technologies. Urban Interventions through Ephemeral Architectures
  • 15:40h | Georg Klein (de): Don’t Call It Art! On Artistic Strategies and Political Implications of Media Art in Public Space
  • 16:00h | Georg Dietzler (de): Radical Ecological Art and No Greenwash Exhibitions

About my talk:

Automata is the working title for a counter-surveillance internet bot that will record and display the mutually-beneficial interrelationships between institutions for higher learning, the global defense industry, and world militaries. Give Me My Data is a Facbook application that help users reclaim and reuse their Facebook data. The two projects, both ongoing, address important issues surounding contemporary forms of communication, surveillance, and control.