Category: teaching

Convert a PHP/MySQL site to Jamstack and host on Github Pages

One of the first items on my sabbatical to do list last year was to convert the Camp La Jolla Military Park website to a static Jamstack site hosted at Github Pages. This post documents that process.

I originally created the site as a research tool using PHP and a MySQL database for data entry, to record, organize, and ultimately publish research about the military history of UCSD. Moving the project to Github Pages would simplify maintenance and hosting costs, but was no small task. Technically it didn’t cost me extra to host, but the domain was $20/year, which adds up. Further, maintaining a PHP/MySQL website with a custom admin tool requires a bit of overhead for security and updates. Additionally, I wanted to continue having the benefit of using template engines or some way to include files (like PHP’s require()) to save time coding the site.

There are a plethora of tools available to publish a database-driven site on a static host. They mostly can be categorized as follows (ordered by most to least dynamic data sources):

  • Move the database to a remote service like Firebase, Atlas, et al. — ⚠️ Ultimately creates vendor lock-in, potential hidden costs, and probably more maintenance issues in the future.
  • Create a static version of the database by converting MySQL to flat files (JSON, CSV, etc.), which still allows for querying using one of various tools — 🤔 This method is interesting, because it still decouples the data from the presentation and allows for potentially more options later, including a remote database. I am already using this in various forms but worry again about maintenance, or issues with the client’s ability to manipulate the data as needed.
  • Generate a static version of the site, including all its pages which contain the data formerly dynamically inserted from MySQL. — 🤔 I like this concept, and have been using it to auto-publish my class lectures and tutorials using Markdown and Grunt with Marp. Plus, there are many site generators to choose from, which I explore below.
  • Scrape the site and save pages as static files. After all, everything is static once the HTML is rendered on the frontend. — ⚠️ While fast, this requires the site be built first and would make updates difficult.

Essential Concepts

Here I outline some concepts on rendering. See SSR vs CSR vs ISR vs SSG for a more in-depth look:

  • Client-Side Rendering (CSR) (e.g. Single-Page Application (SPA)) – JS renders pages in client’s browser with empty HTML files + raw data from the server. (e.g. React, Vue, Angular)
  • Server-Side Rendering (SSR) (e.g. Fullstack) – Generates pages on server with dynamic data before delivering to client. Improves SEO and loading, decreases server performance. (e.g. EJS, PHP, or Next.js)
  • Static Site Generation (SSG) (e.g. JAMStack) – Pre-renders all pages as static HTML during build process. Fast loading times, enhanced security, and dynamic content can still be achieved via “islands”.
  • Incremental Static Regeneration (ISR) – Combines SSR and SSG to pre-render static pages during build time while periodically regenerating specific pages with updated data.
  • Edge-Side Rendering (ESR) – Rendering process is pushed to the CDN’s edge servers which handles requests and generates HTML. Requires Nitro engine.

Requirements

Requirements for most of the projects I mention above.

  1. Javascript-based
  2. The data will be static, but editable using Git
  3. The data should be searchable using a tool like chosen
  4. The site design will be simple, but I should have creative control
  5. The site will be public, and be presented mostly using the same design as the original site
  6. HTML/CSS validation + Accessibility checks
  7. Convert from Google Maps to Leaflet
  8. Host the site on Github Pages

While planning the project I also wanted to select a method I could use for other potential static (or nearly static) sites like:

  1. A new site documenting the hundreds of examples and overlapping categories of Critical Web Design and internet art I’ve been collecting for an upcoming book. Update: I built it with SvelteKit
  2. Converting tallysavestheinternet.com to a static site (currently SSR) and unlinking the extension from the server to use local data only
  3. Moving away from WordPress more generally (and its security problems) and blogging with Markdown instead, starting with this blog (NO MORE TYPING IN A WEB FORM!!)

It’s nice to learn new tools. I already have some experiences with React, Next.js, and Vue.js, but wouldn’t say I’m confident in any. I found that publishing apps using React Native is very frustrating and unreliable, and my general distrust of Facebook makes me lean towards Vue. Still, I went in with with an open attitude.

Tool Comparison

A breakdown of some tools I considered. Some of the icons are clickable…

Framework SPA SSR SSG MD Themes Search Cons
Jekyll ❌ ❌ ✅ ✅ ✅ ? ❌ Ruby
Hugo ❌ ❌ ✅ ✅ ✅ ? ❌ Go
Astro ✅ ✅ ✅ ✅ ✅ ✅ React, Vue, Svelte in dynamic islands, simple routing
Vue.js ✅ ✅ ✅ ✅ ✅ ? ✅ vuepress
Nuxt.js ✅ ✅ ✅ ✅ ✅ ? Built on Vue
SvelteKit ✅ ✅ ✅ ✅ ✅ ✅ MD rendering not native but supported via MDX
Next.js ✅ ✅ ✅ ✅ ✅ ✅ MD rendering not native but supported via MDX

Next

My experience with Next.js SSG was painful.

  • Constantly running into subtle changes in syntax across versions
  • Adding an image is ridiculously difficult.
  • Exporting the SSG ultimately created dead links, packaged up in Next code that was hard to understand.
  • So much automation to make a static site in the end you can’t do anything unless you have the exact right code, in the right version, in the right paths.
  • Error messages are rarely helpful or pinpoint the line number causing the issue in the stack.
  • I followed so many tutorials that ultimately ended in dead-ends, incorrect versions, or that completely didn’t work.

Astro

I ended up choosing Astro for the final project, which can be seen here https://omundy.github.io/camplajolla/. While they appear to have many similarities, I found Astro soooooo much easier than Next.js. It’s fairly simple to install, even using a starter theme, and it didn’t take long torip out tailwind to add bootstrap.

Upcoming workshop at FSU, “I Know Where Your Cat Lives”: The Process of Mapping Big Data for Inconspicuous Trends

I’m doing a workshop / lecture as part of the ongoing Digital Scholars digital humanities discussion group here at Florida State University. Workshop is free and open to the public.

Wednesday, March 25, 2:00-3:30 pm
Fine Arts Building (FAB) 320A [530 W. Call St. map]

Screen Shot 2014-10-29 at 4.32.37 PM

“I Know Where Your Cat Lives”: The Process of Mapping Big Data for Inconspicuous Trends

Big Data culture has its supporters and its skeptics, but it can have critical or aesthetic value even for those who are ambivalent. How is it possible, for example, to consider data as more than information — as the performance of particular behaviors, the practice of communal ideals, and the ethic motivating new media displays? Professor Owen Mundy from FSU’s College of Fine Arts invites us to take up these questions in a guided exploration of works of art that will highlight what he calls “inconspicuous trends.” Using the “I Know Where Your Cat Lives” project as a starting point, Professor Mundy will introduce us to the technical and design process for mapping big data in projects such as this one, showing us the various APIs (Application Program Interfaces) that are constructed to support them and considering the various ways we might want to visualize their results.

This session offers a hands-on demonstration and is designed with a low barrier of entry in mind. For those completely unfamiliar with APIs, this session will serve as a useful introduction, as Professor Mundy will walk us through the process of connecting to and retrieving live social media data from the Instagram API and rendering it using the Google Maps API. Participants should not worry if they do not have expertise in big data projects or are still learning the associated vocabulary. We come together to learn together, and all levels of skill will be accommodated, as will all attitudes and leanings. Desktop computers are installed in FAB 320A, but participants are welcome to bring their own laptops and wireless devices.

Participants are encouraged to read the following in advance of the meeting:

and to browse the following resources for press on Mundy’s project:

For further (future) reading:

Term vs. Term for Digital Public Library of America hackathon

I made a small app to compare the number of search results for two phrases from the Digital Public Library of America for a hackathon / workshop here at Florida State next week.

http://owenmundy.com/work/term-vs-term

dpla term vs term

Digital Humanities Hackathon II – Digital Public Library of America

Monday, April 21, 2:00-3:30 p.m.
Strozier Library, Scholars Commons Instructional Classroom [MAP]

The Digital Scholars Reading and Discussion Group will simulate its second “hackathon” on April 21, allowing participants to learn more about the back-end structure of the Digital Public Library of America. With its April 2013 launch, the DPLA became the first all-digital library that aggregates metadata from collections across the country, making them available from a single point of access. The DPLA describes itself as a freely available, web-based platform for digitized cultural heritage projects as well as a portal that connects students, teachers, scholars, and the public to library resources occurring on other platforms.

From a critical point of view, the DPLA simultaneously relies on and disrupts the principles of location and containment, making its infrastructure somewhat interesting to observe.

In this session, we will visit the DPLA’s Application Programming Interface (API) codex to observe some of the standards that contributed to its construction. We will consider how APIs function, how and why to use them, and who might access their metadata and for what purposes. For those completely unfamiliar with APIs, this session will serve as a useful introduction, as well as a demonstration of why a digital library might also want to serve as an online portal. For those more familiar with APIs, this session will serve as an opportunity to try on different tasks using the metadata that the DPLA aggregates from collections across the country.

At this particular session, we are pleased to be joined by Owen Mundy from FSU Department of Art and Richard Urban from FSU College of Communication and Information, who have considered different aspects of working with APIs for projects such as the DPLA, including visualization and graphics scripting, and developing collections dashboards.

As before, the session is designed with a low barrier of entry in mind, so participants should not worry if they do not have programming expertise or are still learning the vocabulary associated with open-source projects. We come together to learn together, and all levels of skill are accommodated, as are all attitudes and leanings.

Participants are encouraged to explore the Digital Public Library of America site prior to our meeting and to familiarize themselves with the history of the project. Laptops will be available for checkout, but attendees are encouraged to bring their own.

Internet service just got creepy: How to set up a wireless router with Comcast internet service

I just moved back to Florida after a one year research project in Berlin and have subscribed to Comcast broadband service. The whole experience left a bad taste in my mouth, though not because the tech showed-up 2 hours after the installation appointment window. Nor was it because he held loud personal conversations on his cell phone while he was setting up the service. No, the icky feeling is more corporate and selfish, and impedes much more into my private space than “Joe the cable guy” ever could.

Comcast made me install software on my computer in order to use their broadband.

Upon his arrival, “Joe” announced he would need access to my computer to setup broadband service. Understanding that most of the people Joe deals with might not be IT whizzes, and could manage to not be able to connect their machines without his help, I decided to let him use it rather than attempt to prove I was not a member of the usual group. After half an hour of complaining about previous customers to his friend on his cellphone, waiting for an other Comcast person to flip a switch allowing him to do his job, and multiple trips to his truck, he showed me that the internet was indeed accessible on my computer.

At this point the laptop was directly connected to the cable modem via an ethernet cable. He announced I was to follow the steps on the screen and he was out the door. The web page he had left up required me to agree to some terms, create a username and then… install software? Really? I tried to access the net without the final step but nothing doing. Unless I installed this software I was stuck. So I did it, still not believing that a company had really initiated this final invasion onto every customer’s computer. After it was done I had new bookmarks everywhere, for Comcast email, security, and some branding nonsense called “XFINITY” (I thought “X” was out with the ’90’s and “X”games?)

So I thought, “OK, Comcast, you got me, hit me with your best marketing slime. Whatever, I can delete the bookmarklets you installed in my browser, just let me access the service I paid for, wirelessly, on whichever device I want.”

But this is where the relationship got really creepy. Apparently when I installed the Comcast (spyware?) on my machine, it made note of my MAC address, a unique identifier of networked machines, so that it would only allow my machine (or another machine with that MAC address) to connect to the internet. This means when I attached a wireless router to the cable modem I could connect to the wifi, but there was no internet.

So it turns-out that Comcast is not only forcing their adware on customers, it’s also making it difficult (though not impossible) for them use more than one device. Presumably Comcast is doing this in order to circumvent sharing of services among neighbors, but the end result is that you can’t share the service between more than one device, or between roommates or spouses for that matter.

An example (albeit a geeky one): between my wife and I we have 2 laptops, 2 smartphones, and a desktop computer that all might be talking to each other or accessing the net. Comcast’s so-called internet service didn’t allow for any such geekery because it only allows one device, with the correct MAC address, to connect.

So, here’s what I did, on my Mac, with some help from my sister’s boyfriend, Tom, and a lot from Google, to get my linksys wireless router to work with Comcast internet.

  1. Confirm you can access the internet with your machine connected directly to the Comcast cable modem.
  2. Open Terminal and type (without the quotes): “ifconfig en0 | grep ether”
  3. Now disconnect your computer from the modem and connect the modem ethernet cable to your wireless router. Make sure both are plugged-in.
  4. Connect to your wireless router via the airport on your machine.
  5. Go to the following link: http://192.168.1.1
  6. Under Setup, choose DHCP as the Internet Connection Type. Save Settings.
  7. Under Setup : Mac Address Clone, enter the alpha numeric characters returned from Terminal. Save Settings.
  8. Configure your wireless router like you normally would and you are up and running.
  9. Snicker at Comcast

Freedom for Our Files: Code and Slides

A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours!

Here are the slides and some of the code samples from the Freedom for Our Files (FFOF) workshop I just did in Linz at Art Meets Radical Openness (LiWoLi 2011).

The first one is a basic scraping demo that uses “find-replace” parsing to change specific words (I’m including examples below the code)

<?php

/*	Basic scraping demo with "find-replace" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$contents = file_get_contents($url);	// 1. get contents of page in a string

					// 2. search and replace contents
$contents = str_replace(		// str_replace(search, replace, string)
			"News",					
			"<b style='background:yellow; color:#000; padding:2px'>LIES</b>",
			$contents);

print $contents;			// 3. print result

?>

Basic scraping demo with “foreach” parsing

<?php

/*	Basic scraping demo with "foreach" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";	// 0. url to start with

$lines = file($url);			// 1. get contents of url in an array

foreach ($lines as $line_num => $line) 	// 2. loop through each line in page
{		
					// 3. if opening string is found
	if(strpos($line, '<h2 class="top-story-header ">')) 	
	{
		$get_content = true;	// 4. we can start getting content
	}
	
	if($get_content == true)
	{
		$data .= $line . "\n";	// 5. then store content until closing string appears
	}

	if(strpos($line, "</h2>")) 	// 6. if closing HTML element found
	{
		$get_content = false;	// 7. stop getting content
	}
}

print $data;				// 8. print result

?>

Basic scraping demo with “regex” parsing

<?php

/*	Basic scraping demo with "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */
 
$url = "http://www.bbc.co.uk/news/";		// 0. url to start with

$contents = file_get_contents($url);		// 1. get contents of url in a string
											
						// 2. match title
preg_match('/<title>(.*)<\/title>/i', $contents, $title);

print $title[1];				// 3. print result

?>

Basic scraping demo with “foreach” and “regex” parsing

<?php

/*	Basic scraping demo with "foreach" and "regex" parsing
 *	Owen Mundy Copyright 2011 GNU/GPL */

// url to start
$url = "http://www.bbc.co.uk/news/";

// get contents of url in an array
$lines = file($url);

// look for the string
foreach ($lines as $line_num => $line) 
{	
	// find opening string
	if(strpos($line, '<h2 class="top-story-header ">')) 
	{
		$get_content = true;
	}
	
	// if opening string is found 
	// then print content until closing string appears
	if($get_content == true) 
	{
		$data .= $line . "\n";
	}

	// closing string
	if(strpos($line, "</h2>")) 
	{
		$get_content = false;
	}
}

// use regular expressions to extract only what we need...

// png, jpg, or gif inside a src="..." or src='...' 
$pattern = "/src=[\"']?([^\"']?.*(png|jpg|gif))[\"']?/i";
preg_match_all($pattern, $data, $images);

// text from link
$pattern = "/(<a.*>)(\w.*)(<.*>)/ismU";
preg_match_all($pattern, $data, $text);

// link
$pattern = "/(href=[\"'])(.*?)([\"'])/i";
preg_match_all($pattern, $data, $link);

/* 
// test if you like
print "<pre>";
print_r($images);
print_r($text);
print_r($link);
print "</pre>";
*/

?>

<html>
<head>
<style> 
body { margin:0; } 
.textblock { position:absolute; top:600px; left:0px; }
span { font:5.0em/1.0em Arial, Helvetica, sans-serif; line-height:normal; 
background:url(trans.png); color:#fff; font-weight:bold; padding:5px } 
a { text-decoration:none; color:#900 }
</style>
</head>
<body>
<img src="<?php print $images[1][0] ?>" height="100%"> </div>
<div class="textblock"><span><a href="<?php print "http://www.bbc.co.uk".$link[2][0] ?>"><?php print $text[2][0] ?></a></span><br>
</div>
</body>
</html>

And the example, which presents the same information in a new way…

Advanced scraping demo with “regex” parsing. Retrieves current weather in any city and colors the background accordingly. The math below for normalization could use some work.

<?php

/*	Advanced scraping demo with "regex" parsing. Retrieves current 
 * 	weather in any city and colors the background accordingly. 
 *	The math below for normalization could use some work.
 *	Owen Mundy Copyright 2011 GNU/GPL */

?>

<html>
<head>
<style> 
body { margin:20; font:1.0em/1.4em Arial, Helvetica, sans-serif; } 
.text { font:10.0em/1.0em Arial, Helvetica, sans-serif; color:#000; font-weight:bold; } 
.navlist { list-style:none; margin:0; position:absolute; top:20px; left:200px }
.navlist li { float:left; margin-right:10px; }
</style>
</head>

<body onLoad="document.f.q.focus();">

<form method="GET" action="<?php print $_SERVER['PHP_SELF']; ?>" name="f">

	<input type="text" name="q" value="<?php print $_GET['q'] ?>" />
	<input type="submit" />

</form>

<ul class="navlist">
	<li><a href="?q=anchorage+alaska">anchorage</a></li>
	<li><a href="?q=toronto+canada">toronto</a></li>
	<li><a href="?q=new+york+ny">nyc</a></li>
	<li><a href="?q=london+uk">london</a></li>
	<li><a href="?q=houston+texas">houston</a></li>
	<li><a href="?q=linz+austria">linz</a></li>
	<li><a href="?q=rome+italy">rome</a></li>
	<li><a href="?q=cairo+egypt">cairo</a></li>
	<li><a href="?q=new+delhi+india">new delhi</a></li>
	<li><a href="?q=mars">mars</a></li>
</ul>

<?php

// make sure the form has been sent
if (isset($_GET['q']))
{
	// get contents of url in an array
	if ($str = file_get_contents('http://www.google.com/search?q=weather+in+'
						. str_replace(" ","+",$_GET['q'])))
	{
		
		// use regular expressions to extract only what we need...
		
		// 1, 2, or 3 digits followed by any version of the degree symbol 
		$pattern = "/[0-9]{1,3}[º°]C/";
		// match the pattern with a C or with an F
		if (preg_match_all($pattern, $str, $data) > 0)
		{
			$scale = "C";
		}
		else
		{
			$pattern = "/[0-9]{1,3}[º°]F/";
			if (preg_match_all($pattern, $str, $data) > 0)
			{
				$scale = "F";
			}
		}
		
		// remove html
		$temp_str = strip_tags($data[0][0]);
		// remove everything except numbers and points
		$temp = ereg_replace("[^0-9..]", "", $temp_str);
		
		if ($temp)
		{
			
			// what is the scale?
			if ($scale == "C"){
				// convert ºC to ºF
				$tempc = $temp;
				$tempf = ($temp*1.8)+32;
			}
			else if ($scale == "F")
			{
				// convert ºF to ºC
				$tempc = ($temp-32)/1.8;
				$tempf = $temp;
			}
			// normalize the number
			$color = round($tempf/140,1)*10;
			// cool -> warm
			// scale -20 to: 120
			$color_scale = array(
					'0,  0,255',
					'0,128,255',
					'0,255,255',
					'0,255,128',
					'0,255,0',
					'128,255,0',
					'255,255,0',
					'255,128,0',
					'255,  0,0'
					);	
		
?>

<style> body { background:rgb(<?php print $color_scale[$color] ?>) }</style>
<div class="text"><?php print round($tempc,1) ."&deg;C " ?></div>
<?php print round($tempf,1) ?>&deg;F

<?php
		
		}
		else 
		{
			print "city not found";	
		}
	}
}
?>

</body>
</html>




For an xpath tutorial check this page.

For the next part of the workshop we used Give Me My Data to export our information from Facebook in order to revisualize it with Nodebox 1.0, a Python IDE similar to Processing.org. Here’s an example:

Update: Some user images from the workshop. Thanks all who joined!

Mutual friends (using Give Me My Data and Graphviz) by Rob Canning

identi.ca network output (starting from my username (claude) with depth 5, rendered to svg with ‘sfdp’ from graphviz) by Claude Heiland-Allen

Freedom for Our Files: Creative Reuse of Personal Data Workshop at Art Meets Radical Openness in Linz, Austria

This weekend I am presenting a lecture about GIve Me My Data and conducting a two-day data-scraping workshop at Art Meets Radical Openness in Linz, Austria. Here are the details.

The Self-Indulgence of Closed Systems
May 13, 18:45 – 19:15
Part artist lecture, part historical context, Owen Mundy will discuss his Give Me My Data project within the contexts of the history of state surveillance apparatuses, digital media and dialogical art practices, and the ongoing contradiction of privacy and utility in new media.

Freedom for Our Files: Creative Reuse of Personal Data
May 13-14, 14:00 – 16:30
A two-day workshop, with both technical hands-on and idea-driven components. Learn to scrape data and reuse public and private information by writing custom code and using the Facebook API. Additionally, we’ll converse and conceptualize ideas to reclaim our data literally and also imagine what is possible with our data once it is ours! Register here

Art Meets Radical Openness (LiWoLi 2011),
Date: 12th – 14th May 2011
Location: Kunstuniversität Linz, Hauptplatz 8, 4020 Linz, Austria

Observing, comparing, reflecting, imitating, testing, combining

LiWoLi is an open lab and meeting spot for artists, developers and educators using and creating FLOSS (free/libre open source software) and Open Hardware in the artistic and cultural context. LiWoLi is all about sharing skills, code and knowledge within the public domain and discussing the challenges of open practice.

Art Meets Radical Openness (LiWoLi 2011), May 12-14, Linz, Austria

Art Meets Radical Openness (LiWoLi 2011)

Date: 12th – 14th May 2011
Location: Kunstuniversität Linz, Hauptplatz 8, 4020 Linz

Observing, comparing, reflecting, imitating, testing, combining

LiWoLi is an open lab and meeting spot for artists, developers and educators using and creating FLOSS (free/libre open source software) and Open Hardware in the artistic and cultural context. LiWoLi is all about sharing skills, code and knowledge within the public domain and discussing the challenges of open practice.

This year’s event offers an exhibition, artists’ workshops and – like every year – lectures, presentations and sound-performances.

with:
minipimer.tv (ES), Aileen Derieg (A), Martin Howse (UK/DE), Jorge Crowe (AR)pending), Malte Steiner(DE), Servando Barreiro (ES), Enrique Tomás(ES/A), Peter Bubestinger (A), Stefan Hageneder (A), Audrey Samson (CA/NL), Sabrina Baston (DE), Sebastian Pichelhofer (A), Barbara Huber (A), Max Nagele (A), Helen Varley Jamieson (NZ/DE), Adnan Hadzi (GB), André Carvalho Hostalácio (BR/DE), Claudia González Godoy (CL), Dominik Leitner (A), Rob Canning (IR/UK), Marloes de Valk (NL), Owen Mundy (US/DE), Dušan Barok (Sk), Nicolas Malevé (BE), Margaritha Köhl (A), Pippa Buchanan (AU/A), Birgit Bachler (A/NL),…

More information: http://liwoli.at/