Convert a PHP/MySQL site to Jamstack and host on Github Pages

One of the first items on my sabbatical to do list last year was to convert the Camp La Jolla Military Park website to a static Jamstack site hosted at Github Pages. This post documents that process.

I originally created the site as a research tool using PHP and a MySQL database for data entry, to record, organize, and ultimately publish research about the military history of UCSD. Moving the project to Github Pages would simplify maintenance and hosting costs, but was no small task. Technically it didn’t cost me extra to host, but the domain was $20/year, which adds up. Further, maintaining a PHP/MySQL website with a custom admin tool requires a bit of overhead for security and updates. Additionally, I wanted to continue having the benefit of using template engines or some way to include files (like PHP’s require()) to save time coding the site.

There are a plethora of tools available to publish a database-driven site on a static host. They mostly can be categorized as follows (ordered by most to least dynamic data sources):

  • Move the database to a remote service like Firebase, Atlas, et al. — ⚠️ Ultimately creates vendor lock-in, potential hidden costs, and probably more maintenance issues in the future.
  • Create a static version of the database by converting MySQL to flat files (JSON, CSV, etc.), which still allows for querying using one of various tools — 🤔 This method is interesting, because it still decouples the data from the presentation and allows for potentially more options later, including a remote database. I am already using this in various forms but worry again about maintenance, or issues with the client’s ability to manipulate the data as needed.
  • Generate a static version of the site, including all its pages which contain the data formerly dynamically inserted from MySQL. — 🤔 I like this concept, and have been using it to auto-publish my class lectures and tutorials using Markdown and Grunt with Marp. Plus, there are many site generators to choose from, which I explore below.
  • Scrape the site and save pages as static files. After all, everything is static once the HTML is rendered on the frontend. — ⚠️ While fast, this requires the site be built first and would make updates difficult.

Essential Concepts

Here I outline some concepts on rendering. See SSR vs CSR vs ISR vs SSG for a more in-depth look:

  • Client-Side Rendering (CSR) (e.g. Single-Page Application (SPA)) – JS renders pages in client’s browser with empty HTML files + raw data from the server. (e.g. React, Vue, Angular)
  • Server-Side Rendering (SSR) (e.g. Fullstack) – Generates pages on server with dynamic data before delivering to client. Improves SEO and loading, decreases server performance. (e.g. EJS, PHP, or Next.js)
  • Static Site Generation (SSG) (e.g. JAMStack) – Pre-renders all pages as static HTML during build process. Fast loading times, enhanced security, and dynamic content can still be achieved via “islands”.
  • Incremental Static Regeneration (ISR) – Combines SSR and SSG to pre-render static pages during build time while periodically regenerating specific pages with updated data.
  • Edge-Side Rendering (ESR) – Rendering process is pushed to the CDN’s edge servers which handles requests and generates HTML. Requires Nitro engine.

Requirements

Requirements for most of the projects I mention above.

  1. Javascript-based
  2. The data will be static, but editable using Git
  3. The data should be searchable using a tool like chosen
  4. The site design will be simple, but I should have creative control
  5. The site will be public, and be presented mostly using the same design as the original site
  6. HTML/CSS validation + Accessibility checks
  7. Convert from Google Maps to Leaflet
  8. Host the site on Github Pages

While planning the project I also wanted to select a method I could use for other potential static (or nearly static) sites like:

  1. A new site documenting the hundreds of examples and overlapping categories of Critical Web Design and internet art I’ve been collecting for an upcoming book. Update: I built it with SvelteKit
  2. Converting tallysavestheinternet.com to a static site (currently SSR) and unlinking the extension from the server to use local data only
  3. Moving away from WordPress more generally (and its security problems) and blogging with Markdown instead, starting with this blog (NO MORE TYPING IN A WEB FORM!!)

It’s nice to learn new tools. I already have some experiences with React, Next.js, and Vue.js, but wouldn’t say I’m confident in any. I found that publishing apps using React Native is very frustrating and unreliable, and my general distrust of Facebook makes me lean towards Vue. Still, I went in with with an open attitude.

Tool Comparison

A breakdown of some tools I considered. Some of the icons are clickable…

Framework SPA SSR SSG MD Themes Search Cons
Jekyll ? ❌ Ruby
Hugo ? ❌ Go
Astro React, Vue, Svelte in dynamic islands, simple routing
Vue.js ? ✅ vuepress
Nuxt.js ? Built on Vue
SvelteKit MD rendering not native but supported via MDX
Next.js MD rendering not native but supported via MDX

Next

My experience with Next.js SSG was painful.

  • Constantly running into subtle changes in syntax across versions
  • Adding an image is ridiculously difficult.
  • Exporting the SSG ultimately created dead links, packaged up in Next code that was hard to understand.
  • So much automation to make a static site in the end you can’t do anything unless you have the exact right code, in the right version, in the right paths.
  • Error messages are rarely helpful or pinpoint the line number causing the issue in the stack.
  • I followed so many tutorials that ultimately ended in dead-ends, incorrect versions, or that completely didn’t work.

Astro

I ended up choosing Astro for the final project, which can be seen here https://omundy.github.io/camplajolla/. While they appear to have many similarities, I found Astro soooooo much easier than Next.js. It’s fairly simple to install, even using a starter theme, and it didn’t take long torip out tailwind to add bootstrap.

DIY Bike wheel hub end cap puller

How to make an end cap puller using a leftover piece of carbon bar and some hose clamps.

  1. Carefully place a piece of carbon (or metal) bar into a vise.
  2. Make two cuts along the length, removing about 1/4 in. (~5mm) to make room for the piece to tighten around the end cap.
  3. Slide the piece of bar over the end cap. Then slide a hose clamp flush with the hub and tighten it (I used two small interlocked clamps).
  4. Wiggle and pull the cap off!

Escape from music subscription services

Like the rest of the world, I’ve moved on from CDs for music. However, since I’ve converted all my discs to digital format, and I’m not likely to pay for a smartphone that can hold 140 GB of audio, my music collection has only received attention while I’m bleeding MTB brakes in my garage. Since I recently upgraded my radio in my car to a Boss Carplay radio, which supports loading media from a thumb drive, I decided to try to revive my old tunes.

The first issue is that while the Boss Radio BE7ACP.WX (an upgrade from the original BE920WCPA I purchased, and had to warranty once for the screen flashing on and off repeatedly, and another time for the screen information appearing completely upside down!) only supports MP3 files (and a thumb drive formatted with FAT32 with a max size of 32 GB!~). No problem, this can be done with FFMPEG…

However, what if you have 140 GB of music, with a folder for each artist, and inside that, a folder for each album?

Thus I share this project github.com/omundy/convert-music-library, which will convert an entire iTunes (or other) music library from .m4a to .mp3 (or .wav), maintaining the original directory structure (Artist/Album/Song). It still requires FFMPEG, and Node, but it may help someone else out there avoid locking themselves into a stupid subscription service. Enjoy!

Scoreboard Operator’s Instructions (Redesigned) MPC Control (Model Code 134 Basketball)

If you know me then you already know I have little patience for bad interface and user experience design. Like most users, I leave websites when they fail to provide intuitive paths to the information I need. However, the physical world is full of bad design you can’t escape.

Take this device. The MPC Control (Model Code 134 Basketball) sold by NEVCO. This is a pretty standard piece of equipment in North American youth basketball games (like my daughter’s). While I used this thing for a whole season, I hated every moment. Examine the below and try to describe how you would correct a mistake if you added 3 to the guest team’s score and should have added 2. Before you try, note that this equipment does not have an “undo” button.

This year is going to be different. This is the year I pour over their manual (another atrocious bag of text describing all the unintuitive steps and making up for the poor UI design) and redesign the instructions using a single side of one letter-sized page. So here, I offer to parents everywhere, those who worked all day at their “job” and have about five minutes to figure out how to use this thing, my redesign of the Scoreboard Operator’s Instructions for the Nevco MPC Control (Model Code 134 Basketball). There’s also a PDF. Enjoy.

Here’s a permanent link to this post: https://bit.ly/scoreboard-control-134-basketball

Algorithmic Lifestyle @ Roehrs & Boetsch, Zurich

I haven’t posted here for a while, as I’ve been busy with projects and returning to teaching. I am proud to share that I Know Where Your Cat Lives is in a show opening today in Zurich. The statement is below.

GROUP EXHIBITION – ALGORITHMIC LIFESTYLE
31. 8. – 13. 10. 2018

We click, we like, we share. In today’s attention economy the new currency is “Likes”; Internet giants constantly compete for our attention through advertisement, register our every move, hope for a second of our time. At long last, the power lies in the hands of the people – grassroots democracy in the World Wide Web where everybody gets to voice their opinion. But looks are deceiving and while we share the latest picture of our cat on social media, we are always producing data to be mined by algorithms for our providers, always under surveillance and always targeted by even more likeable content. We live in a world full of pleasing distractions, surrounded by like-minded peers and without interference of reality – an “algorithmic lifestyle”.

In light of contemporary society being increasingly permeated by the Internet, Roehrs & Boetsch is pleased to announce the group exhibition ‘ALGORITHMIC LIFESTYLE’. The show displays works by ten artists und artist duos, of whom seven are presented at the gallery for the first time. In this exhibition different aspects of our digitalised lifestyles are explored by looking closely at the networks we belong to. The selected artists themselves are part of the digital world – often working with algorithms – while studying the Internet and revealing the mechanisms of the web economy. Above all, ‘Algorithmic Lifestyle’ aims to critically engage the viewer, to give a starting point for discussion and alternative options to our current role as ignorant users in this networked society.

Artists: Lauren Huret, Marc Lee, Olia Lialina, Jonas Lund, Shawn Maximo, !Mediengruppe Bitnik & Low Jack, Milena Milosavljevic, Owen Mundy, Sebastian Schmieg, UBERMORGEN

Curated by Nina Roehrs

Image: Shawn Maximo, 2017, Open Doors

Stasi / Facebook / Big Data DAAD Day 28 – The Stasi Archive and BStU

After some traveling and conferences and development sprints on other projects I have time this week to come back to the Stasi work. The first post is about the Stasi Records Agency (BStU), whose full name (in English) is the impressive, “Federal Commissioner for the Records of the State Security Service of the former German Democratic Republic”.


Monday Demonstration in Leipzig on 16 October 1989

At the end of 1989 the Peaceful Revolution spread across the former East Germany. These protests led to the opening of the border via the emotional breaking apart of the Berlin Wall, as well as the end of the reign of the SED government, and with it, the Stasi.

During the transition protesters realized the Stasi agencies were destroying the files that could incriminate them. Groups of citizens occupied Stasi offices across East Germany and sealed the buildings to prevent further destruction of information. On January 15, 1990, citizens gained access and occupied the Berlin headquarters of the Stasi.

After German reunification the BStU was established to preserve the archives and investigate the past of the Stasi. Since January 1992 they have provided a service to allow citizens to view the Stasi files that concern them.

In Berlin the BStU is located at 31 Karl-Liebknecht-Straße, with the files appropriately continuing to be located in the former Stasi headquarters on Frankfurter Allee in Lichtenberg. Here are some photos I took inside the archives in the former Stasi headquarters.


An example card from the Stasi card index now used to locate information for citizens wanting to know what data the Stasi collected, Stasi Records Agency, 2017


Shelves of citizen files now open to view by the citizens the Stasi surveilled, Stasi Records Agency, 2017


Microfiche shredded by Stasi agents, Stasi Records Agency, 2017


Shelves of surveillance video, Stasi Records Agency, 2017

In addition to providing access to citizen’s files, the BStU assists scholars by allowing viewing of other Stasi documents. In my case, their staff helped me to find and view schematics of postal surveillance machines prepared by the Stasi OTS.


Detail of the Stasi Automatic Letter Closing Machine schematic, Stasi Records Agency, 2017

The BStU also has a thorough library at the Karl-Liebknecht-Straße location. Unlike the Stasi documents, which require one to make an application as a scholar, the library is open to the public.


The BStU library, 2017

A Deutsche Welle report on the Stasi archives, 2010

Stasi / Facebook / Big Data DAAD Day 27 – Exhibits on surveillance at Museum für Fotografie

Yesterday I headed to the Museum für Fotografie to see two exhibitions on surveillance.

The Field Has Eyes. Images of the Surveillant Gaze includes 75 prints, books, photographs and examples of optical apparatus to present a visual and cultural history of the controlling gaze from the 16th to the 20th century. I don’t have many images because ironically the museum of photography does not allow photographs.


Daniel Chodowiecki. The Eye of Providence. 1787. Etching, 4 x 5′”

Still, I was able to track down many works online. An example is this one, which focuses on the “all seeing eye” of God as a heavenly entity that watches over everyone like Daniel Chodowiecki’s The Eye of Providence (1787).


Johann Paul Pöhlmann. Gottes Augen schauen auch in die heimlichen Winkel (English: “God’s eyes also look into the secret angles”), Kupferstich: 135 x 165 mm, 1809

This view of God as a watcher later evolved into a sort of big brother figure like in Gottes Augen schauen auch in die heimlichen Winkel (English: “God’s eyes also look into the secret angles”), a copperplate by Johann Paul Pöhlmann from 1809. Here the all seeing eye is essentially a surveillance camera, encouraging potential law breakers to heed the warning that their bad deeds won’t go unpunished. This sort of indoctrination was easily transferred to messages from political bodies wishing to keep the behavior of their citizens inline as nation states emerged.

Watching You, Watching Me. A Photographic Response to Surveillance shows works by 10 contemporary artists using photography to address surveillance. This exhibition was sponsored by the Open Society Foundation.


Andrew Hammerand. The New Town. 2013.

In Andrew Hammerand’s The new town (2013), the artist captured surveillance images from a camera placed in the center of an idealized planned community in the American Midwest. These are haunting and beautiful reminders of the gaze of the all seeing eye, and the inability to escape from it in even the most ideal locations.


Julian Roeder. Thermal Imaging Camera, Northern Greece, 2012, Archival Pigment Print, 152 x 109 cm


Julian Roeder. Tracking Dog, Northern Greece, 2012, Archival Pigment Print, 152 x 109 cm

Julian Roeder’s Mission and Task (2012) consists of images of the technology and people working in contemporary surveillance at the border of the Schengen zone.


Mari Bastashevski & Privacy International. Private Interests. 2014

Mari Bastashevski & Privacy International presented photographs and evidence of Central Asian governments use of electronic surveillance technologies to spy on activists and journalists, domestically and abroad, in order to clamp down on dissent and to reinforce their political
control. This installation included the stories of several Uzbekistan journalists and lawyers who were surveilled, intimidated, and threatened with prison for their work. This project is covered in detail in this PDF.



Hasan Elahi. Thousand Little Brothers.

In response to questioning by the FBI in suspicious post-9/11 USA, Hasan Elahi created Thousand Little Brothers which includes 32,000 photographs he made during his ongoing project to give the FBI every detail of his life, especially the mundane.

Both exhibitions are up until 2 July, 2017.

Stasi / Facebook / Big Data DAAD Day 26 – Gedenkstätte Berlin-Hohenschönhausen

This week has been full of work on this project, including research in the BStU reading room, and making photographs at the archives at the former headquarters and the Stasi Museum. Today, for the first time, I visited The Memorial to Berlin-Hohenschönhausen, a large former prison complex which includes a former Stasi detention and interrogation center. Berlin-Hohenschönhausen began as a Nazi barrack camp for prisoners of war during WWII. After the fall of Nazi Germany it was used by the Soviet NKVD (the secret police) as a camp for interning political prisoners and former Nazis.

The facility was taken over by the GDR Ministry for State Security (MfS) in 1951. For nearly 40 years the MfS used part of it as a secret (it was not located on official city maps) detention and interrogation center. Here, over 10,000 political prisoners were detained and subjected to physical and psychological torture to gain information in preparation for sentencing by the courts.

The MfS staff who worked here were trained at the “Stasi University” (Potsdam) to destabilize and destroy the personality of their prisoners. Their methods included sleep deprivation, isolation, and other harassment that left the prisoners psychologically damaged. They also made references to prisoners’ loved ones, spouses and children, in order to coerce confessions and encourage cooperation.

The methods for social isolation were particularly effective. Prisoners were often kidnapped and placed in a cage inside a van disguised as a delivery vehicle. The vehicle would drive around endlessly to disorient the prisoner and then arrive at a special closed-off garage in the detention center.

Once inside the buildings an elaborate series of electronic circuits controlled red and green lights in the hallways to direct the flow of prisoners and make certain they would never have the chance to encounter one another.

This last note offers an important comparison to today’s social networks and surveillance capitalism methodologies. In social networks, the organization benefits from the flow of information between participants. They want to stimulate their users to interact and continue generating profit by using the site, producing data, and viewing ads. The Stasi’s torture of political prisoners, on the other hand, intentionally removed all stimulation and human contact. The sensory deprivation and isolation that prisoners experienced in their cells dehumanized, weakened, and prepared them to to give up information the Stasi wanted.

Our information society has transformed the way we think about private information. While we freely give it up in exchange for social capital, companionship, and the serve the impulse to communalize our experiences, it does not mean that the social networks and corporations who deal in that data are not responsible when our information is misused. The recent revelation that the U.K. defense contractor turned data dealer, BAE, sold cyber-surveillance tools to Arab states is proof of this. BAE peddled their sophisticated total surveillance system, with a generic yet telling name, “Evident” (a marketing variation of “criminal evidence”?) to six Middle Eastern countries previously criticised for repressing their citizens. These countries can now use the tool to scour the internet for sites, blogs, and social networks for information on anyone they consider an opponent. A former BAE employee put it like this:

“You’d be able to intercept any internet traffic,” he said. “If you wanted to do a whole country, you could. You could pin-point people’s location based on cellular data. You could follow people around. They were quite far ahead with voice recognition. They were capable of decrypting stuff as well.”

Digital information is inherently insecure. It’s easy to surveill, copy, and duplicate by design. Equally, it is easy to reimplement a surveillance software system that is designed to be sold to do so. The Stasi was a terrible organization, but their work was generally confined to the GDR. There was collaboration across the USSR satellite police organizations, but their work wasn’t on a global market. Looking back at the Stasi provides this evidence; The employees and stockholders of any company that benefits from the sale of such a system to inflict human rights abuses are as guilty as the Stasi employees who worked in the offices and never directly inflicted damage. Surveillance capitalism isn’t just about the profit generated by social network ads, it includes the manufacture and sales of systems like Evident, and all those who derive profit from such decentralized institutions.

Stasi / Facebook / Big Data DAAD Day 25 – Automation in Stasi postal surveillance: Opening letters with sensitive materials using hot air


Heißluftgebläse. Credit: Museum in der Runde Ecke

The next artifact in my list is the “Heißluftgebläse”—a hot air blower constructed by the Stasi for opening envelopes and packages sealed with rubberized self-adhesive tape. Unlike the various steam methods the Stasi employed, which only worked on water-based glues and could potentially damage the contents of letters, this machine was used to focus hot, dry air directly to the flap in order to soften and release the adhesive.

Like other “hacked” machines constructed for postal surveillance the Stasi Operative-Technical Sector fashioned this blower using off the shelf, reused, and hand-made parts. In this case, they adopted a commercially-available vacuum cleaner to force air through a heating cartridge and down a steel tube towards a platform where workers laid envelopes. At the end of the tube there is a metal apparatus that could be used to focus the hot air and gently pry the envelope flap open. Impressively, the Runde Ecke index states this method be used to open up to 150 letters per hour.


Heißluftgebläse. April and July Credit: Wikepedia user Appaloosa

I’m particularly interested in the different representations of this machine. In these two photographs, both posted by the same Wikipedia user in 2008 just months apart, it appears there were two separate machines on exhibit. This is suggested by comparing the scratches on the base and stand. Yet, while they are clearly different, they are strangely positioned in exactly the same location—an exhibition at the Museum in der Runde Ecke which contains objects found in the Leipziger district administration for state security (BVfS)—with other obvious differences. In the 22 April photo (left) the vacuum cleaner is still attached, showing how the machine was originally used. But in the 27 July image the vacuum and tape have been removed. Was it needed for cleaning in the museum? Where could it have gone? There are additional small differences that point to these being distinct machines, the most noticeable of which is that a different bracket is holding the heating apparatus.

It’s thanks to deviations like this that the telling of the story of Stasi postal surveillance continues to evolve. Aesthetic decisions that seem inconsequential—like a museum staff member replacing the device with the yellowing tape with the one that doesn’t look as disheveled—change how we are able to perceive the past. Here, those moments which make the economic and material constraints the Stasi OTS experienced while hacking and iterating across their analog surveillance devices become less clear. While, thanks to the BTsU we are fortunate to be able to dissect and interpret the methods of the Stasi, it is not possible to collect everything. This unfortunately obscures the parallels with our 21st century hacker culture that, like the Stasi, have grown into immense organizations intent on perfecting and perpetuating their power through surveillance.

  1. Heißluftgebläse zum öffnen von gummierten selbstklebenden Briefen (Inventory no. 00020). Objekt- und Fotodatenbank Online im Museum in der Runden Ecke. Accessed June 13, 2017.
  2. BStU. Die Stasi und das Postgeheimnis. Accessed June 10, 2017.

Stasi / Facebook / Big Data DAAD Day 24 – Automation in Stasi postal surveillance: Opening letters with sensitive materials using cold steam


Credit: BStU

Since hot steam could damage sensitive contents like photographs, the Stasi used the C-83 (“Cellophan 83”) “cold steam” generating device to open those letters. Produced in Hungary, the C-83 used a special film (cellophane) stretched over a tray filled with distilled water. Up to 25 letters could be placed on the film, flaps down, so that when the water in the tray was heated to 50ºC the “cold steam” diffused through the film and dissolved the glue on the envelope flaps.

The procedure was time-consuming because cold steam dissolved the glue at a much slower rate than the hot steam method. Once the glue started to soften, workers had could only lift the letters off the foil for a short time in order to carefully open them using a special spatula.


Credit: BStU

It is unclear if this is an original mass-produced device or is a product modified by the Stasi OTS. It is constructed of painted sheet metal, cast plastic, and other materials that suggest it was primarily constructed in a factory, though it is possible the Stasi made modifications to assist in letter-opening.


Credit: Museum in der Runde Ecke


Credit: Museum in der Runde Ecke


Credit: Museum in der Runde Ecke

  1. Kaltdampf-Öffnungsgerät zum öffnen von Briefen (Zellofanbodengerät) (Inventory no. 00006). Objekt- und Fotodatenbank Online im Museum in der Runden Ecke. Accessed June 13, 2017.
  2. BStU. Die Stasi und das Postgeheimnis. Accessed June 10, 2017.