Research shows 25% of web pages posted between 2013 and 2023 have vanished. A few organisations are racing to save the echoes of the web, but new risks threaten their very existence.
It’s possible, thanks to surviving fragments of papyrus, mosaics and wax tablets, to learn what Pompeiians ate for breakfast 2,000 years ago. Understand enough Medieval Latin, and you can learn how many livestock were reared at farms in Northumberland in 11th Century England – thanks to the Domesday Book, the oldest document held in the UK National Archives. Through letters and novels, the social lives of the Victorian era – and who they loved and hated – come into view.
But historians of the future may struggle to understand fully how we lived our lives in the early 21st Century. That’s because of a potentially history-deleting combination of how we live our lives digitally – and a paucity of official efforts to archive the world’s information as it’s produced these days.
However, an informal group of organisations are pushing back against the forces of digital entropy – many of them operated by volunteers with little institutional support. None is more synonymous with the fight to save the web than the Internet Archive, an American non-profit based in San Francisco, started in 1996 as a passion project by internet pioneer Brewster Kahl. The organisation has embarked what may be the most ambitious digital archiving project of all time, gathering 866 billion web pages, 44 million books, 10.6 million videos of films and television programmes and more. Housed in a handful of data centres scattered across the world, the collections of the Internet Archive and a few similar groups are the only things standing in the way of digital oblivion.
I’m part of the problem, a tiny bit. For altruistic reasons - ok more like “I’m kinda weird, maybe this will make people on IRC like me more” reasons - I ran mspencer.net and hosted web pages for people for free. Ended up with web content for around 100 people, and they weren’t all just using it as a drop box. (Older than wikipedia.org by 199 days, woo!)
Hosted on ancient hardware, nothing even remotely approaching a modern security architecture, I eventually left it to run un-maintained until the IDE HDD died. More recently I got the data off of it. (Heads unstuck themselves while in a cardboard box for a decade? Dunno.) But I don’t know how to get everything back online in a safe way.
I’m a proper software engineer now, I can kinda see how work handles securely hosting web services. Now just throwing everything together on one box feels too lazy and insecure. But I can’t figure out a reasonable security architecture to use. I thought I had one, but I failed to account for VM jackpotting attacks. And it feels like it takes me a month to do what a competent ops person can do in a day.
But that’s a discussion for a different comment section.
Last time I went snooping:
15 installs of phpbb, which would require work to put back online as their communities are of course gone. Remove spam, undo defacement, etc.
7 installs of Dormando’s Oekaki BBS Clone
5 installs of WonderCatStudio BBS
4 installs of OekakiPotato / RanmaGuy etc.
and several users who just used php to ‘include’ headers and table of contents page parts.
(Yes I was quite the weeb. Still am, but I was one too. :-) )
If this was my problem to solve, I would host it internally, as-is, on a virtual machine of your choice, then create a a static html mirror version from the public information and put that up on AWS S3 as a static website.
But I can’t figure out a reasonable security architecture to use. I thought I had one, but I failed to account for VM jackpotting attacks. And it feels like it takes me a month to do what a competent ops person can do in a day.
You’re overthinking it, just secure things enough that you’re ahead of the script kiddies automated scan tools (which isn’t a lot tbh)
The people with actual real skill don’t care about you, they’d rather go after juicy targets, like companies or politicians or rich people
