Archiving web sites

Posted Sep 25, 2018 14:48 UTC (Tue) by anarcat (subscriber, #66354)
Parent article: Archiving web sites

As usual, here's the list of issues and patches generated while researching this article:

fix broken link to specification in the WARC website
sample Apache configuration for pywb
make job status less chatty in ArchiveBot
Debian packaging of the ia commandline tool

I also want to personally thank the folks in the #archivebot channel for their assistance and letting me play with their toys.

Archiving web sites

Posted Oct 4, 2018 17:58 UTC (Thu) by anarcat (subscriber, #66354) [Link]

As it turns out, I couldn't stop working on this topic and opened two more PRs upstream after submitting WARC files to the internet archive:

mention collections in the ia documentation
fix warnings in docs builds of ia

The Pamplemousse crawl is now available on the Internet Archive, it might end up in the wayback machine at some point if the Archive curators think it is worth it.

Another example of a crawl is this archive of two Bloomberg articles which the "save page now" feature of the Internet archive wasn't able to save correctly (but webrecorder.io) could! Those pages can be seen in the web recorder player to get a better feel of how faithful a WARC file really is.