If this actually works... [LWN.net]

If this actually works...

Posted May 31, 2026 15:43 UTC (Sun) by ecm (subscriber, #129897)
In reply to: If this actually works... by Cyberax
Parent article: Nesbitt: Protestware for coding agents

Had the same problem with our hgweb, for which I'm not hopeful the crawlers will ever become smart enough. Initially in 2025 October we gated the hgweb behind a password check (with publicly known conditions to allow access), though since earlier this year we switched to anubis.

Also, on our server we provide tarballs that contain all hg repos directly copied from the hgweb repo directory, allowing a much lighter way to read all the repos with their entire history. I'm very much not against scraping, I just wish they wouldn't DDoS our server in the process.

to post comments

If this actually works...

Posted May 31, 2026 15:51 UTC (Sun) by mb (subscriber, #50428) [Link] (1 responses)

>we gated the hgweb behind a password check (with publicly known conditions to allow access), though since earlier this year we switched to anubis.

I'm wondering why you changed that.
To me anubis seems to be a worse solution, because it's much more complex and it does prevent some legitimate uses like users using simple scripts to fetch things.
For example anubis breaks RSS feeds that are provided by repository web interfaces. I had subscribed to some Arch git RSS feeds to monitor things, but that broke when they switched to anubis. With http basic auth this would be trivial to fix on my side in the RSS reader. With anubis I'm not so sure what to do.

If this actually works...

Posted May 31, 2026 19:19 UTC (Sun) by zdzichu (subscriber, #17118) [Link]

You contact the provider of RSS feeds. These are expected to be consumed by automation, not people. Putting RSS feeds behind bot-protection is a misconfiguration.