Anubis doesn't seem to be working anymore here [LWN.net]

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 14:02 UTC (Sat) by dskoll (subscriber, #1630)
In reply to: Anubis doesn't seem to be working anymore here by danielbaumann
Parent article: An update on the scraper situation

Yes, I do the same Basic Auth trick with the username and password disclosed on the front page and it seems to be working very well for now. I check the user agent and if it's git, I don't require the authentication, so a git clone works without authentication. At some point, I suspect the scrapers will catch on and we'll be one step further along the arms race. 🙁

to post comments

CGit accesses are indeed problematic

Posted Jul 13, 2026 3:31 UTC (Mon) by zaitseff (subscriber, #851) [Link]

To add what some have already said: I also found that incessant accesses to my CGit repositories was causing significant load on my 14-year-old 1RU server, with well over 50,000 accesses per day, over and over again.

My first thought was to put in HTTP Basic Authentication, but that just made the bots try (and fail) more often: it got to the point where I was seeing up to half a million IP addresses trying to access the repos each and every day! The bots' motto must be "must try harder"!

I ended up doing the following:

1. Returning a 410 Gone response to my CGit instance,
2. Moving the CGit frontend to another hostname and URL prefix (both previously unadvertised), and
3. Putting HTTP Basic Authentication in front of those new URLs.

Finally:

4. Git repos over HTTPS also need Basic auth, unless it comes from a recognisable Git client.

After three months, I've seen accesses on the old URLs go from 500,000 to a trickle of 300-500 per day. I can live with that!