CGit accesses are indeed problematic [LWN.net]

CGit accesses are indeed problematic

Posted Jul 13, 2026 3:31 UTC (Mon) by zaitseff (subscriber, #851)
In reply to: Anubis doesn't seem to be working anymore here by dskoll
Parent article: An update on the scraper situation

To add what some have already said: I also found that incessant accesses to my CGit repositories was causing significant load on my 14-year-old 1RU server, with well over 50,000 accesses per day, over and over again.

My first thought was to put in HTTP Basic Authentication, but that just made the bots try (and fail) more often: it got to the point where I was seeing up to half a million IP addresses trying to access the repos each and every day! The bots' motto must be "must try harder"!

I ended up doing the following:

1. Returning a 410 Gone response to my CGit instance,
2. Moving the CGit frontend to another hostname and URL prefix (both previously unadvertised), and
3. Putting HTTP Basic Authentication in front of those new URLs.

Finally:

4. Git repos over HTTPS also need Basic auth, unless it comes from a recognisable Git client.

After three months, I've seen accesses on the old URLs go from 500,000 to a trickle of 300-500 per day. I can live with that!