Anubis doesn't seem to be working anymore here [LWN.net]

Anubis doesn't seem to be working anymore here

Posted Jul 10, 2026 22:34 UTC (Fri) by koverstreet (subscriber, #4296)
Parent article: An update on the scraper situation

I started seeing a ton of AI crawlers hammer all my git endpoints, and they're going right past the anubis checks; it seems a least some crawlers are using full web browsers that can do the proof of work.

Ouch.

So I'm back to a script that I run regularly that just scrapes the hammering IPs from the nginx log and iptables blocks them...

to post comments

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 0:21 UTC (Sat) by joey (guest, #328) [Link] (9 responses)

I'm coming to the conclusion that it doesn't make sense to have a whole public gitweb or cgit or similar on the web anymore. There are a few pages of that are valuable to provide to my users, like the most recent commit log and the current file tree, and a things like commits that I link to specifically from blog posts. But the ability to explore deeply through the whole git history is a marginal value, and the value proposition for that has overall gone negative. The interested user can make their own clone and use their own tools on it.

LWN's mailing list archives may have a similar value distribution.

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 0:58 UTC (Sat) by koverstreet (subscriber, #4296) [Link] (6 responses)

I use cgit pretty frequently, and my users are perusing it regularly too, so it's pretty painful here.

Realistically, the solution needs to be some kind of throttling baking into the webserver, and a more efficient cgit implementation with some caching would help a lot. I've hit quite a few perf issues with git, I'm hoping now that they're starting to use Rust in the core codebase that'll make performance work easier.

Definitely not excited by the prospect of yet more sysadmin work, though.

And. If web access to too many git repos gets shut down I have to imagine the AI companies would switch to just chain cloning repos with no caching. Everything they're doing is just really anti social.

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 3:32 UTC (Sat) by danielbaumann (subscriber, #38804) [Link] (5 responses)

I started to use basic auth with a "dummy" user and password that is written on the webpage. It's unexpected/inconvenient for non-regular/first-time/one-time-only human visitors, but it works for now.

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 5:27 UTC (Sat) by koverstreet (subscriber, #4296) [Link]

Heh, the cat and mouse games are getting out of hand...

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 14:02 UTC (Sat) by dskoll (subscriber, #1630) [Link] (1 responses)

Yes, I do the same Basic Auth trick with the username and password disclosed on the front page and it seems to be working very well for now. I check the user agent and if it's git, I don't require the authentication, so a git clone works without authentication. At some point, I suspect the scrapers will catch on and we'll be one step further along the arms race. 🙁

CGit accesses are indeed problematic

Posted Jul 13, 2026 3:31 UTC (Mon) by zaitseff (subscriber, #851) [Link]

To add what some have already said: I also found that incessant accesses to my CGit repositories was causing significant load on my 14-year-old 1RU server, with well over 50,000 accesses per day, over and over again.

My first thought was to put in HTTP Basic Authentication, but that just made the bots try (and fail) more often: it got to the point where I was seeing up to half a million IP addresses trying to access the repos each and every day! The bots' motto must be "must try harder"!

I ended up doing the following:

1. Returning a 410 Gone response to my CGit instance,
2. Moving the CGit frontend to another hostname and URL prefix (both previously unadvertised), and
3. Putting HTTP Basic Authentication in front of those new URLs.

Finally:

4. Git repos over HTTPS also need Basic auth, unless it comes from a recognisable Git client.

After three months, I've seen accesses on the old URLs go from 500,000 to a trickle of 300-500 per day. I can live with that!

Anubis doesn't seem to be working anymore here

Posted Jul 12, 2026 11:59 UTC (Sun) by alx.manpages (subscriber, #145117) [Link]

Heh! Nice trick!

I gave up with similar stuff, but maybe I should give this one a try.

Currently, I serve my cgit under a different server_name, so if the user passes the appropriate domain name in the header fields, they're allowed, but otherwise they can only see the basic static site. I tell the secret domain name to users that I want to give access to, and they put it in their hosts file, with the same IP of my public DNS name (so they can update it with whatever the DNS says in the future).

Anubis doesn't seem to be working anymore here

Posted Jul 12, 2026 18:14 UTC (Sun) by ecm (subscriber, #129897) [Link]

We also did this between 2025 October and 2026 January for our hgweb, though we've switched to Anubis with some success since.

Blogged about it at https://pushbx.org/ecm/dokuwiki/blog/pushbx/2026/0616_upd...

Dumb transport is very efficient

Posted Jul 11, 2026 4:04 UTC (Sat) by DemiMarie (subscriber, #164188) [Link]

The dumb transport just requires a static file server, which means it is vastly more efficient on the server side unless it (for some reason) requires far more bandwidth.

Anubis doesn't seem to be working anymore here

Posted Jul 12, 2026 9:08 UTC (Sun) by danc (subscriber, #74798) [Link]

A while ago I replaced my cgit installations with stagit. It generates a basic view of your git repositories as static HTML pages. You get a page for every commit with diffs, and a complete log of the master branch. Not much else. It's perfect for putting small personal projects online, and the cost to serve the static HTML is negligible.

https://codemadness.org/stagit.html

There are quite a few forks of it floating around with extra features, like rendering Markdown READMEs and so forth.