Anubis doesn't seem to be working anymore here [LWN.net]

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 3:32 UTC (Sat) by danielbaumann (subscriber, #38804)
In reply to: Anubis doesn't seem to be working anymore here by koverstreet
Parent article: An update on the scraper situation

I started to use basic auth with a "dummy" user and password that is written on the webpage. It's unexpected/inconvenient for non-regular/first-time/one-time-only human visitors, but it works for now.

Anubis doesn't seem to be working anymore here

Posted Jul 11, 2026 14:02 UTC (Sat) by dskoll (subscriber, #1630) [Link] (1 responses)

Yes, I do the same Basic Auth trick with the username and password disclosed on the front page and it seems to be working very well for now. I check the user agent and if it's git, I don't require the authentication, so a git clone works without authentication. At some point, I suspect the scrapers will catch on and we'll be one step further along the arms race. 🙁

CGit accesses are indeed problematic

Posted Jul 13, 2026 3:31 UTC (Mon) by zaitseff (subscriber, #851) [Link]

To add what some have already said: I also found that incessant accesses to my CGit repositories was causing significant load on my 14-year-old 1RU server, with well over 50,000 accesses per day, over and over again.

My first thought was to put in HTTP Basic Authentication, but that just made the bots try (and fail) more often: it got to the point where I was seeing up to half a million IP addresses trying to access the repos each and every day! The bots' motto must be "must try harder"!

I ended up doing the following:

1. Returning a 410 Gone response to my CGit instance,
2. Moving the CGit frontend to another hostname and URL prefix (both previously unadvertised), and
3. Putting HTTP Basic Authentication in front of those new URLs.

Finally:

4. Git repos over HTTPS also need Basic auth, unless it comes from a recognisable Git client.

After three months, I've seen accesses on the old URLs go from 500,000 to a trickle of 300-500 per day. I can live with that!

Anubis doesn't seem to be working anymore here

Posted Jul 12, 2026 11:59 UTC (Sun) by alx.manpages (subscriber, #145117) [Link]

Heh! Nice trick!

I gave up with similar stuff, but maybe I should give this one a try.

Currently, I serve my cgit under a different server_name, so if the user passes the appropriate domain name in the header fields, they're allowed, but otherwise they can only see the basic static site. I tell the secret domain name to users that I want to give access to, and they put it in their hosts file, with the same IP of my public DNS name (so they can update it with whatever the DNS says in the future).

Anubis doesn't seem to be working anymore here

Posted Jul 12, 2026 18:14 UTC (Sun) by ecm (subscriber, #129897) [Link]

We also did this between 2025 October and 2026 January for our hgweb, though we've switched to Anubis with some success since.

Blogged about it at https://pushbx.org/ecm/dokuwiki/blog/pushbx/2026/0616_upd...