If this actually works... [LWN.net]

If this actually works...

Posted May 30, 2026 4:26 UTC (Sat) by Cyberax (✭ supporter ✭, #52523)
In reply to: If this actually works... by dskoll
Parent article: Nesbitt: Protestware for coding agents

I'm thankfully seeing AI bots' traffic slowly subsiding. I guess they just scraped all they could and/or fixed their damn crawlers enough to not recurse into every gitweb page.

If this actually works...

Posted May 30, 2026 12:39 UTC (Sat) by pizza (subscriber, #46) [Link] (4 responses)

> I'm thankfully seeing AI bots' traffic slowly subsiding.

Lucky you. Every time I see a small downtick, it's followed by an even larger jump as some new distributed crawler botnet performs another massive DDoS.

The most recent one happened less than a week ago, nearly entirely consisting of residential or cellular IP blocks, making no more than two requests per source IP. over 600K requests over only a few hours.

> I guess they just scraped all they could and/or fixed their damn crawlers enough to not recurse into every gitweb page.

Each individual scraper, maybe. Except for every one that "finishes" three more spin up and start from scratch.

If this actually works...

Posted Jun 3, 2026 15:01 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

> The most recent one happened less than a week ago, nearly entirely consisting of residential or cellular IP blocks, making no more than two requests per source IP. over 600K requests over only a few hours.

I must be too naive but I'm really puzzled by this... I have no illusion that big AI players are "ethical" (ethics dies a while ago) and I know they are constantly flirting with legal lines but... using botnets, really? That sounds like they could lose way too much if that was proven in court. So I can imagine only this:

- Legit BigTech companies are delegating the dirty work to other, shadowy companies, pretending not to know how the latter operate
- Buried in the fine print of Terms and Conditions that no one reads, there's a clause that lets their client software double as a crawler.
- Other?

If this actually works...

Posted Jun 3, 2026 16:35 UTC (Wed) by corbet (editor, #1) [Link]

Both of those conditions are true (and certainly "other" as well). Look up a company called Bright Data sometime.

If this actually works...

Posted Jun 3, 2026 16:48 UTC (Wed) by pizza (subscriber, #46) [Link]

> I must be too naive but I'm really puzzled by this... I have no illusion that big AI players are "ethical" (ethics dies a while ago) and I know they are constantly flirting with legal lines but... using botnets, really?

Who said anything about it being the "big AI players" ?

Everyone and their dog is independently trying to scrape everything these days.

If this actually works...

Posted Jun 3, 2026 17:18 UTC (Wed) by smurf (subscriber, #17840) [Link]

Umm, in this case, why not attach the cookie to a specific IP address?

Scraper traffic

Posted May 30, 2026 14:48 UTC (Sat) by corbet (editor, #1) [Link]

We saw a major drop-off in scraper traffic after Google took down IPIDEA, but it has been working its way back up again as they rebuild their botnets. Such nice people, these AI folks.

If this actually works...

Posted May 31, 2026 15:43 UTC (Sun) by ecm (subscriber, #129897) [Link] (2 responses)

Had the same problem with our hgweb, for which I'm not hopeful the crawlers will ever become smart enough. Initially in 2025 October we gated the hgweb behind a password check (with publicly known conditions to allow access), though since earlier this year we switched to anubis.

Also, on our server we provide tarballs that contain all hg repos directly copied from the hgweb repo directory, allowing a much lighter way to read all the repos with their entire history. I'm very much not against scraping, I just wish they wouldn't DDoS our server in the process.

If this actually works...

Posted May 31, 2026 15:51 UTC (Sun) by mb (subscriber, #50428) [Link] (1 responses)

>we gated the hgweb behind a password check (with publicly known conditions to allow access), though since earlier this year we switched to anubis.

I'm wondering why you changed that.
To me anubis seems to be a worse solution, because it's much more complex and it does prevent some legitimate uses like users using simple scripts to fetch things.
For example anubis breaks RSS feeds that are provided by repository web interfaces. I had subscribed to some Arch git RSS feeds to monitor things, but that broke when they switched to anubis. With http basic auth this would be trivial to fix on my side in the RSS reader. With anubis I'm not so sure what to do.

If this actually works...

Posted May 31, 2026 19:19 UTC (Sun) by zdzichu (subscriber, #17118) [Link]

You contact the provider of RSS feeds. These are expected to be consumed by automation, not people. Putting RSS feeds behind bot-protection is a misconfiguration.