If this actually works... [LWN.net]

If this actually works...

Posted May 30, 2026 12:39 UTC (Sat) by pizza (subscriber, #46)
In reply to: If this actually works... by Cyberax
Parent article: Nesbitt: Protestware for coding agents

> I'm thankfully seeing AI bots' traffic slowly subsiding.

Lucky you. Every time I see a small downtick, it's followed by an even larger jump as some new distributed crawler botnet performs another massive DDoS.

The most recent one happened less than a week ago, nearly entirely consisting of residential or cellular IP blocks, making no more than two requests per source IP. over 600K requests over only a few hours.

> I guess they just scraped all they could and/or fixed their damn crawlers enough to not recurse into every gitweb page.

Each individual scraper, maybe. Except for every one that "finishes" three more spin up and start from scratch.

to post comments

If this actually works...

Posted Jun 3, 2026 15:01 UTC (Wed) by marcH (subscriber, #57642) [Link] (3 responses)

> The most recent one happened less than a week ago, nearly entirely consisting of residential or cellular IP blocks, making no more than two requests per source IP. over 600K requests over only a few hours.

I must be too naive but I'm really puzzled by this... I have no illusion that big AI players are "ethical" (ethics dies a while ago) and I know they are constantly flirting with legal lines but... using botnets, really? That sounds like they could lose way too much if that was proven in court. So I can imagine only this:

- Legit BigTech companies are delegating the dirty work to other, shadowy companies, pretending not to know how the latter operate
- Buried in the fine print of Terms and Conditions that no one reads, there's a clause that lets their client software double as a crawler.
- Other?

If this actually works...

Posted Jun 3, 2026 16:35 UTC (Wed) by corbet (editor, #1) [Link]

Both of those conditions are true (and certainly "other" as well). Look up a company called Bright Data sometime.

If this actually works...

Posted Jun 3, 2026 16:48 UTC (Wed) by pizza (subscriber, #46) [Link]

> I must be too naive but I'm really puzzled by this... I have no illusion that big AI players are "ethical" (ethics dies a while ago) and I know they are constantly flirting with legal lines but... using botnets, really?

Who said anything about it being the "big AI players" ?

Everyone and their dog is independently trying to scrape everything these days.

If this actually works...

Posted Jun 3, 2026 17:18 UTC (Wed) by smurf (subscriber, #17840) [Link]

Umm, in this case, why not attach the cookie to a specific IP address?