Fighting the AI scraperbot scourge

> avoid forcing readers to enable JavaScript

You already generate a different page for logged-in readers, right? Would it be easy to simply omit the JavaScript countermeasures for subscribers?

(And yes, that attention is appreciated)

Thank you.

1- A fake index.html asking for simple math challenge not JS challange then goto real index.html
dont forget w3m like surfers as vieving images not available for them so challange can't contain images.
math challenge per IP, math challenge per EPOCH, limit retry so on ... combine try hard if you need.

2- Put *just only* old/some type contex behind such challenge.

3- Reject all clients other than that dont have some custom "http header" or some special "user agent" value.
Maybe you put that "specials" to your fake index.html

4- Every link on every page must be suffixed(concetanated) with some "special keyword" only visible to human *eyes* to access.

5- So on, so on ...

forgive me that being *too* clever and my english:)

... which, admittedly, it should probably actually say somewhere in the site footer and in the FAQ, not just on some old 'about the site' pages.

During the subscriber period (one week after the first weekly edition in which the article appears), they are plain old all-rights-reserved, though.

The GPL itself might not be relevant, but surely we can craft a specific license for that usage, one that 1) plays nice with all other licenses, including non-free ones 2) is highly viral by nature.

The goal would not be to make all published text freeely available, but to either force the distribution of the dataset in which the text is included or, if the dataset contains copyrighted materials that cannot be redistributed, forbid the integration of the work. The license clauses may only trigger when the work is integrated in a dataset and the license text could even be written so that it also contaminate adjacent datasets that are used in conjunction with a dataset already covered by this license.

Of couse, once the dataset is contaminated, you'd also better make sure that the models created using this dataset shall be made free (as in speech). A license that creates new freedoms for users - that do not seem impossible to do, doesn't it? A "Dataset Freedom License".

(Also, make sure that the type of work is not enforced by the license, so that you could protect anything with it, ranging from code to images to songs...)

With such a license, the scrapping bots are still an annoyance, but any indication that a particularly covered work ended in a specific dataset could be the cause of some spectacular court actions :) And I thinks that can make the dataset creator think twice.

For better or for worse, this is not *too* far removed from how courts would actually analyze the situation (except that courts in most jurisdictions would remove the "in its entirety" part and replace it with a big complicated gray area). See e.g. https://en.wikipedia.org/wiki/Substantial_similarity

Except that that is - sort of - codified in law! After all, what GenAI scrapers are doing is *exactly* *the* *same* as you reading a book. The book is copyright, and you've just copied that material from the pages of the book into your brain! And if you can't regurgitate that book word for word, then that MAY be fine.

Which is - in Europe at least - EXACTLY how AI scrapers are treated. If the *output* bears little resemblance to the training data, then copyright is not violated. If it's regurgitated exactly, then copyright is violated. European law explicitly says slurping the data IN is not a copyright violation. It says nothing about regurgitating it ... and it says nothing about other laws, like trespassing to gain access to it.

The problem of course, is the people who don't give a damn about wasting OTHER PEOPLE's money, like LWN's. In other words, people who ignore restrictions like robots.txt. I think European law treats that like a door-lock - "don't come in without permission", but the trouble is enforcing it :-(

Cheers,
Wol

Well if not paywalled then very old articles could perhaps be rate limited, aka not per ip but for all articles as such, ofc problem then is that the bots will fill up the limit preventing the real readers from accessing the old articles... Yeah this stuff is hard.

That way, direct links work for archival purposes, but if you want to navigate the site, you need to log in.

Heck, based on https://lwn.net/Articles/1010764/ (farnz's CFAA comment), it wouldn't even need to be a real login, just a page where you agree to terms of service and get a cookie recording that after POSTing your agreement.

People don't have to create a login, remember a password, any of that crap. Yes it's a hassle, but if you change it every six weeks (maybe less) with an eight week "remember me" cookie, it's not much grief to real people.

Cheers,
Wol

This is a site full of pedantic nerds. So yes, it was necessary.

On the other hand, the idea of class A/B/C networks is still taught in networking courses, even though it has been irrelevant and unhelpful for decades now. The sooner we stop referencing it, the better.

Firstly LWN is paying for that bandwidth. And secondly, what happens if the scraping overwhelms the bandwidth? Ancient case in point - a company on an ISDN link was filtering its mail based on message header. Even downloading JUST the header, spam was arriving at their ISP faster than they could bounce it.

Cheers,
Wol

For those who wish to avoid JavaScript at any cost — we do also take payment for subscriptions by check, although only on US banks.

But you're right that if things get bad enough, we may be forced to come up with some kind of CAPTCHA system. Let's hope things don't get to that point.

You might argue that the attackers do not care, because they are (presumably) using a botnet anyway. But even if they are stealing the resources, they still have to allocate them, and they will likely prefer to target sites that do not require a headless browser over sites that do.

...unless you use a botnet, as reported by this article, in which case someone else is paying for the internet link (and potentially for the storage and processing power to deduplicate and filter and compress the data before sending the valuable parts back to you). Then it becomes another instance of fraud, where the person benefiting from the activity is not the person paying for the activity, and increasing the cost of traffic will have no impact.

I wonder how many executives at telcos even remember toll calling, but I'm sure they'd like to bring back billing based on netflow data or DNS if they can take their cut, in the same way that cable TV based ISPs have reintroduced bundled billing after the migration to streaming using SAML SSO AFAICT. So you could either use a sample based guesstimate that has more room for fraud, or have a strong identity federated with your ISP as a prerequisite for access, requiring SAML/OAuth2/OIDC to handle the accounting for billing. You might end up with a system bifurcated by wealth where "free" data is predatory nonsense trying to defraud / manipulate and paid data is higher quality, although catered to the biases of the more wealthy.

In some ways it represents a regression from the original ideology of the Internet, where it was based on a flat fee to cover the infrastructure costs, and you could use it as much or as little as you like for whatever purpose, but that was assuming a more peer-like relationship between collections of endpoints rather than the more strictly client/server networks we have today, where the cost to be a publisher was fairly low and mostly covered by excess capacity in already-paid-for infrastructure (networks and personal computers/workstations), and there was mechanisms of accountability for abuse, if you got out of pocket they could call your Dean and hold your degree hostage, threatening your potential future lifestyle, until you learned to behave. In a better world instead of pretending that the Internet is some cyberspace separate from governments, a land of FREEDOM (*eagles screaming*), we'd have sensible accountability measures as part of peering so networks are only connected when there is an agreed upon ToS. That might split the network into several zones of influence with small filtered pipes in between but it could more closely align with human behavior and human centered governance systems, eg. if you don't want this kind of scraping and there wasn't already a rule against it then you'd have a clear process to talk to your representative to negotiate a change to the ToS (laws) that could be enforced against those violating community norms.

Many people are laughing now because the mechanisms for feedback and adjustment to the rules by which society operates are so broken right now that what I've described seems like a utopian pipe dream, because the trend has been toward less accountability the more power and wealth you have, when we need stronger accountability for the effects of decisions that ultimately a person(s) somewhere are making. There is a person somewhere setting up a scraping botnet, there is a person somewhere using that tainted data, those people have names and faces, they live in a place with rules of some kind, they should be identified or cut off from the network.

Is that idea the end of anonymity, yeah kind of, it should only be attempted when there are working feedback mechanisms between the people making and enforcing the rules and the people subject to the rules so that there is mutual consent and ways to adjust to keep everyone in harmony and alignment, democracy is one of those mechanisms but there are certain to be many stable and effective ways for human societies to maintain healthy feedback loops.

There is just a limit as to what an individual website can do when the problem is the fundamental rules of the system that don't provide negative feedback on abusive behavior, like a broken TCP or bufferbloat which allows one jumbo flow to starve the rest of the link.

Ok, time to get a second cup of coffee and get off the Internet ;-)