I had to password-protect my sites [LWN.net]

I had to password-protect my sites

Posted Aug 29, 2026 16:32 UTC (Sat) by logang (subscriber, #127618)
In reply to: I had to password-protect my sites by dskoll
Parent article: Ryabitsev: Creepy crawlies

This was my first thought too. If we need an account to access everything on a website that could be relatively easy for a human to setup once and use indefinitely. Then a random bot needs a unique machine and a unique email address to register an account. Rate limiting by account would prevent multiple distributed bots from using the same account. Vanilla clones could still be available to users without accounts they just don't get access to the fancy HTML browsing features.

If the bots get more clever and do create accounts, add an hour delay in initial account setup. Mildly annoying for first time users but pretty difficult for bots trying to scrape from multiple accounts.

to post comments

I had to password-protect my sites

Posted Aug 29, 2026 17:08 UTC (Sat) by mb (subscriber, #50428) [Link] (3 responses)

I have implemented this by always automatically setting a cookie on entry from the main site entry point and then checking this cookie on entry to restricted sub-sites (mainly cgit) that cause heavy load.
So this is essentially an automatically granted and stored account, if the service is entered from the "human side".

The only annoyance is that a human entering the restricted service via direct link gets presented a "login" button first that must just be pressed, if the cookie was cleared.

So far this works pretty well since many months ago.
AI scrapers are not (yet) clever enough to circumvent this.

I had to password-protect my sites

Posted Aug 31, 2026 12:51 UTC (Mon) by marcH (subscriber, #57642) [Link] (2 responses)

> I have implemented this by always automatically setting a cookie on entry from the main site entry point and then checking this cookie on entry to restricted sub-sites (mainly cgit) that cause heavy load.
> So this is essentially an automatically granted and stored account, if the service is entered from the "human side".

This is "half-anonymous" access and we need to perfect and generalize this everywhere (I don't know how)

Anonymous access is dead because bots just killed it.

Authenticated access is a privacy and complexity nightmare.

We desperately need something in the middle.

> AI scrapers are not (yet) clever enough to circumvent this.

As long as they are stupid enough to query git objects one by one...

I had to password-protect my sites

Posted Aug 31, 2026 16:08 UTC (Mon) by mb (subscriber, #50428) [Link] (1 responses)

Actually, the cookie is fully anonymous. It is identical for everybody. There's not really an account created on the server side. I was just paraphrasing to make the mechanism more clear.

I had to password-protect my sites

Posted Aug 31, 2026 16:24 UTC (Mon) by marcH (subscriber, #57642) [Link]

By all means, please find a better name than "half-anonymous" for that middle-ground!

I had to password-protect my sites

Posted Aug 30, 2026 17:59 UTC (Sun) by dskoll (subscriber, #1630) [Link]

I don't do anything clever. I list the username and password on my web site where a human user can easily find it. If for some reason they enter bad credentials, the error page tells them what the username and password should be.

I want to make it as easy as possible for humans, and so far, the AI scrapers haven't caught on to this yet.

If you want to see what proportion of my traffic is AI scrapers, even almost a year after I password-protected the site and made it useless to try to scrape:

$ grep -c ' 401 ' access.log
4164
$ wc -l access.log
5603 access.log