Anubis sends AI scraperbots to a well-deserved fate [LWN.net]

By Joe Brockmeier
July 10, 2025

Few, if any, web sites or web-based services have gone unscathed by the locust-like hordes of AI crawlers looking to consume (and then re-consume) all of the world's content. The Anubis project is designed to provide a first line of defense that blocks mindless bots—while granting real users access to sites without too much hassle. Anubis is a young project, not even a year old. However, its development is moving quickly, and the project seems to be enjoying rapid adoption. The most recent release of Anubis, version 1.20.0, includes a feature that many users have been interested in since the project launched: support for challenging clients without requiring users to have JavaScript turned on.

Block bots, bless browsers

AI scraping bots, at least the ones causing the most headaches, are designed to ignore robots.txt and evade detection by lying about their User-Agent header, and come from vast numbers of IP addresses, which poses a serious problem for site owners trying to fend them off. The swarms of scrapers can be devastating for sites, particularly those with dynamically generated content like Git forges.

How do site owners block bots while leaving the door open for the people those sites are supposed to serve? Most methods that might help also subject users to annoying CAPTCHAs, block RSS readers and Git clients, or introduce other friction that makes a site less usable. Blocking bots without too much friction is the problem that Anubis hopes to solve.

The MIT-licensed project was announced in January by its author, Xe Iaso. Anubis is written in Go (plus some JavaScript used for testing clients); it is designed to sit between a reverse proxy, such as Caddy or NGINX, and the web-application server.

In a write-up of a lightning talk given at BSDCan 2025 in June, Iaso said that Anubis was initially developed simply to keep scrapers from taking down Iaso's Git server. They hosted the project on GitHub with the organization name Techaro. The name was supposed to be for a fake startup as a satire of the tech industry. Then GNOME started using the project, and it took off from there. Now Techaro is the name that Iaso is actually using for their business.

Many LWN readers are already familiar with the project's mascot, shown to the left, which is displayed briefly after a client browser has successfully passed Anubis's test. In the short time that the project has been available, it has already been pressed into service by a number of free-software projects, including the Linux Kernel Mailing List archive, sourcehut, FFmpeg, and others. A demo is available for those who have not yet encountered Anubis or just want a refresher.

The project takes its name from the Egyptian god of funerary rites; Anubis was reputed to weigh the hearts of the dead to determine whether the deceased was worthy of entering paradise. If the heart was lighter than a feather, the deceased could proceed to a heavenly reward—if not, they would be consumed by Ammit, the devourer of the dead.

How Anubis works

Sadly, we have no crocodile-headed god to dispatch AI scrapers for eternity. The next best thing is to deny them entry, as Anubis does, by putting up a proof-of-work challenge that helps sniff out scrapers in disguise. Iaso was inspired to take this approach by Adam Back's Hashcash system, originally proposed in 1997.

Hashcash was intended to make sending spam more expensive by requiring a "stamp" in each email's headers. The stamp would be generated by causing the client to run a hash-based proof-of-work task, which required some CPU time for every email. Email without a stamp would be routed to /dev/null, thus sparing a user's inbox. The idea was that Hashcash would be little burden for users sending email at normal volume, but too costly to generate large amounts of spam. Hashcash did not take off, to the detriment of all of our inboxes, but that doesn't mean the idea was entirely without merit.

When a request comes to Anubis, it can issue a challenge, evaluate the response, and (if the client passes) issue a signed JSON web token (JWT) cookie (named "techaro.lol-anubis-auth") that will allow the browser access to the site's resources for a period of time.

Anubis decides what action to take by consulting its policy rules. It has three possible actions: allow, challenge, or deny. Allow passes the request to the web service, while deny sends an error message that is designed to look like a successful request to AI scrapers. Challenge, as one might expect, displays the challenge page or validates that a client has passed the challenge before routing the request.

The default policy for Anubis is to challenge "everything that might be a browser", which is usually indicated by the presence of the string Mozilla in the User-Agent header. Because the creators of AI scrapers know that their bots are unwelcome (and apparently have no ethics to speak of) most scraper bots try to pass themselves off as browsers as well. But they generally do not run client-side JavaScript, which means they can be stymied by tools like Anubis.

Administrators have the choice of JSON or YAML for writing custom policy rules. Rules can match the User-Agent string, HTTP request header values, and the request path. It is also possible to filter requests by IP address or range. So, for example, one might allow a specific search engine bot to connect, but only if its IP address matches a specific range.

The difficulty of Anubis's challenge is also configurable. It offers fast and slow proof-of-work algorithms, and allows administrators to set a higher or lower difficulty level for running them. The fast algorithm uses optimized JavaScript that should run quickly. The slow algorithm is designed to waste time and memory. A site might have a policy that lowers the difficulty for a client that has specific session tokens or ratchets up the difficulty for clients that request specific resources. One might also set policy to automatically allow access to some resources, like robots.txt, without any challenge. The difficulty level is expressed as a number; a difficulty of 1 with the fast algorithm should take almost no time on a reasonably fast computer. The default is 4, which is noticeable but only takes a few seconds. A difficulty of 6, on the other hand, can take minutes to complete.

New features

The 1.20.0 release includes RPMs and Debian packages for several architectures, as well as other binaries for Linux and source code. Iaso would like to expand that to providing binary packages for BSD, as well as better testing for BSDs in general. In addition to the packages supplied by the project, Anubis is also packaged by a few Linux distributions, FreeBSD Ports, the Homebrew project, and others. However, many of the packages are not up-to-date with the most recent Anubis stable version. A container image for Anubis is also available for those who would prefer that method of deployment.

The release introduces robots2policy, a command-line tool that converts a site's robots.txt to Anubis policy. I tried it with LWN's robots.txt and it correctly added rules against allowing bots to access site search and mailing-list search, but only included one rule denying access to a specific crawler, as it expects each User-agent line to have its own allow or disallow rule.

Another new feature in 1.20.0 is custom-weight thresholds, a way for administrators to set variable thresholds for client challenges based on how suspicious a client is—much like scoring used by spam filters. Does a client's user-agent have the string "bot" in it? Administrators can assign more weight points and up the difficulty level of the proof-of-work challenge. Does a client present a cookie that indicates that it already has a session with a trusted service? It can have points taken off its weight score. If a score is low enough, a client can be waved through without any challenge at all. If the score is too high, the client can be assigned a particularly difficult challenge that will take longer to complete.

Some administrators have been hesitant to use Anubis because they would rather not block users who turn off JavaScript. Indeed, that was one reason that we opted not to deploy Anubis for LWN's bot problem. With 1.20.0, administrators can use the metarefresh challenge in addition to, or in place of, the JavaScript proof-of-work challenge. As the name suggests, it makes use of the meta refresh HTML element that can instruct a browser to refresh a page or redirect to a new page after a set time period, usually a few seconds. This method is currently off by default, as it is still considered somewhat experimental.

I installed 1.20.0 on a test Debian bookworm system running Apache. The documentation for Apache was missing a line in the sample configuration (the opening stanza for the mod_proxy module), but it was otherwise easy to get up and running. The project also has documentation for other environments that might require special configuration, such as using Anubis with Kubernetes or WordPress. Given the project's speed of development, the documentation seems well-tended overall; the pull request I filed for the missing stanza in the Apache documentation was reviewed almost immediately.

Anime girl and other complaints

Some have complained that the Anubis mascot is "not appropriate for use on customer facing systems" and asked for a flag to disable it. Iaso has said that the ability to change or disable branding is an "enterprise feature", though they are open to making it a feature if Anubis becomes fiscally sustainable. While Anubis is free software, Iaso has asked ("but not demand, these are words on the internet, not word of law") people not to remove the anime girl character from Anubis deployments unless they support Anubis development financially.

Initially, there was some controversy because Anubis was using a mascot that was generated with AI, which did not sit well with some folks. In March, Iaso noted that they were commissioning a new mascot from CELPHASE, an artist based in the European Union, which became the current anime girl mascot.

Even with a professionally designed character, some organizations, such as Duke University, are hesitant to deploy Anubis in part because of its mascot. Duke experienced problems with bot traffic overwhelming several services, such as the Digital Repositories at Duke and its Archives & Manuscripts catalog, and performed a pilot project with Anubis.

The Assessment & User Experience Strategy department of the university library released a report in June, written by Sean Aery, about the pilot project. Aery reported that Anubis was effective—it blocked more than 4 million unwanted HTTP requests per day—while still allowing real users and well-behaved bots. Some real users were blocked too; 12 people reported having a problem with Anubis in one week, but that was usually due to having cookies disabled, according to the report.

However, despite Anubis's efficacy, the first limitation listed by Aery was the user interface for the challenge page:

The images, messages, and styles presented in the default challenge page UI are not ideal -- particularly its anime girl mascot image. Anubis lacks convenient seams to revise any of these elements.

As of this writing, Duke is "in discussions" with Iaso about "a sustainable way forward" and is still using Anubis; in the meantime, it has customized the page interface with artwork it has deemed more appropriate.

Aery does list some real limitations that users should be aware of before adopting the project, however. For example, the report notes that it is very early days for the project and for the practice of fending off AI scrapers. The project is evolving rapidly, so administrators will have to stay on top of frequent updates. And there is no guarantee that Anubis will maintain its edge against scrapers in the long run. Today, Anubis works very well at blocking unwanted visitors, but tomorrow? Many scrapers aren't willing to abide by a gentle "no" now; they are unlikely to give up easily if Anubis and other bot-blockers start denying access to resources their ethically-challenged owners really want. Bot makers will start looking for and finding ways around the blockers. Blockers versus bots will no doubt be a arms race the same way that spam filtering is a never-ending battle against spammers.

Another limitation is Anubis's bus factor; Iaso is responsible for about half the commits to the project. More than 80 people have contributed to the project since its inception, but only one contributor (Jason Cameron) has more than ten commits to the repository.

Iaso is not just carrying most of the development load for Anubis; they are also trying to build a business around the project. The boundaries between Anubis, the open-source project, and Iaso's budding business are fuzzy; combining BDFL-type governance with the potential conflicts of interest coming from a related business has been known to go poorly. That is not to say that it will in this instance, of course.

Anubis already ships with advanced reputation checking that works only with a paid service called Thoth offered by Techaro. Thoth allows Anubis to filter IP addresses by geography (GeoIP) or BGP autonomous system numbers so administrators can block or challenge clients by region or provider (such as Cloudflare). As the project continues to gain traction, it will be interesting to see if Iaso accepts contributions that might conflict with Techaro paid services.

The future

Iaso said at BSDCan 2025 that they were not sure what the end game for Anubis is:

I want to make this into a web application firewall that can potentially survive the AI bubble bursting. Because right now the AI bubble bursting is the biggest threat to the business, as it were.

Since Anubis is, essentially, a one-person show, it raises the question of whether they will be able to stay ahead in the inevitable arms race against scrapers. Iaso did say that they hope to hire another developer, and to provide Anubis as a hosted service at some point. Iaso is also trying out new ways to sort bots from real browsers with various browser fingerprinting techniques, such as John Althouse's JA4 TLS and a novel method of Iaso's own devising called Techaro HTTP Request Fingerprinting Version 1 (THR1).

Anubis only denies scrapers access to content; it does not, as some might wish, actually feed nonsense data to scrapers to poison the data sets that are being created. There are, at least, two open-source projects that do set out to punish unwanted crawlers by sending the bots data to give them indigestion: Nepenthes and iocaine. Anubis has an open issue to add a target for unverified requests that would allow administrators to redirect clients to Nepenthes or iocaine instead of simply blocking access. Iaso has said that they are willing to do this but thinks "Anubis itself should not be directly generating the poison".

User "gackillis" has put together a proof-of-concept project called jackal's carapace that works together with Anubis. It "very slowly spits out garbage data", pointing to a tarpit for scrapers while Anubis is loading. If the scraper takes the bait, it winds up in the tarpit instead of being fed real content. However, gackillis warns (in all caps) that the project is not ready for real-world deployments and could spike CPU and network usage dramatically.

On July 6, Iaso announced a pre-release of Anubis 1.21.0. The major changes in the upcoming release include the addition of new storage types for Anubis's temporary data, support for localized responses, and allowing access to Common Crawl by default "so scrapers have less incentive to scrape". It also includes a fix for a pesky bug that delivers an invalid response for some browsers after passing Anubis's test. That bug was thought to be fixed in 1.20.0, but the bug was reopened after reports that users were still seeing the bug.

In a better world, Anubis would never have been necessary—at least not in its current form. It's an additional layer of complexity for site owners to manage and introduces friction for users browsing the web. But Anubis is a lesser evil when compared to having sites knocked offline by overzealous bot traffic or employing more burdensome CAPTCHAs. With luck, Iaso and other contributors will be able to stay one step (or more) ahead of the scrapers until the AI bubble bursts.

to post comments

AI vs AI

Posted Jul 10, 2025 18:47 UTC (Thu) by cen (subscriber, #170575) [Link] (2 responses)

I think our future is bleak. It will be AI vs AI. "Bad" AI to scrape and spam around the internet and due to inadequate tooling, we'll be forced to use "good" AI to combat and detect bad AI. Tools like Anubis will be bypassed and become obsolete very fast since AI will get better and better immitating real human traffic. Trillions in compute power will go to waste but it is what it is..

AI vs AI

Posted Jul 10, 2025 21:40 UTC (Thu) by bojan (subscriber, #14302) [Link]

Future? Isn't 99% of the internet spam and useless social media garbage already? 🤪

AI vs AI

Posted Aug 11, 2025 15:00 UTC (Mon) by Rudd-O (guest, #61155) [Link]

Dead internet theory, except real.

FSF claims

Posted Jul 10, 2025 22:02 UTC (Thu) by knewt (subscriber, #32124) [Link] (8 responses)

As a 'fun' aside, the FSF has recently come out with their own take on Anubis....

> If we made our website use Anubis, we would be pressuring users into running malware.

https://www.fsf.org/blogs/sysadmin/our-small-team-vs-mill...

FSF claims

Posted Jul 11, 2025 5:20 UTC (Fri) by linuxrocks123 (guest, #34648) [Link] (7 responses)

I often, even usually, agree with the FSF, but not this time. Since the proof-of-work program is free software, I feel like that should be the end of any ethical concerns.

FSF claims

Posted Jul 11, 2025 7:13 UTC (Fri) by Ranguvar (subscriber, #56734) [Link] (1 responses)

Likewise from me. Ian's argument is presented curiously: "A program which does calculations that a user does not want done is a form of malware. [...] If we made our website use Anubis, we would be pressuring users into running malware." This suggests that he would consider Anubis morally acceptable if it demanded confirmation of intent before performing the proof of work.

FSF claims

Posted Jul 14, 2025 5:59 UTC (Mon) by ofranja (subscriber, #11084) [Link]

And that's the point, indeed.

The fact that the user accessed the website with a Javascript-capable browser having Javascript enabled by default makes it reasonable to assume the user consents to the execution of Javascript.

By presupposition, it can be said the user already consented via tacit or implicit consent.

Considering the opposite case: if they did not consent, they would have taken action by either:

a) using a browser without Javascript support;
b) disabling Javascript globally or for any website not explicitly whitelisted.

One possible argument against this line of thought is the fact that the user might be unaware some external code is being executed. Which is fair, though the responsibility of clarifying this (and gathering consent) likely lies at the browser side.

FSF claims

Posted Jul 11, 2025 7:27 UTC (Fri) by taladar (subscriber, #68407) [Link] (4 responses)

What about the ethical concern over wasted energy?

FSF claims

Posted Jul 11, 2025 7:36 UTC (Fri) by gf2p8affineqb (subscriber, #124723) [Link]

The alternative is wasting energy serving bot requests.

FSF claims

Posted Jul 11, 2025 9:38 UTC (Fri) by ras (subscriber, #33059) [Link] (2 responses)

I wonder what costs more energy: collecting the data, or pumping it through an AI learning engine. I have no idea. But I wouldn't write off the idea that Anubis saves energy on average. To according to AI's, training AI's consumes roughly 20 TWhr/year.

FSF claims

Posted Jul 11, 2025 12:35 UTC (Fri) by ibukanov (subscriber, #3942) [Link] (1 responses)

AI companies typically outsource data collection to other businesses. Data collectors often use very distributed proxies from residential IP addresses. Running that infrastructure is not free at all. Then they use complex machine learning models to solve captures. While not exactly LLMs, this takes energy.

FSF claims

Posted Jul 18, 2025 3:59 UTC (Fri) by ssmith32 (subscriber, #72404) [Link]

Not to mention that the way the residential ips are "acquired" is often dubious at best, and illegal at worst.

Duke employs killjoys

Posted Jul 11, 2025 0:13 UTC (Fri) by raven667 (subscriber, #5198) [Link] (32 responses)

Its a little sad that Duke was throwing so much of a fit about "professionalism" because the logo wasn't a branded thing, I've seen the Anubis interstitial graphics in tons of places these days and think it looks professionally made, and fun. We can have a little fun and whimsy even in serious operational contexts, it's not illegal (yet). ;-)

Duke employs killjoys

Posted Jul 11, 2025 1:33 UTC (Fri) by anarcat (subscriber, #66354) [Link] (28 responses)

It's especially sad because on the Anubis home page there's a pretty obvious link on top that says Unbranded Version which, if you bother to click on, you learn that you can use a customized version by sponsoring the author for this for 50$ per month.

Seems like a small fee for a large university that complains about a minor cosmetic detail. But I guess they feel they deserve to negotiate that down further?

Duke employs killjoys

Posted Jul 11, 2025 7:29 UTC (Fri) by taladar (subscriber, #68407) [Link] (1 responses)

50$ a month actually seems like a huge amount considering you can probably get something like 8-10 cloud servers for that amount. Might as well just scale up to take the AI scraper load at that point.

Duke employs killjoys

Posted Jul 18, 2025 5:04 UTC (Fri) by ssmith32 (subscriber, #72404) [Link]

$50 a month is most definitely not going to get you the resources you need to deal with ai bots. Whatever your definition of "cloud server" is, if you're paying $50 / month, and have a non-trivial site to defend, of, at least, passing interest to AI companies, it's not going to cut it.

Duke employs killjoys

Posted Jul 11, 2025 11:00 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

Do we know that the Unbranded version was available before Duke expressed concerns or is it, perhaps, the result of Duke's inquiry?

Duke employs killjoys

Posted Jul 11, 2025 13:28 UTC (Fri) by jzb (editor, #7867) [Link] (7 responses)

I thought this detail was too far afield for the article, but perhaps not -- in the report, the author noted that the problem with paying for Anubis was that none of the methods that Iaso currently accepts are ways that Duke is approved to pay for things. So, AIUI, it is less of a problem of the amount of money and more of a problem of how to make the payment in a way Duke's processes allow.

This is a pretty common problem of mismatch between open-source maintainers and corporate/organizational support. It is *hard* as an individual to meet the requirements of organizations like Duke or public companies that have a lot of controls around payments. It often takes special dispensation to be able to just pay somebody via GitHub sponsors or to shoot someone money via PayPal, etc.

Duke employs killjoys

Posted Jul 12, 2025 5:13 UTC (Sat) by raven667 (subscriber, #5198) [Link] (6 responses)

If one was trying to set them up in the financial system as an approved vendor then sure, getting a DUNS number, tax id, a PO, validating the company and billing address are legit takes work, but for a $50 expense it seems like someone should be able to put it on a card and expense it without going through that level of scrutiny. or spend a day cloning into the local git forge and CI to repackage with own branding, or just enjoy the whimsical artwork and move on

Duke employs killjoys

Posted Jul 12, 2025 6:06 UTC (Sat) by mjg59 (subscriber, #23239) [Link]

It'd probably be easy as a one-off, but making sure it's something that continues being paid for even if whoever set that up originally retires or decides to earn actual money or finishes their internship and starts their PhD is a more complicated problem

Rules around expenses

Posted Jul 13, 2025 15:37 UTC (Sun) by farnz (subscriber, #17727) [Link] (4 responses)

Part of the problem here is that it's not $50 one-off, but $50/month. One of the reasons the financial controls get strict is the risk that someone will expense a regular $50/month to their card so that their bowling league (or other group outside the organisation) can have an annual party at the organisation's expense.

And bodies supported by charitable donation (like Duke University) often go overboard on controls for handling this, in part because whereas headlines like "Google Funds Stripper, Booze, For Local Bowling League" are not going to have a significant impact on Google's income (especially once the article body explains that the embezzling employee got fired), headlines like "Duke University Funds Stripper, Booze, For Local Bowling League" are likely to result in potential donors getting put off (even if the embezzling employee got jail time, not just fired).

While a DUNS number and validating the addresses seems like a lot of work for a $600/year expense, requiring a formal purchase order and invoice does not - the paperwork back-and-forth lets you validate that you're dealing with someone who is either genuine, or happy to take the risk of being busted for criminal fraud over $600/year.

Finally, having it done through departmental purchasing instead of as someone's personal expenses means that it can be kept going even if the person who set it up leaves the organization; the responsibility for signing off attaches to a job role, not a person, and thus is kept even if the person goes away.

Rules around expenses

Posted Jul 13, 2025 23:02 UTC (Sun) by raven667 (subscriber, #5198) [Link] (1 responses)

Thanks for the more detailed example, I said something because so many comments end up with "why don't they *just* ..." when people don't understand or appreciate the complexity outside of their experience.

Rules around expenses

Posted Jul 18, 2025 5:17 UTC (Fri) by ssmith32 (subscriber, #72404) [Link]

I see no comments of that form here. And it still stands, that whatever work is involved, Duke pays *a lot* of vendors for a lot of things, so there is a way. Just do the paperwork, and get them paid. After all, whoever is responsible for managing software procurement is *also* getting paid to do a job.

Rules around expenses

Posted Jul 18, 2025 5:13 UTC (Fri) by ssmith32 (subscriber, #72404) [Link] (1 responses)

I doubt $600 would cover lane rentals and pizza for a decent sized bowling league, let alone beer. Proper drinks are right out. And, unless you're paying your league mates to strip..

My ¡per-diem! at a painfully cheap company decades ago was around $70-$90.

And I doubt whatever the process is, it's less work than configuring and maintaining Anubis for their site.

They're being cheap and lazy.

Just do whatever paperwork and get them paid.

Rules around expenses

Posted Jul 18, 2025 9:48 UTC (Fri) by farnz (subscriber, #17727) [Link]

It doesn't have to cover the full costs of the league to result in a problematic headline; one drink, plus entry to a strip club (just the door fee, nothing else) for a team of 5 bowlers is enough to trigger that sort of headline, and will cost you under $500 in total.

The paperwork burden is not just on Duke's side, but also on Techaro's side - Duke will want a proper paper trail (purchase order + invoice at a minimum) to cover this risk for them.

Remember that the goal here is just enough traceability that Duke's lawyers believe that they can either force the headline to be "Techaro Funds …" (which keeps Duke out of the worst of the storm), or win a serious sum of money for defamation, not to prevent all sorts of possible issues.

And companies tend to be much more generous than charitably supported organisations and also much less paranoid about the risks of not having sufficient control - like I said, Google can ride out the headline "Google Funds …" with barely any impact, if it's a mere $600 per year embezzled, where a charitable organisation will lose donations from people who see the headline. Note that Duke's standard per-diem rate for 2025, for example, is $68 - less than your "painfully cheap company decades ago", despite inflation.

Duke employs killjoys

Posted Jul 11, 2025 15:34 UTC (Fri) by cultpony (subscriber, #167240) [Link] (15 responses)

Not only that but it's MIT licensed and although the author says that one should rather support the project financially, you *CAN* replace the assets by compiling the software yourself. Though so far every single group I've seen talk about this and complain about the mascot has also respected the author's wishes to not do that.

Duke employs killjoys

Posted Jul 11, 2025 20:19 UTC (Fri) by neilbrown (subscriber, #359) [Link] (14 responses)

> Though so far every single group I've seen talk about this and complain about the mascot has also respected the author's wishes to not do that.

So it isn't actually "free software" then?
One big value of having a license is that the rules are clear. Muddying those rules with wishes seems to miss the point.

Duke employs killjoys

Posted Jul 11, 2025 20:23 UTC (Fri) by mb (subscriber, #50428) [Link]

>Permission is hereby granted, free of charge, to any person obtaining a copy
>of this software and associated documentation files (the "Software"), to deal
>in the Software without restriction, including without limitation the rights
>to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
>copies of the Software

Duke employs killjoys

Posted Jul 12, 2025 11:10 UTC (Sat) by Wol (subscriber, #4433) [Link] (12 responses)

> Muddying those rules with wishes seems to miss the point.

I hate to say it but I think you're being very unfair. So long as the distinction between the licence (what you MUST do), and the wishes (what you SHOULD do) is kept clear and unambiguous, what's wrong? And it's obvious others feel the same - witness how they *chose* not to exercise their right to strip the logo.

The guy wants to be paid - he wants to eat!

Rather different, but it's why my photos on flickr were CC-licenced share-and-share-alike. I wanted to download anything nice for personal use for wallpapers etc etc, so I felt I had to pay by licencing my photos the same.

Cheers,
Wol

Duke employs killjoys

Posted Jul 12, 2025 13:25 UTC (Sat) by dskoll (subscriber, #1630) [Link]

The guy wants to be paid - he wants to eat!

FYI: The developer of Anubis goes by they/them or she/her.

Duke employs killjoys

Posted Jul 12, 2025 20:11 UTC (Sat) by neilbrown (subscriber, #359) [Link] (1 responses)

If the users of the code don't feel free to make a change, then they are not free.
Just because the prohibition might have a different legal standing than the permission doesn't mean much in practice.

If the author wants to make money from their creation that is good and proper. Attempting to do it by making contradictory statements about how they expect people to use their code is not likely to fill their customers with confidence.

Duke employs killjoys

Posted Jul 12, 2025 21:01 UTC (Sat) by pizza (subscriber, #46) [Link]

> Just because the prohibition might have a different legal standing than the permission doesn't mean much in practice.

One (ie the actual license text [1]) carries legal weight, and the other ... doesn't.

Note the "modify without limitation" and the conspicuous lack of "don't change the logo" in the list of conditions.

[1]
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Duke employs killjoys

Posted Jul 12, 2025 22:57 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (8 responses)

> So long as the distinction between the licence (what you MUST do), and the wishes (what you SHOULD do) is kept clear and unambiguous, what's wrong?

A license is permission to do something. It is ethically incoherent to say "I hereby give you permission to change the logo free of charge, but it is also unethical for you to change the logo without paying me." You don't get to dictate rules of ethics to me. If you gave me permission to change the logo, then it is ethical for me to change the logo, because I have your permission to do so.

Duke employs killjoys

Posted Jul 13, 2025 9:30 UTC (Sun) by cultpony (subscriber, #167240) [Link] (4 responses)

They're not dictating you any rules of ethics. They've merely set down the rules (the license) and their preferences in regards to that. You can respect or disrespect the author's wishes, but by putting it under MIT license, it's very clear they see no ethical issue in the matter, mostly one of disappointment on their end if you choose to not reward the work that went into the project.

Duke employs killjoys

Posted Jul 15, 2025 20:54 UTC (Tue) by NYKevin (subscriber, #129325) [Link] (3 responses)

> mostly one of disappointment on their end if you choose to not reward the work that went into the project.

It is incoherent to give permission to do something, and then be disappointed when that thing is done. If you do not want people doing it, do not give permission to do it.

Duke employs killjoys

Posted Jul 15, 2025 21:46 UTC (Tue) by excors (subscriber, #95769) [Link]

> It is incoherent to give permission to do something, and then be disappointed when that thing is done. If you do not want people doing it, do not give permission to do it.

That seems an excessively binary way of looking at things. Technical standards sensibly distinguish "must" from "should": you have explicit permission to violate a "should", even though they generally don't want you to, because they recognise you might have unusually good reasons to violate it and the overall harm from strictly banning it would be greater than the harm from occasional violations. They still provide the "should"s because it's better if most people follow them, and most people do.

Similarly, human societies distinguish laws (must) from ethics (should), because lots of behaviours are considered harmful to society but not so harmful that it's worth having any formalised punishment, or because they're too context-dependent to codify them without a lot of collateral damage. The scorn from your peers is sufficient disincentive for unethical-but-not-illegal behaviour.

Software licences are musts, enforced by law, so it seems perfectly normal and reasonable to have a separate list of things you think people shouldn't do with your software but that you're not going to enforce legally.

Duke employs killjoys

Posted Jul 18, 2025 5:24 UTC (Fri) by ssmith32 (subscriber, #72404) [Link]

Most anyone who has been responsible for guiding another person to a more independent state has likely gone through this exact process.

Giving someone permission to disappoint you is not in the least bit incoherent.

On the contrary it's part and parcel of good management/leadership/parenting.

Duke employs killjoys

Posted Jul 18, 2025 10:01 UTC (Fri) by jzb (editor, #7867) [Link]

It is not incoherent, it's part of living in a society. We give people permission to post comments on LWN. Sometimes they disappoint us. Sometimes they delight us. And a lot of stuff in the middle.

I give my kids permission to disappoint me. "You can do this, but I don't like it and I'm telling you I don't like it, but this is a decision you get to make" is completely coherent with also setting hard and fast rules. You can wear an outfit that I don't approve of, but I'll tell you my opinion and I am free to be disappointed when they don't listen. But they cannot take my car without asking. Totally a rational way of interacting with others.

Putting Anubis under a permissive license but *asking* users not to do a thing is not incoherent. First of all, it's infinitely better that Anubis is under a well-understood license rather than the author inventing or adopting a source-available license that punishes everyone else because a percentage of people only operate on a binary "I can or cannot do this legally" principle.

Additionally, it is binary thinking to imagine that the only use of Anubis code is to run the project as-is. Some people might want to use a component in Anubis within another open-source project, where displaying the mascot isn't even in the equation. Why throw up a roadblock for those people because the author would like to make Anubis development sustainable?

It also allows packaging Anubis for Debian, Fedora, etc. A number of projects ask that distributions do not package their software because they prefer to have a direct relationship with users and/or don't like the way distributions package their software. They are not willing to be anti-FOSS in the process. A packager can thumb their nose at the request, and the upstream is within their rights to complain, but they aren't in a place to sue. ISTM that is a preferable state to using a license as a weapon against any usage that the author doesn't like—and far more civil.

Finally... using a well-understood license for Anubis from the outset reduces everyone's risk in case the author abandons the project for some reason, is hit by a bus, or takes it in a direction that the larger community does not like. You can fork Anubis, strip the mascot, and continue development in a direction that the author doesn't want to go in. A "thou must keep the mascot or pay me" license would restrict that.

Duke employs killjoys

Posted Jul 13, 2025 10:50 UTC (Sun) by Wol (subscriber, #4433) [Link] (2 responses)

>A license is permission to do something. It is ethically incoherent to say "I hereby give you permission to change the logo free of charge, but it is also unethical for you to change the logo without paying me."

It's also ethically incoherent to say "Free Software is about sharing", and then to object to a notice that says "Please share back with me". The gal has put this stuff out - FOR FREE - for everyone to use. What's wrong with asking for something back in return "pretty please"?

And what are the alternatives? Use a licence that demands payment? That's non-Free! Plus it's pretty much guaranteed that your "gift to the world" will lie forgotten and untouched deep in the bowels of a git server in a basement somewhere in Alpha Centauri.

It's a pretty shameful state of affairs when people who are handing out the gift of free labour are reduced to begging "Please Sir, can I have something back". And even worse when people object to somebody requesting "a fair day's pay for a fair day's work".

Cheers,
Wol

Duke employs killjoys

Posted Jul 15, 2025 20:51 UTC (Tue) by NYKevin (subscriber, #129325) [Link] (1 responses)

> What's wrong with asking for something back in return "pretty please"?

Nothing. I never said this was improper. You can accept donations all you like as far as I'm concerned.

I said it was improper to suggest that licensees "should" respect any preferences that are not actually in the license, and I continue to hold that belief. If you give me permission to do X, then you cannot simultaneously claim that I "should not" do X, because that's what it means to give permission in the first place - you are relinquishing the right to tell me how I should or should not use your software.

> That's non-Free!

You yourself have repeatedly argued (in other threads) that non-free software is a legitimate state of affairs.

Duke employs killjoys

Posted Jul 15, 2025 21:09 UTC (Tue) by Wol (subscriber, #4433) [Link]

> > That's non-Free!

> You yourself have repeatedly argued (in other threads) that non-free software is a legitimate state of affairs.

Doesn't mean I *want* to use a non-free licence. In many instances my moral compass says "you ought to share back".

You may also have noticed I'm pragmatic, not religious, about this stuff. If I want other people to use my software, pragmatism says I need to use a free licence. If I want to get paid, pragmatism says 1% of not much is better than 100% of nothing.

So using an Open/Free licence and saying "pretty please" is about the best option available ... I didn't say it was a good option ...

Cheers,
Wol

Duke employs killjoys

Posted Jul 17, 2025 7:02 UTC (Thu) by jond (subscriber, #37669) [Link]

> Seems like a small fee for a large university that complains about a minor cosmetic detail.

When I worked at a university getting even $1 of _recurrent_ spend was disproportionately difficult.

Duke employs killjoys

Posted Jul 11, 2025 8:44 UTC (Fri) by ballombe (subscriber, #9523) [Link] (2 responses)

Also describing the character as 'Anime girl' is inaccurate to the point of FUD. Compare the anubis mascot with the example given in the "Anime girl" wikipedia page.

Duke employs killjoys

Posted Jul 11, 2025 9:19 UTC (Fri) by jhe (guest, #164815) [Link]

Its not a Bishojo, but a Nekomimi. Both qualify as anime girl. It likely also qualifies as a furry.

Duke employs killjoys

Posted Jul 11, 2025 9:59 UTC (Fri) by excors (subscriber, #95769) [Link]

An anime girl is just a girl from an anime, not specifically the subtype that Wikipedia redirects to. This is clearly an anime-inspired style, with the big eyes and cat ears. (Looks like it's also inspired by the developer's character in Final Fantasy XIV, which is not really anime but is anime-adjacent). The project's official site says "Yes, we made the GNOME anime woman anti-bot captcha", and the developer calls it an "anime image" on their blog. So it seems a perfectly reasonable description of the character.

SourceHut switched to go-away

Posted Jul 11, 2025 9:31 UTC (Fri) by ddevault (subscriber, #99589) [Link]

SourceHut has since switched to go-away, which was more configurable and helped to reduce the user impact to the bare minimum:

https://git.gammaspectra.live/git/go-away

Anubis was clutch when we needed it, though.

Pity about the logo change

Posted Jul 11, 2025 12:44 UTC (Fri) by Baughn (subscriber, #124425) [Link] (1 responses)

I thought the old logo looked much better. You can see it here: https://github.com/TecharoHQ/anubis/pull/204

Pity about the logo change

Posted Jul 12, 2025 5:00 UTC (Sat) by raven667 (subscriber, #5198) [Link]

thanks for the link, I don't recall seeing their old iconography before but I strongly believe the newer ones are better designed, each one conveys its intended meaning more clearly and is not cluttered with unnecessary detail that distracts from the information they are trying to communicate, but you can like what you like, it hasn't been made illegal (yet ;-).

PoW solutions are not practical

Posted Jul 11, 2025 14:28 UTC (Fri) by milek7 (subscriber, #141321) [Link] (20 responses)

This really only works until bots gain dedicated code for this. This is a problem for all proof-of-work solutions, challenge needs to be easy enough to not be overly annoying on low-end battery powered devices with generic implementation, while bots can have optimized software and hardware with grid power. At approximate market rate (via BTC) default Anubis challenge is worth around 0.23 picodollars.

PoW solutions are not practical

Posted Jul 11, 2025 14:35 UTC (Fri) by corbet (editor, #1) [Link] (19 responses)

The bots are not using that "hardware with grid power". They are using the resources of ordinary people across the net. Have a look at how delightful operations like "Bright Data" operate, or the fun tricks with browser extensions.

Of course, I suspect that these operators are even less concerned about expending CPU time on systems they don't own. One of the reasons that LWN has not adopted something like Anubis, beyond the fact that we were able to stabilize things without it (so far), is a feeling that, once these systems become sufficiently annoying, the scrapers will just incorporate the code to do the challenge and continue on their merry way.

PoW solutions are not practical

Posted Jul 11, 2025 23:46 UTC (Fri) by gerdesj (subscriber, #5446) [Link] (13 responses)

We will eventually sort this out. You and most of us here are old enough (and ugly enough) to be able to draw obvious comparisons to spam (UBE). Spam is still a bit of an issue but I think we generally accept it is a largely solved problem in a similar way that non Texans live with measles and the like. If most people accept that it exists and get inoculated and don't toy with injecting bleach (it worked for Covid!) then we can learn to live with some pretty nasty pathogens.

I think Anubis is an early Spam Assassin and eventually we will get the rspamd that we need and we will go through the bit where we have to train it if we want really good results or not bother and put up with a bit of rubbish. Remember that a properly trained bayesian classifier will dump nearly all spam for a particular mailbox and will work well for an organisation and only slightly less well for the world (because what on earth is spam?)

I quite like the Hashcash approach and I do think it will work for quite a while. There are loads of "proof of work" thingies and the clever kids keep on making up more. You could randomly switch hashing algos from quite a large pool and that will always make it expensive for a bot but acceptable for a browser. The bots will end up becoming too expensive at some point (I hope).

I run quite a few web sites off one IP address through a HA Proxy cluster. They all have their own SSL/TLS cert from Lets Encrypt and there is no default, SNI is required. That means you need to know the DNS name to connect to the various websites. A naiive port scan and connect will just get a 503 error.

Even if you have one site on an IP address - require SNI and don't redirect http to https. That used to be a nice neighborly thing but the internet is no longer nice. I also suggest that you stop the cute domain == www thing that slashdot "invented" years back. Require www. on your URLs.

Bizarrely, you can hide in plain sight, quite a lot.

I know that most of my recommendations are bollocks when you consider "security by obscurity is no security" but the world has changed and moved on. Modern hardware is phenomenal compared to when I were a lad.

We only have to make life nasty for a bot a bit and it becomes uneconomical to run. I wonder if you could get them to do the POW for Bitcoin et al? That would be delightful.

PoW solutions are not practical

Posted Jul 12, 2025 11:24 UTC (Sat) by Wol (subscriber, #4433) [Link] (7 responses)

>I know that most of my recommendations are bollocks when you consider "security by obscurity is no security" but the world has changed and moved on. Modern hardware is phenomenal compared to when I were a lad.

As long as you accept that "security by obscurity" is no good on it's own, it's a good extra layer.

What I'd like is for the powers to be to extend the definition of "trespass" to include a computer. Don't know about other jurisdictions, but we have the legal concept "knew or should have known". So the mere fact that eg spam is designed to dodge spam-filters, and these AI bots use botnets to appear to come from multiple computers to avoid detection, would instantly land all this stuff into criminal territory (under English/Welsh law).

Everyone (from Presidents and Royalty, down to beggars) should have a "privacy wall", and unauthorised breach of that wall should be trespass, with all that entails.

Cheers,
Wol

PoW solutions are not practical

Posted Jul 12, 2025 11:59 UTC (Sat) by pizza (subscriber, #46) [Link] (2 responses)

> So the mere fact that eg spam is designed to dodge spam-filters, and these AI bots use botnets to appear to come from multiple computers to avoid detection, would instantly land all this stuff into criminal territory (under English/Welsh law).

So... who do you charge with this new crime?

PoW solutions are not practical

Posted Jul 12, 2025 16:31 UTC (Sat) by Wol (subscriber, #4433) [Link] (1 responses)

> So... who do you charge with this new crime?

Is it really a new crime? And we'll charge anyone we can find (yes I know that is usually "nobody").

The problem at the moment is even if we find them, what do we charge them with? Bit like the Prince Philip Prestel hack.

There's too many people who are quite happy to abuse other people and their property, for their own personal gain. To me, extending the law of trespass seems the obvious solution - the act of trespass itself, while "wrong", is a civil misdemeanour. In most cases it's assumed permission would be retrospectively granted. It's when you "knew or should have known" it wouldn't be granted that it becomes criminal.

And yes, the chances of finding anybody to charge may be slim - but it might not. Plus most people stick to the law. There's probably a lot of "hey it's not actually illegal" people who, if they realise actually it is, and it's criminal, might stop. And if it's companies involved, all of a sudden the Nazgul might get involved ...

Cheers,
Wol

PoW solutions are not practical

Posted Jul 12, 2025 16:57 UTC (Sat) by pizza (subscriber, #46) [Link]

> There's probably a lot of "hey it's not actually illegal" people who, if they realise actually it is, and it's criminal, might stop.

The fact that nearly everyone actively tries to masquerade as something they're not tells me that they simply don't care.

And each one actor in of themselves isn't a problem; the issues come from everyone and their dog doing their own scraping.

Then consider that stuff that is actually criminal today (eg botnets performing DoSes and active breakin attacks) goes nearly completely unpunished.

PoW solutions are not practical

Posted Jul 12, 2025 12:39 UTC (Sat) by tzafrir (subscriber, #11501) [Link] (3 responses)

I'm not sure I follow. I visit a site, it tells me to go away because my browser is not supported. I change the user agent string and it works. Is it criminal? I should have known that the site does not support my system.

PoW solutions are not practical

Posted Jul 12, 2025 16:59 UTC (Sat) by Wol (subscriber, #4433) [Link] (2 responses)

No. Telling you your browser isn't supported isn't telling you you're not welcome. The reasonable implication here is "come back with a supported browser". And if you change the user agent that's effectively what you're doing.

If, on the other hand, the response was "500 Go away, Firefox user", then it probably would be criminal to change the agent to bypass it. On the other hand, an error message like that is likely to fall foul of all sorts of equal access legislation.

It's the difference between rejecting the person/user, and the agent/Firefox.

The crime, basically, is "ignoring a refusal of permission". So for example this would actually make bypassing robots.txt a criminal act - if it says "no crawlers", or "no AI bots" or whatever. But the wording is intentionally vague, so most people won't get into trouble most of the time. It would typically take something blatant, but if you're doing something anti-social, and you are actively evading measures intended to stop you, that is pretty blatant. Likewise if your actions cause easily predictable harm ...

Cheers,
Wol

PoW solutions are not practical

Posted Jul 12, 2025 20:44 UTC (Sat) by NYKevin (subscriber, #129325) [Link] (1 responses)

> On the other hand, an error message like that is likely to fall foul of all sorts of equal access legislation.

I don't know what laws they have in Europe for this, but such laws almost certainly don't exist in the US.

Now, if you said something like "go away [woman/black person/etc.]," that would probably be illegal (I'm not 100% sure our anti-discrimination laws apply to all websites across the board no matter their purpose, but they at least apply in contexts like employment, housing, etc.). But "Firefox user" is not a protected class. You can discriminate against Firefox users all you like.

> The crime, basically, is "ignoring a refusal of permission". So for example this would actually make bypassing robots.txt a criminal act - if it says "no crawlers", or "no AI bots" or whatever.

In the US, that would amount to a violation of the CFAA. Put another way, you are claiming that Aaron Swartz was guilty when he was prosecuted for mass-downloading journal articles from JSTOR.

This is still an open question of law in the US, although I imagine some of you would say that it was settled in Van Buren v. United States (2021). The problem with Van Buren is that it addressed a computer system that was not open to the public, and as the Ninth Circuit explains in the HiQ Labs litigation,[1] the statute seems to be written under the presumption that a "protected computer" is not open to the public (although the statute does not explicitly say that), which makes scraping a materially different situation to what Van Buren was about. The Ninth Circuit reckons that the CFAA is wholly inapplicable to scraping, unless there is some kind of traditional access control like a username and password. They also argue that this is logically coherent with Van Buren's emphasis on a "gates up" or "gates down" metaphor, saying that public websites lack such "gates" altogether. Of course, other circuits may well come to a different conclusion about this, especially if a solution like Anubis is in place (I have no idea if that would count as a "gate" or not).

TL;DR: "No trespassing" signs are for the real world. They don't seem to work in a digital context, at least as far as the courts have determined to date.

[1]: https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/1...

PoW solutions are not practical

Posted Jul 13, 2025 12:13 UTC (Sun) by kleptog (subscriber, #1183) [Link]

> I don't know what laws they have in Europe for this, but such laws almost certainly don't exist in the US.

It would be more in the area of consumer protection. Such an error message would not be an issue for your local pizzeria, but it would be a problem if it were Google or some government website. Since AIUI the US has not enacted any meaningful consumer protection legislation in decades it doesn't surprise me it wouldn't be an issue there.

(The UK regulations around disabilities is somewhat stricter than elsewhere AFAIK.)

> The crime, basically, is "ignoring a refusal of permission". So for example this would actually make bypassing robots.txt a criminal act - if it says "no crawlers", or "no AI bots" or whatever.

This is effectively already a thing in the EU, due to Article 4(3) of regulation 2019/790. [1]

> 3. The exception or limitation provided for in paragraph 1 shall apply on condition that the use of works and other subject matter referred to in that paragraph has not been expressly reserved by their rightholders in an appropriate manner, such as machine-readable means in the case of content made publicly available online.

The robots.txt is a machine readable means of indicating conditions and so would count. It applies to AI model training by Article 53 of the AI Act. [2] As usual with EU regulations, enforcement is an issue. It's at best as sort of "regulatory offense". No-one goes to jail for those, it at most leads to fines.

Note this limitation does not apply to models that are released under an open-source licence.

I think it would also be helpful if the robots.txt format was extended to indicate a reason why a path was restricted, to distinguish between "this part is expensive" and "this part has copyright restrictions". I have not yet seen any will though in this direction.

On the whole though, I can totally imagine the large model producers doing something like:

if IP is not Europe:
ignore robots.txt

[1] https://eur-lex.europa.eu/eli/dir/2019/790/oj/eng
[2] https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

PoW solutions are not practical

Posted Jul 12, 2025 17:21 UTC (Sat) by wtarreau (subscriber, #51152) [Link]

> That means you need to know the DNS name to connect to the various websites. A naiive port scan and connect will just get a 503 error.

Oh I remember when I was doing this long ago when there was no SSL. Then SSL appeared and SNI was not yet the norm, this practice disappeared. But as you said, if you mandate SNI, they won't guess your site name.

> Even if you have one site on an IP address - require SNI and don't redirect http to https.

Or you can simply do it by reusing the host field. With haproxy it's just `http-request redirect scheme https if !{ ssl_fc }`

> I also suggest that you stop the cute domain == www thing that slashdot "invented" years back. Require www. on your URLs.

Just out of curiosity, which use case do you block with that one ? Because if the domain is known, trying again with "www." in front is trivial.

> I know that most of my recommendations are bollocks when you consider "security by obscurity is no security" but the world has changed and moved on. Modern hardware is phenomenal compared to when I were a lad.

I'm all with you when it comes to "security by obscurity" on this. Passwords are the first example where it works. And when I help some people counter an attack, the first thing I tell them is "shut up and don't start bragging on social media about what's happening and even less how you're addressing it", because figuring what changed costs a lot to the attacker and you need to reach the point where it's no longer affordable to attack your site.

> We only have to make life nasty for a bot a bit and it becomes uneconomical to run.

Agreed.

> I wonder if you could get them to do the POW for Bitcoin et al? That would be delightful.

Well, some legit browsers are already being abused by crypto malware, so here the honest visitors would also face this.

I think instead that reintroducing the human in the chain is essential to break the business model, exactly for the reason you mentioned, the hardware becomes extremely powerful, and that's why it's economically interesting to use. When you see these photos of two guys sat in front of 400 smartphones feeding social media and scraping sites, imagine how many it would require if a user interaction was required every minute. That would be 7 per second, requiring 10 times more employees for the same tool.

PoW solutions are not practical

Posted Jul 13, 2025 18:37 UTC (Sun) by NAR (subscriber, #1313) [Link] (1 responses)

I don't think a SpamAssassin-like solution is coming. The incoming spam has a lot of text that can be analyzed and the "is it spam?" decision can be based on it - the incoming HTTP requests of AI bots do not have this data.

I also heard that the spam problem was (mostly) solved by sending e-mail outside a few giants like Google hard. SPF, DKIM, ARC, probably a few other acronyms are involved too. Also, a substantial amount of conversation moved away from e-mails to various messengers.

PoW solutions are not practical

Posted Jul 13, 2025 19:40 UTC (Sun) by mb (subscriber, #50428) [Link]

I think at the HTTP level it is possible to detect such AI bot requests with a high probability and then throttle them, once the DDoS has started.
However, at that point much of the damage has already been done and much CPU has already been consumed.

At a lower level it's really hard to detect. There's basically no information.
They only use an IP address once every day or so.
Therefore there isn't really any information available to make the decision to block a connection or not.

And even if you keep the information about probably-AI-bots based on past connections this database grows large very quickly due to the extremely large amount of IP addresses involved.

I currently basically resort to throttling the bandwidth for everyone for a limited amount of time, once I detected a AI-DDoS storm.
That kind of works at the cost of making the server response sluggish during AI-DDoS. But it will continue to work. An unthrottled AI-DDoS will bring it completely down. So it is an improvement.

PoW solutions are not practical

Posted Jul 18, 2025 14:03 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

> There are loads of "proof of work" thingies and the clever kids keep on making up more. You could randomly switch hashing algos from quite a large pool and that will always make it expensive for a bot but acceptable for a browser.

Ultimately, you make the algorithm exchange something that has its own value. If you're doing a PoW algorithm to make some function (e.g., access websites, append to a distributed log or ledger, etc.) very expensive to spam, then you may as well /reward/ and /remember/ some of that work and allow those rewards to be passed on / exchanged. If you're going to do work and expend energy, you might as well capture the full utility of that work, rather than just completely forget it and throw it away.

Basic Attention Token is one effort kind of this in space, though focused more on the economics of advertising. I think though for a general website access token it should build on a CryptoNote protocol, to provide privacy (the best known and most widely used implementation of which would be Monero, and has roughly ~1 to 5 cent (Euro) transaction fees. A browser could provide some small payment in return for access for X years/months/days/whatever - and would be able to later, at any time, prove it had sent that transaction.

This is, sooner or later, how the web will go. Least, the portion that isn't a hellsite of ads and AI click bait. The issue of ads and remunerating (good) content providers were already applying the (slow burning) pressure to go in this direction from one angle; the AI scraper-bot issue will provide another source of pressure from another angle.

PoW solutions are not practical

Posted Jul 19, 2025 20:56 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

None of the blockchain crap scales to the level needed to replace ads. But it can be done with classic old technologies, with the help of some new technologies - zero-knowledge proofs for anonymity.

And it can be successful. I used to subscribe to Scroll, that provided transparent ad-free browsing experience on supporting websites. They got a reasonably sized network and had some momentum, but then they were acquired by Twitter and shut down. Sigh.

PoW solutions are not practical

Posted Jul 12, 2025 17:04 UTC (Sat) by wtarreau (subscriber, #51152) [Link] (4 responses)

Exactly the same here. We've seen a huge increase of activity on our gitweb at haproxy and given how the requests are spread, for now I'm convinced that proof-of-work will only slightly increase the power bill of those whose browser is being abused, and since it will not at all reduce their ability to solve the challenge, they'll do their request anyway and in the end it will just end up with every one in the chain doing more work (both us and them), except the ones this work is being done for.

I've been thinking instead about requiring user interactions (captchas etc) for such use cases, despite how annoying they can be. For now we managed to reduce the abuse traffic around 20-fold using other tricks so I prefer not to start pissing users off.

With that said, I was pleased to notice that mainstream AI players *do* respect robots.txt, which allows us to keep them away from useless interactive areas like gitweb, without blocking their access to the docs that are used to provide more accurate responses to their users. I think they've already understood that either they respect it and are allowed to retrieve limited info or they're playing bad and are entirely denied.

PoW solutions are not practical

Posted Jul 12, 2025 17:16 UTC (Sat) by mb (subscriber, #50428) [Link] (3 responses)

>With that said, I was pleased to notice that mainstream AI players *do* respect robots.txt

That's not what I see.
GPTBot ignores my robots.txt. It does send a user agent, though, which I use to completely block it.

I have a wildcard-rule that blocks access to stuff like cgit and also an explicit rule for GPTBot that disallows everything.
Doesn't work.

PoW solutions are not practical

Posted Jul 12, 2025 17:34 UTC (Sat) by wtarreau (subscriber, #51152) [Link] (2 responses)

Strange, here are the first 15 ones I just found in todays logs, they did retrieve robots.txt and did not descend /git which is forbidden there:

MJ12bot/v1.4.8, ClaudeBot/1.0, SemrushBot/7~bl, MJ12bot/v2.0.2, DotBot/1.2, YandexBot/3.0, Googlebot/2.1, SemanticScholarBot, SemrushBot, AhrefsBot/7.0, OAI-SearchBot/1.0, Bytespider, GPTBot/1.2, bingbot/2.0

Maybe they have rule number limits or maybe they don't properly match their name in your file ? I'm using "User-agent: *".

PoW solutions are not practical

Posted Jul 12, 2025 18:08 UTC (Sat) by mb (subscriber, #50428) [Link] (1 responses)

The only thing I can think of is that I use all-lowercase field names. Which is explicitly allowed, though.
I changed it to the more common User-agent and Disallow style.

PoW solutions are not practical

Posted Jul 13, 2025 5:17 UTC (Sun) by wtarreau (subscriber, #51152) [Link]

> The only thing I can think of is that I use all-lowercase field names. Which is explicitly allowed, though.

This could indeed be an explanation. I'm pretty sure that even if permitted, many implementations might be broken on this regard. Let's see if it works better with your change.

End-game

Posted Jul 11, 2025 22:36 UTC (Fri) by nirbheek (guest, #54111) [Link] (2 responses)

> Iaso said at BSDCan 2025 that they were not sure what the end game for Anubis is:

Isn't the end-game just that crawlers will start to detect Anubis and swap the UA to curl? Of course you could run Anubis on all UAs, but suddenly GitLab is a lot less useful if git, curl, wget etc can't access it without authentication.

We could just end up with all GitLab instances being login-walled, unless your client can run JS.

End-game

Posted Jul 11, 2025 22:44 UTC (Fri) by mjg59 (subscriber, #23239) [Link]

The number of legitimate curl requests is likely far lower than the number of legitimate requests with Mozilla in the UA, so rate limiting the total number of curl requests is likely to have much less impact than rate limiting the things that look like browsers

End-game

Posted Jul 12, 2025 0:04 UTC (Sat) by gerdesj (subscriber, #5446) [Link]

We have the world we have. You might need an API key for your Gitlab.

How will a crawler/bot "detect" Abubis and somehow bypass it? It will have to do the hashcash thing to move on. A suitably suspicious implementation may require a hashcash calc per URL that a "real" browser wont need to do.

That hashcash thing could involve a random algorithm which can't be embedded in hardware or whacked out to a GPU (which probably does not exist).

The latest version of Anubis does not need JS and I'm sure many, many POW algos can be stuffed into it.

Weird definition of malware

Posted Jul 14, 2025 14:18 UTC (Mon) by rbranco (subscriber, #129813) [Link]

The FSF's definition of malware is so out of touch with reality it's actually cringe.

Same with the naive belief that by using it we're contributing to global warming.

Anubis endgame = Coinhive?

Posted Jul 15, 2025 7:56 UTC (Tue) by gioele (subscriber, #61675) [Link] (2 responses)

> Iaso said at BSDCan 2025 that they were not sure what the end game for Anubis is:
>
> > I want to make this into a web application firewall that can potentially survive the AI bubble bursting. Because right now the AI bubble bursting is the biggest threat to the business, as it were.

The only possible endgame I see is that Anubis evolves into Coinhive [1]: if you want to access this page you have to spend a few seconds of CPU time to mine a crypto-currency as part of this pool, whose proceedings will be split between Techaro and the site owner. Cloudflare is experimenting with something similar.

[1] Weird: Wikipedia in English does not have a page dedicated to Coinhive. In its lieu: https://en.wikipedia.org/wiki/Monero#Mining_malware

Anubis endgame = Coinhive?

Posted Jul 15, 2025 21:29 UTC (Tue) by eean (subscriber, #50420) [Link]

tons of business models for a company with zero employees, maybe one soon

Anubis endgame = Coinhive?

Posted Jul 18, 2025 14:40 UTC (Fri) by paulj (subscriber, #341) [Link]

I also think ultimately all this evolves into the work being put towards crypto-currency, and rewarding the parties in some way. My other comment: https://lwn.net/Articles/1028558/

I think you need to split the reward with the browser owner though - more than the site owner. The browser owner then decides whether the site owner is worth giving some of the reward, for the content they give. That helps solves the ad-spam/click-bait v user-appreciated-good-content-producers angle of the problem.

Anything replacing current practices is desirable

Posted Jul 24, 2025 6:15 UTC (Thu) by callegar (guest, #16148) [Link] (1 responses)

> But Anubis is a lesser evil when compared to having sites knocked offline by overzealous bot traffic or employing more burdensome CAPTCHAs

Many sites appear to employ policies based on operating system choice.

For instance, I have never been able to access the `vinted` web site using Linux, always getting messages like:

```
Your session has been blocked

We've noticed unusual activity with your session (IP xx.xx.xx.xx) that resembles automated or suspicious behaviour.

For security reasons, we’ve temporarily blocked your access. We’re sorry for the inconvenience.
```

regardless of the browser in use (firefox, chrome). Accessing the site *from the same* host using a virtual machine with a windows guest works just fine.

Anything replacing current practices is desirable

Posted Jul 24, 2025 12:02 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> For instance, I have never been able to access the `vinted` web site using Linux, always getting messages like:

I was able to load the homepage in a Firefox Private Window here. But I believe that anonymization settings make my User Agent string report as a Windows machine, so that could be why.