There's AI slop and anti-AI slop

>I had to take down my gitweb instance because it was being DDoS'd by the scrapers.

I have put it behind a cookie check. The cookie is set (always) on the main non-gitweb/cgit entry site that humans would typically use for entry.
If that cookie is not set, Apache will redirect to a small html-only page explaining what is going on.

This works really well, but it took many months for the bots to drop their caches and finally give up fetching the html redirect page over and over again.
They do not seem to be smart enough (yet) to retrieve and store the cookie from the main site.
It immediately helped to reduce the server load to an acceptable level again, because the static page is much less resource consuming than gitweb/cgit.

Of course this also disables traditional scraper bots.

It is a medium annoyance since direct links now no longer work and require two extra clicks instead.
But that's Ok for my use case and it's trivial enough to implement.