Home | Notifications | New Note | Local | Federated | Search | Logout

Note Detail

Dillo browser@dillo@fosstodon.org (2026-10-18 21:39:05)
Moved our git web server from cgit to stagit to reduce load on the server caused by crawlers, as it only uses static files.

https://git.dillo-browser.org/dillo/

Roughly 99.9% of requests are bots fetching commits and files over and over again at a rate of 50 req/s despite our robots.txt excluding any crawler. Stagit only exposes a small number of files and commits so they are not reachable to bots.

Sadly, many anti-bot protections rely on JS or fingerprinting which come with their own set of problems.
---Attachments---
image: https://cdn.fosstodon.org/media_attachments/files/117/461/878/978/456/861/original/592142c4f1fa8bde.png