Am 04.07.25 um 12:00 schrieb Gerd
Hoffmann:
Basically these AI scrapers do not care about any restrictions like robots.txt or whatever. They try access all pages and do so with ridiculous frequency.I'd name that DoS.
"Do not talk with terrorists" .. block theire entire networks. It's what we in our datacenter do.
Somethimes they use a Class C for the GET request and another Class C for the POST request ( in our case a WP cluster )
We stopped blocking ip by ip, we use /24 blocks now.
Theire entire buisness modell is based on our data , so if we stop that data flow, we hit them in the long term.
best regards,
Marius Schwarz
-- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
