https://discord.gg/tcaYeHDz
join the discord to discuss testing
I'm in the same boat, these ai crawlers are nuts and they dont respect usual robot blocks in most cases, you have to manually block entire subnets in some cases, driving me crazy.Steven wrote: ↑27 Feb 2025, 09:55Fingers crossed...
Yesterday caught this meta bot, also new and also for AI learning stuff. That bot crawls from many different IPs, so until you properly recognize them they are registered as separate guests. Over past 3 days we have more than 150K hits from them.
This AI LLM thing is stupid.
I see many people use it because they're too lazy to use their own brain.
And all these bots need to run and consume energy while they put lots of load on all the websites they visit as well.
.
.
over and over again.502 Bad Gateway