

24·
6 days agoSearch engines been going relatively fine for decades now. But the crawlers from AI companies basically DDOS hosts in comparison, sending so many requests in such a short interval. Crawling dynamic links as well that are expensive to render compared to a static page, ignoring the robots.txt entirely, or even using it discover unlinked pages.
Servers have finite resources, especially self hosted sites, while AI companies have disproportinately more at their disposal, easily grinding other systems to a halt by overwhelming them with requests.
That’s actually hilarious, in a good way. But they never miss a chance to drop the PR ball. Would it be that hard to roll with it and earn some points with the community?
Competition of closed source software generally makes sense if you have the advantage in engineering resources. AMD is in a disadvantage on this front, so getting OS contributions could level the field.