

The problem isn’t that the data is already public.
The problem is that the AI crawlers want to check on it every 5 minutes, even if you try to tell all crawlers that the file is updated daily, or that the file hasn’t been updated in a month.
AI crawlers don’t care about robots.txt
or other helpful hints about what’s worth crawling or not, and hints on when it’s good time to crawl again.
Original definition: using AI to create a small bespoke kinda-crap-but-it-works apps or games for personal use. It’s a shitty proto but it’s interesting.
(My reaction: ooo not my jam but that sounds neat I guess)
Modern definition: it’s a multi-quadrispillion dollar industry and it’s the future according to some very important board members.
(My reaction: …capitalism ruins fucking everything)