Wikipedia is suspect now too. They’ve sold the site to AI search. Hard to say that they had much of a choice, but the sure did it swiftly and gleefully.
Its all public anyways. Its more like they sold infrastructure access afaik. Which is a smart move. Money aside, unauthorized scraping is a much bigger PITA and can drain resources quickly, including financial resources. Selling access means you can regulate it and also make it go thru appropriate infra; plus it becomes an asset rather than a liability. Not a fan of the wikimedia org lately, but I don’t fault them for that one.
Exactly this. They are guaranteed to get scraped regardless. Ask any question of an AI model and the first ‘source’ it cites is almost always Wikipedia. (assuming it’s not like a super technical in-depth question)
Might as well save on server overhead costs and get paid while doing it instead of trying (and failing) to block every possible scraper.
My local LLM narrowly fetching and caching a page or two isn’t what’s been hitting infra across the global and making indie sites blow up or die out. Doesn’t matter if the small players can’t buy the same access as the big players; they don’t need it and it’s not really for them.
(liability to asset) For whom?
The balance sheet. I didn’t mean it in the abstract.
Wikipedia is suspect now too. They’ve sold the site to AI search. Hard to say that they had much of a choice, but the sure did it swiftly and gleefully.
Its all public anyways. Its more like they sold infrastructure access afaik. Which is a smart move. Money aside, unauthorized scraping is a much bigger PITA and can drain resources quickly, including financial resources. Selling access means you can regulate it and also make it go thru appropriate infra; plus it becomes an asset rather than a liability. Not a fan of the wikimedia org lately, but I don’t fault them for that one.
Exactly this. They are guaranteed to get scraped regardless. Ask any question of an AI model and the first ‘source’ it cites is almost always Wikipedia. (assuming it’s not like a super technical in-depth question)
Might as well save on server overhead costs and get paid while doing it instead of trying (and failing) to block every possible scraper.
Selling access means the content is primarily available to the extremely privileged - huge “AI” grifts, etc.
For whom?
My local LLM narrowly fetching and caching a page or two isn’t what’s been hitting infra across the global and making indie sites blow up or die out. Doesn’t matter if the small players can’t buy the same access as the big players; they don’t need it and it’s not really for them.
The balance sheet. I didn’t mean it in the abstract.