Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.
Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.
The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).
In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.
Huffman said:
Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.
“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.
Damn I feel sorry for the guy now /s
Honestly, my biggest issue with LLMs is how they source their training data to create “their own” stuff. A meme calling it a plagiarism machine struck a chord with me. Almost anyone else I’d sympathize with, but fuck Spez.
Yep they now get paid for the data we have them. I have no sympathy lol. At least these models can’t actually store it all losslessly by any stretch of the imagination. The compression factors would have to be like 100-200X+ anything we’ve ever been able to achieve before. The numbers don’t work out. The models do encode a lot though and some of it is going to include actual full text data etc but it’ll still be kinda fuzzy.
I think we do need ALL OPEN SOURCE. Not just for AI, but I know on that point I’m preaching to the choir here lol
What resonated with me is people calling LLMs and Stable Diffusion “copyright laundering”. If copyright ever swung in AI’s favor it would be super easy to train an AI on stuff you want to steal, add in some generic training, and now you have a “new” piece of art.
LLMs and Stable Diffusion are just compression algorithms for abstract patterns, only one level above data.
Reddit is dying anyway.
It’s easy to say this as someone who is “on the other side”. But the data doesn’t really back up that statement.
I don’t have data, but the quality of the content certainly seemed to be declining, even as the quantity went up.
I think autocorrect might’ve gotten you: You posted “quality” twice in a contradictory way.
In all the ways that matter, it’s already dead. Once something enshittifies beyond a certain point, is its zombified, shambling corpse really considered “alive” anymore?
It’s awful. Politics is unavoidable at this point, and the amount of general anger on the platform is crazy.
People love watching their videos of people getting TBIs… Or getting too excited about a “justice served” post where a woman gets hit.
It’s kinda nice to see someone get their comeuppance, but then you look in the comments and there are just weirdos saying stuff like “glad that bitch got hit”, like… wtf?
Late stage capitalism
Honestly, any platforms hosting user-generated content who use the legal argument that they only provide hosting and aren’t responsible for what their user post shouldn’t also be able to sell the same data and claim owning any of it.
Otherwise, take away their legal immunity. Nazis or pedophiles post something awful? You get in front of the judge.
edit: typo
Can’t sell something you don’t own.
So if they’re selling the parts people want, they need to own the parts no one wants.
Well, you can give money to Reddit for a piece of paper, but unless Reddit is claiming copyright to the content posted there, then they can’t sue anyone for not paying. It would be very interesting to see the text of these “licensing agreements”.
Exactly this. You can claim that their scraping is abusing your servers, but the moment you claim copyright for the content of the site, then you give up your Section 230 rights.
You’d also probably lose a whole lot more processing power trying to stop the crawlers vs just letting them have API access with some sort of limit to queries.
Eh, not really.
I block bot user agents to my Lemmy instance, and the overhead is pretty negligible for that (it’s all handled in my web firewall/load balancer).
Granted, those are bots that correctly identify themselves via user agent and don’t spoof a browser’s.
It’s also cheaper and easier to add another load balancer than size up or scale out my DB server to handle the bot traffic.
I don’t think they actually block malicious bots, the change they’ve made is just to the robots.txt, they don’t have to do anything.