Reddit CEO Steve Huffman is standing by Redditâs decision to block companies from scraping the site without an AI agreement.
Last week, 404 Media noticed that search engines that werenât Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: âReddit believes in an open Internet, but not the misuse of public content.â Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.
The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).
In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. Itâs unclear how much Redditâs OpenAI deal is worth.
Huffman said:
Without these agreements, we donât have any say or knowledge of how our data is displayed and what itâs used for, which has put us in a position now of blocking folks who havenât been willing to come to terms with how weâd like our data to be used or not used.
â[Itâs been] a real pain in the ass to block these companies,â Huffman told The Verge.
Everyone always says this like itâs some kind of gotcha, but all of my nuked posts still have my âfuck you, redditâ content and havenât been reverted. Itâs been nearly exactly a year.
Maybe reddit has an offline copy of my old content and that of others somewhere, but if so theyâd be handing that directly over to whoever under some kind of agreement â that certainly wouldnât be the subject of any kind of site crawling which is the crux of the issue here.
Youâre ignoring the idea that they could still be working on a way to restore content and havenât completed that process yet
Or that they could start feeding your archived (not cached) data directly to the AI companies anyway for a price
IMO, you can win by jamming your âtransmissionsâ with noise. Itâs easier to hide in noise as noise than it Is to be silent IMO. Muddy the waters as it were
Youâre ignoring the idea that they could still be working on a way to restore content and havenât completed that process yet
thereâs no evidence to suggest this, though.
Content is absolutely archived and they have financial incentive to restore the quality of their âknowledge baseâ
Thatâs a fair amount of circumstance and motivation to support my idea, regardless of tangible evidence
it never was deleted, all that happened is that an extra line was added to a database that said âcomment 65432426542654 now should be displayed as âfuck you, redditâ rather than the original textâ. The original post is still in an earlier row available to reddit, it just isnt being displayed on their web page.