Reddit Blocks Major Search Engines and AI Bots — Only Allows Paid Access

Reddit is intensifying its efforts to restrict web crawlers. Recently, the platform has begun blocking search engines from accessing recent posts and comments unless they pay for the privilege, according to 404 Media. Currently, Google is the only major search engine displaying recent Reddit results when users search with the “site:reddit.com” command. Other search engines, like Bing and DuckDuckGo, are excluded, likely due to Google's $60 million deal allowing it to leverage Reddit content for AI training.

Reddit spokesperson Tim Rathschmidt clarified, “This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines and have not reached agreements with all of them, as some are either unable or unwilling to commit to enforceable terms regarding their use of Reddit content, including for AI.”

To strengthen its policy against data scraping, Reddit updated its robots.txt file, which advises web crawlers about site access. Reddit’s chief legal officer, Ben Lee, explained this move: "It’s a signal to those who don’t have an agreement with us that they shouldn’t be accessing Reddit data."

In response, Microsoft’s Caitlin Roulston stated, “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models.” She noted that Bing ceased crawling Reddit following the robots.txt update on July 1.

While it may seem surprising for a major site like Reddit to block access for popular search engines, it aligns with the platform's recent trend of safeguarding its data. As Reddit seeks new revenue sources and aims to satisfy investors, it has raised API costs for certain developers and threatened to cut off Google if it continued using Reddit data for free AI training.

With AI-generated content saturating the internet, the demand for authentic human-written information is growing. Many users, including myself, have begun appending “Reddit” to their searches to find genuine responses, making it frustrating that this option will now primarily be available on Google and supported platforms, particularly as I often use Bing for my searches.