Reddit Blocks All Search Engines Except Google

In February 2024, it was widely reported that Reddit had struck a deal with Google. The contract allowed Google to train its AI model on Reddit’s content. The deal is reported to cost Google $60 million per year. While initially, the deal appeared like a licensing agreement for AI training, now it seems the deal has gone beyond the remit of the contract.

404 Mediareports that thanks to the AI licensing deal between Google and Reddit, Google is the only search engine allowed to crawl Reddit’s new posts and content. This is a significant development as it demonstrates Google’s anti-competitive behavior in the search engine space. Google is also routing massive traffic to Reddit.Google Search alternativeslike Bing, DuckDuckGo, Mojeek, or Qwant are unable to crawl new and recent Reddit posts.

Now,Search Engine Landhas confirmed with Microsoft that Reddit is indeed blocking other search engines except Google. The report says that Reddit modified its “robot.txt” file on July 1, 2024, blocking many search engines and AI bots from crawling its content. Surprisingly, Google is the only major search engine that can crawl Reddit’s content.

I did try to find new Reddit posts on Bing and DuckDuckGo, but they kept throwing older posts from a month ago. Search engines are respecting the “robots.txt” standard and have stopped crawling new Reddit posts.

It raises several questions about Google’s monopolistic behavior. Google is already being investigated by the US Department of Justice over its antitrust practices. This development may make things worse for Google.

Arjun Sha

Passionate about Windows, ChromeOS, Android, security and privacy issues. Have a penchant to solve everyday computing problems.

Add new comment

Name

Email ID

Δ

01

02

03

04

05