Reddit Is Giving Google a Search Monopoly
“Reddit” has quickly become one of my favorite search engine keywords. When I want to know what actual people have to say about restaurants near me or a hotel I might be looking at (as opposed to a bunch of ads or questionable AI junk), Adding “Reddit” to the end of my search is a good way to get it. But going forward, you'll only be able to use this trick with Google: Microsoft today confirmed that Reddit is blocking Bing from accessing it, and a 404 Media investigation further showed that most non-Google sites don’t work with Reddit anymore.
The news comes after an update to Reddit’s robots.txt file, which controls which search engines are allowed to crawl through the social media site for information. Search experts later took to X (formerly Twitter) to report they had noticed Reddit seemed to be blocking Bing and other search engines, after which Microsoft confirmed to Search Engine Land that “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”If it seems like it took a while for the news to circulate, that’s because it can take time to catch these kinds of behind the scenes changes. Bing and other affected search engines can still show old links their robots had already crawled, but going forward, they won’t be able to gather new ones. When reporters noticed something fishy was afoot, they reached out for details, although Microsoft is the only company to provide official confirmation as of yet.
Still, according to 404 Media, Reddit isn’t populating new links on other popular Google alternatives like DuckDuckGo, either. The outlet did say links are still updating on independent paid search engine Kagi, but it's worth noting that Kagi buys some of its search index from Google.
Currently, there’s no official reason for the lockout, but the news coincidentally follows a $60 million-a-year deal that allows Google to train its AI on posts from Reddit users. In a statement to The Verge, Reddit spokesperson Tim Rathschmidt said, “This is not at all related to our recent partnership with Google,” before clarifying that “We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
It’s true that more search engines are getting in on the AI game, but reading between the lines, it seems like Reddit’s concerns for its users’ content could actually be a hint that companies now need to pay to play with the social media giant, and that so far, Google is the only one that’s been willing to pony up.
Despite the recent $60 million-a-year agreement between Reddit and Google for AI training, Reddit has decided to block Bing and other search engines from accessing its content for crawling. This decision has also affected popular Google alternatives like DuckDuckGo, highlighting the potential need for search engines to pay for access to Reddit's content to use in their AI development. In relation to technology, this move underscores the growing importance of data and its monetization in the tech industry. Whenever you're conducting a search on Google, remember that Reddit's agreement with Google might impact the results you get, especially if the information you're looking for originates from Reddit.
