Wikimedia Clarifies it is Not Building a Global Web Crawler

SMS Text
Wikimedia Clarifies it is Not Building a Global Web Crawler

Recently released documents detailed the terms of a grant received by the Wikimedia Foundation to work on a search projected called “Knowledge Engine.” The terms of the grant suggested plans for the nonprofit organization to build a search engine that could potentially compete with Google.

Wikimedia Foundation has responded to reports that have been published since the documents were made public, clarifying what it intends to do and not do.

“What are we not doing? We’re not building a global crawler search engine… Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects.”

Wikipedia and its sister projects contain over 35 million articles across nearly 300 languages. Citing a need to improve the search and discovery of its projects, Wikimedia Foundation lays out its plans as a result of receiving the grant:

“We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects.”

Where the Confusion Came From

Plans for the Knowledge Engine project now sound more ambiguous than what was originally laid out in the grant terms. The documents outline plans to “democratize the discovery of media, news, and information,” by providing an opposition to “commercial search engines.”

In addition, the documents state success of the project may be at risk if “Google, Yahoo, or another big commercial search engine” attempt to develop something similar.

According to what is being described now in Wikimedia’s latest blog post, the organization is working on a new engine for search and discovery within its own projects, driven by community feedback.

That’s quite a contrast to the original grant application documents. At least now there’s some clarification on Wikimedia’s future plans.

Featured Image Credit: dennizn /

Matt Southern
Matt Southern has been the lead news writer at Search Engine Journal since 2013. His passion for helping people in all aspects of online marketing... Read Full Bio
Matt Southern
Subscribe to SEJ!
Get our weekly newsletter from SEJ's Founder Loren Baker about the latest news in the industry!
  • jytdog

    Your article is really troubling. It does seem very true that the Knowledge Engine will not “crawl” like Google – it was never intended to and there is no – zero – confusion that it would do that. Instead the WMF apparently intends to build up Wikidata, and link other database to Wikidata, and use that to formulate responses to queries. It is very much intended to compete with other search engines with regard to being the place you go, to find information. This is all explained kind of clearly in the Knight grant application – but there is no reference there to crawling and nobody really following this, ever thought there was. The WMF blew smoke at you – the diversion that the KE will not crawl – and you just regurgitated it.