Why Search Sucks/ Why Search Rocks

SMS Text

My introduction to search engines came about in 1994. I’d been using the “world wide web” in email, ftp, and newsgroup form for several years by then. I was hired by a company to the write the user documentation for their search engine module.

It was offline code and used for a number of high-profile projects around the world. And there were three levels of documentation.

After nearly two years of docs – I’d been taking a break from programming – I managed to get on the then new online search engine that the company created and became the first official webmaster in 1996. The online version doesn’t exist anymore, but the company is now huge and happily selling content management systems.

While I never saw any of the inner engine code, I gained an understanding of full-text indexing and retrieval methods, as well as spider behavior, etc., and have been hooked on search ever since, despite all the inherent limitations. Fast forward eleven years. Search interfaces haven’t changed all that much, excluding “Web 2.0” engines, which I hope to talk about in the near future. Search as it exists today still sucks. Here are a few reasons why:

  1. Millions of non-relevant results. I.e., non-semantic.
  2. False results on pages that have transient content. I.e., nah nah, can’t see me.
  3. Can’t find stuff on the Invisible Web – though that’s changing.
  4. Can’t always find the latest content as opposed to the most relevant.
  5. Limited search paradigms.

On the other hand, Google and the other big SEs are at least making an effort to provide more access to parts of the Invisible Web. In recent years, they’ve added search capability for images, audio, and video content, as well as some text content from dynamically generated sites.

Lets not forget the all the shiny new Web 2.0 search engines – some of which are implementing new search paradigms. They’re buggy, but they’re exciting. And the semantic web (aka Web 3.0 to some) may not be here quite yet, but tools like Yahoo Pipes sure take us a ways towards extracting meaning from content – in this case web feeds.

That said, I love search. I use it daily in my resesearch, which drives me to learn more efficient ways of finding what I need quickly. SEs and feed readers are my primary source of insipiration on a daily basis. Without them, I cannot see my freelancing being all that successful.


The very simplified diagram above shows the intersection of different “webs”, as well as where content is indexed. (The purple bubbles represent indexes.)

  • Traditional engines give us both the Visible Web (hence it’s name) and a sliver of the Invisible Web. (As implied by the size of circles, the Invisible Web is much larger than the Visible Web.) There are manually generated indexes into the Invisible Web, but a lot of them become obsolete quickly.
  • Social bookmarking sites and Web 2.0 engines give us only an intersection each of the Visible and Invisible Web, though it’s unlikely they overlap. Social bookmarking sites help expose the Invisible and this information is more persistent than older lists of “invisible” content URLs.
  • Custom mashup tools like Yahoo Pipes, OpenKapow, and Teqlo give us a view of Visible, Invisible and Semantic Web. At the least, they allow us to extract a bit of meaning and thus create if not expose Semantic content. Web 3.0 / Semantic search engines are apparently in experimental stages

As far as I’m concerned, there is a lot about search to be excited about.

Get the latest news from Search Engine Journal!
We value your privacy! See our policy here.
  • http://www.rajdash.com Raj Dash

    I of course forgot to mention RSS feeds, which help expose the Invisble Web. As well, I’ve left out private indexes to keep the diagram from becoming too cluttered. Have I missed anything else?

  • Charles Knight


    Have you seen the whole list of 100 Alternative Search Engines on Read/WriteWeb? It’s here:


  • http://www.rajdash.com Raj Dash

    @Charles: Yes, I have seen it but had forgotten. Thanks for reminding me.

  • http://www.yaxis.in Rajesh Kumar

    Very interesting. I like the diagram.

  • http://www.rajdash.com Raj Dash

    @Rajesh: Thanks. It’s not completely accurate, but I had a hard time express the intersections of social bookmarking and web 2.0. Maybe I need to re-look at the article Charles pointed above.

  • http://directory.vbulletinsetup.com brandon

    why is the page shifted to the left ?
    ps. im on ie6

  • http://www.rajdash.com Raj Dash

    @Brandon: I don’t honestly know. Maybe it’s the diagram. I’ll have to test the next post. Thanks for bringing it to our attention.

  • http://www.jaankanellis.com Jaan Kanellis

    Aren’t we seeing the subscription web appear often in the SERPs? NY Times for example? I would assume that would fall into part of the visible web, yet at the same time invisible without paying….:(

  • http://www.rajdash.com Raj Dash

    @Jaan: Good point. I didn’t want to get too complex with the diagram. NY Times is an example of one of those sites that are both visible and invisible, depending on which version of the URL you use for a story, etc.

  • http://www.odinjobs.com Carl Williams

    Search engine can be frustrating to use, especially Google- who I call the mafia of the internet. If you are looking for an alternative, try Quintura. We reviewed it last week. Check it out at our company blog: http://www.odinjobs.com/blogs/page/thatsinteresting

  • http://www.rajdash.com Raj Dash

    @Carl: Thanks for the tip. I’m finding a lot of fascinating new search paradigms being used, and no doubt we’ll continue to see more.