Today’s Guest Author, Nitin Karandikar of The Software Abstractions Blog, kicks off a three part series on Alternative Search Engine Algorithms vs. User Interfaces.  Which matters more, beauty or brains?  Is it the snazzy look of the car, or what’s under the hood that really counts?  Nitin starts out solidly in favor of the Algorithm.  Next Thursday, as part two, I will pitch the Interface.  Finally, in part three, Kaila Colbin will be placed in the awkward position of deciding between us!

How can a search startup take on the Google juggernaut?

Given Google’s dominant position in the Web Search space, what does it take for a small company to start making inroads in terms of market share and thought leadership? A trio of small, well-funded startups are taking divergent approaches to break into this area of general-purpose web search: Powerset, of San Francisco, CA, is licensing technology originally from Xerox PARC to tackle the problem of Semantic Search; Mahalo, of Santa Monica, CA, is at work on a “human-powered” search engine; and Quintura, originally from Russia and now based in Alexandria, VA, is creating an effective “see-and-find” visual search engine for the web.

Components of Search

To compare these approaches, let us first consider the various components of a general-purpose web search engine. At a very simple level, you can consider it to be made up of the following four elements:

1.  The underlying Index of web-based content, which needs to be regularly refreshed.

2.  The core Algorithm for finding relevant results

3.  The Query Specification interface

4.  Results Display and Visualization 

[For the more technical-minded, here’s a picture of the Google architecture (via Phil Butler’s fine article about Semantic Web); or my earlier abstract architecture for Search.]

Any small search engine with limited resources is forced to focus on one of these areas to improve upon the major search engines.

Areas of Focus

So how about our three champions? They are indeed taking vastly different approaches:

Powerset (potentially a serious challenger to Google in the future) is using powerful new technology to create a fundamentally richer representation of the content in its index, which can dramatically improve the relevance of search results. For the Powerset approach, the algorithm is the key differentiator over competitors. [Another challenger to watch in this space is Hakia, which is also focusing on a “meaning-based” search engine.]
Note: Powerset has not yet been launched.

Mahalo, the latest venture from Jason Calacanis, uses human beings to create highly-relevant, spam-free results for a small set of very popular web search queries. This approach has been tried in the past (most notably by ChaCha ) without much success, and it’s debatable if it will succeed this time (as Rich Skrenta notes ). In this case, Mahalo’s selection of algorithm has a direct impact on the content; specifically, the scope of search queries supported is severely restricted.

Quintura is taking the opposite approach, by focusing on the UI. Quintura’s dynamic tag cloud allows you to visually navigate the results space, enabling the user to find related results, drill down into detailed results, and so on.

Which of these approaches will succeed in making a dent in Google’s dominance of Web Search?

Current Stats

The current traffic pattern and buzz for these three engines (and Hakia) are shown below. The first image depicts the Daily PageViews chart from Alexa; the second depicts trend history in Google Trends.

[I realize that Alexa data has been discredited to some extent, but the corresponding chart from Compete.com seems to be consistent with this one at a high level; I chose the Alexa chart because it can be configured to show only the past three months, which makes the data clearer.]

Alexa Chart: Daily Page Views

Google Trends Chart: Search Volume and News Volume

The Algorithm’s the thing!

My take? Although all of these components are extremely important - a search engine neglects any of them at its own peril - my money is strongly on the “improved algorithm” approach to success. Of course, I’m assuming that every serious contender has a basic level of coverage of web content, and a basic serviceable UI (after all, Google itself is the greatest proponent of the “minimal UI” approach). Given this basic level of functionality, I think that a search engine that provides dramatically better relevance in search results without much additional effort on the part of the user, is a much bigger threat to the dominance of the major search engines than one that provides an über-cool UI.

My rationale, for giving priority to the algorithm over the UI, is two-fold [¹]:

If a search engine does not provide a high degree of relevance in results, then even the greatest UI in the world will not help capture market share; conversely, a bare-bones functional UI on top of very good search results would, in my opinion, make a big difference.

An improved algorithm - be it semantic parsing of the query or index, or social-based approaches like Ant Colony Optimization, or something entirely new - is a feature that has to be “baked-in” into the architecture; I imagine it is truly difficult to add it in later without a big impact on existing users (e.g. the process of tweaking Google’s search algorithms seems to be excruciatingly complex). On the other hand, a superior UI on top can be added in later, as Quintura has proved by leveraging search results from Google and other major search engines.

Alternate Realities

As always, I’ve been having a vigorous debate on this topic with Charles Knight of AltSearchEngines and Kaila Colbin of VortexDNA - we have widely divergent views. In the blog posts to follow, they will be laying out other scenarios and different ways of thinking about it. You, dear Reader, get to choose which approach you believe!

[The latest Software Abstractions survey addresses this question. Do you have a strong opinion? Vote now and let us know!]

Footnotes:

[¹] Of course, there is UI design and there is UI Design. Apple has convincingly proved, not once but twice recently, that “design matters!”. Now if Apple were to start building a search engine tomorrow, I might switch my allegiance and become a believer in the dominance of the UI approach. [As Guy Kawasaki once told me in an email interview: “… when all is said and done, it’s very dumb to bet against Steve (Jobs)”].

This entry was posted on Thursday, September 13th, 2007 at 5:03 pm and is filed under Guest Authors. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Sphere: Related Content