Cooperation of Alt Search Engines: A Manifesto
In honor of Alternative Search Engines Day:
Background
I’ll be at the Alternative Search Engines Day tomorrow, a unique event in San Francisco put together by Charles and the AltSearchEngines team. The event is sponsored by SeeqPod, UpTake, Matchpoint, Cluuz, HealthPricer, GoPubMed and Blogdimension. (Unfortunately, it’s not open to the general public.) If you’re part of an Alternative Search Engine, I hope to see you there!
As I was getting ready for the event, it got me thinking about ASEs and how they can work together.
The Case for the Alts
I love the ASEs - Alts rock!
Without them, there would be little innovation in Search, no new frontiers to be explored.
The Alts are the ones that keep pushing the envelope with new directions in search technology, whether it’s algorithms, user interface, social search or something else. Although Google has some fine technology and is synonymous with search, I firmly believe that we’re still at Search 1.0, and have a long way to go. Because of all this competition from the Alts, and the resulting innovation, web search continues to improve.
Why Cooperate?
Although there are over 1000 alternate search engines, their combined market share is only about 1.7% of the overall market; in contrast, Google alone commands a market share of about 70%. As I suggested in an earlier post, market forces combine to further reinforce Google’s undisputed lead in web search. Until a web search destination reaches a tipping point in traffic (say, around 3-5% of market share) and people start passing on the message by word-of-mouth, there is no significant mindshare; without mindshare, the Alts will continue to remain marginal regardless of their algorithmic improvements , innovative new UIs or specialized focus.
Perhaps by pooling their resources in some areas, ASEs can make a dent in this dominance, which would benefit everyone.
How?
But how can ASEs cooperate in certain areas, while at the same time continuing to compete fiercely among themselves and with the major search engine leaders?
Here are five possibilities for consideration by the ASE community; doubtless there are many others:
1. Search Federations of complementary ASEs
Until the percentage of searches hits a certain threshold, a search engine doesn’t enter the non-techie public consciousness, and therefore doesn’t hit the tipping point where the momentum accelerates dramatically.
In order to generate meaningful traffic, complementary Search Engines could get together to offer users a common destination site on the web, so that overall web traffic climbs above the minimum threshold for getting traction. Most ASEs will not try to satisfy long-tail searches in all areas; if a partner provides those, while preserving a trivially-simple user experience (a la Google), the combined search percentage would shoot up.
It’s amazing to me that more alternate search engines don’t band together - especially those that service different domains/verticals/niches - in order to get a better combined market share percentage!
2. A Comprehensive Shared Index
(Thanks to Mark Cramer of SurfCanyon for this suggestion.)
The amount of content on the web is increasing fast, and building a massive index of as much of this content as possible is becoming prohibitively capital-intensive; especially so if you start adding in various types of media. There are already efforts to build an open web index, such as freebase from Metaweb, which need our support.
By building and using a common set of crawlers and a shared global index, smaller search engines could focus their attention on adding value in other areas of the search space.
3. Standardized UI Paradigms
Google’s minimalist UI is not the only approach, nor necessarily the best, for displaying search results. But users have a strong level of comfort with it; a radical new UI from an Alt, even if it’s better, pushes against this comfort level. If ASEs were to standardize on an alternative improved UI paradigm (which is unlikely) - or even a few - then it could be a competitive advantage. Instead of trying to out-Google Google, this could be a way to change the playing field.
4. Industry Benchmarks
Users have no idea if any other search engine gives better results than Google, and generalized claims of increased relevance by the ASEs are seen as vague and self-serving. If we can define a standard measure of Relevance of Search Results, then by applying that measure, an ASE could highlight its differentiation from the status quo. This measure would be available to the industry and to researchers in order to compare the relevance of results between search engines, and allow the search engines to measure their own progress.
[On that note: I've been thinking about a Measure of Relevance for search results, and discussing this with a few people. Such a measurement would have to take into account the different dimensions of search relevance: breadth, depth, disambiguation, reaction to trick queries and special innovations. If you're interested in participating in this discussion, please drop me a note at: nitink@softwareabstractions.com.removethis ].
5. Universal Identity
In the future, this is likely to be a key differentiator for web sites. Dave McClure called it the always-logged-in web: as users navigate between web sites, their identity, preferences and personalization are automatically carried around with them, of course subject to their level of privacy opt-in.
For users who want to take advantage of this feature, groups of cooperative search engines could preserve user context and history among many different web sites, using this information to provide more relevant search results.
Brave New World
Will the smaller search engines be able to set aside their competitive differences long enough to cooperate on these and other similar features? That remains to be seen. I know that Charles and the AltSearchEngines team, as well as others, are working on it. If this type of cooperation takes off, though, it can only be a good thing for the end user!
