Can Powerset Unseat Google in Web Search?
Recently I was doing some research for a client about the relevance of search results. Part of the task involved comparing the search results from Google and Yahoo! for the same query. I couldn’t believe what I found - although Yahoo! matches Google for some queries, for the majority of the queries I tried, Google results were convincingly more relevant than Yahoo! results.
Even if the Yahoo! search algorithms were to suddenly and magically improve to catch up to Google, beating Google in relevance would be very different ballgame, because the Google results set such a high bar in the first place.
What about an alternate search engine that comes in with an entirely new approach - could it catch up to or even beat Google? The search engine that immediately comes to mind is Powerset, the not-quite-so-secret search engine with semantic processing technology that’s the darling of the tech media. Although some industry observers think it’s over hyped, a majority of tech bloggers and especially journalists in the MSM are gaga about it.
Indeed, when I saw an early demo last year, I was impressed enough to call it a potential Google-killer. I have now changed my mind, for reasons that are only partially based on search technology.
Regardless of how good Powerset’s semantic matching algorithms are, and how effective they are at finding relevant search results, I think there are three reasons why Powerset will have a great deal of difficulty in unseating Google as the leader in Web Search:
1. Relevance Algorithms
This is the simplest and most obvious difficulty. Google has a FORMIDABLE lead in its search algorithms. The intial PageRank algorithm was a brilliant breakthrough, and Google has spent a great deal of energy and resources since then in trying to improve its search technology to produce increasingly relevant results, across a broad spectrum of information - including different types of media and many different languages. By itself, this is a significant barrier that Powerset and any other search engine will have to overcome in order to beat Google; yet this is only the ante, the basic requirement to even play in the game.
.
2. Optimized Websites
As I was thinking about Google’s significant lead over Yahoo! in relevance of search results, it suddenly struck me that Google’s algorithms are only part of the reason. The other part is that, as Google has slowly ascended to its position as the preeminent web search engine, web sites themselves have been reacting to this reality by changing their web pages to fit Google’s algorithms.
In other words, not only is Google tweaking its algorithms to return relevant web pages for a given query, but the web sites are also tweaking themselves equally to be relevant to those queries (SEO gaming aside, of course!).
3. User Inertia
The twitterati aside, a majority of users are slow to adopt new technologies. Although there are over 1000 alternate search engines, their combined market share is only about 1.7% of the overall market ; in contrast, Google alone commands a market share of about 70%.
A large majority of Internet users now use the Google search engine by default; they are all slowly and inexorably getting trained in the “Google way” of searching: keyword-ese, a highly minimalist user interface, the SERPs results page, and so on. Whether or not it’s the best way to do things, that’s what users are accustomed to, and that’s what they expect.
This means that even if a new search startup comes along with a better user experience (say, a radical new User Interface) or more relevant search results, this new company will have an uphill battle in convincing a large-enough number of users to even try out their search engine, let alone to switch. (The techmeme-twitter-tech blog crowd don’t count!)
And until the percentage of overall searches hits a certain threshold (say 5%?), a search engine doesn’t enter the non-techie public consciousness, and therefore doesn’t hit the tipping point where the momentum accelerates dramatically.
This type of service-product-user co-dependence exists in other industries, and creates a tremendous inertia that is very hard to overcome.
An example of this type of symbiotic relationship is the current system we use to fill gas in our cars. As cars became more frequent a hundred years ago, gas stations started springing up everywhere to meet the need to fill up the cars with gasoline fuel; at the same time, users got used to this process of filling up cars. Apart from a few experimental outliers and some marginal technologies (like diesel), almost all the new cars that are built today are compatible with the existing system of filling gas, as are any new gas stations that are built today.
If an efficient new type of fuel is suddenly invented that needs a radically different type of fueling system (say, hydrogen vehicles ), this new technology now has a huge uphill battle in gaining acceptance: the cars won’t change until there are enough fuel-dispensing stations available, and the stations won’t change until enough cars use the new fuel. And neither will change until users are comfortable with the new versions of each.
In the same way, even if a new search engine suddenly becomes available that’s more effective than Google in searching (which is a BIG if), users may still prefer to use their current choice since the web pages being searched are themselves optimized for Google. At the same time, the pages won’t change until a majority of users change to using this new search engine. A true catch-22!
Of course, the Powerset folks realize that this is not an easy battle to win. In a recent interview with VentureBeat, Powerset founder Barney Pell laid out Powerset’s new unofficial motto: “We’re not a search engine.” Powerset may instead focus on trying to answer user questions by building tools to extract information from within web pages, rather than just trying to find the most relevant web pages for a user query.
I hope they do - this type of service would be incredibly useful to users. Most of all, I hope they hurry up and release something so that we can start using this wonderful new technology!
-Nitin Karandikar
April 15th, 2008 at 10:58 am
I would suggest a 4th challenge - a massive index of as much of the information on the internet as possible. Building an maintaining such an index has become prohibitively capital intensive, making it almost impossible for small entrants, even if they have tens of millions in funding. This is perhaps one of the reasons why Powerset decided to start with just Wikipedia.
By the way, I’ve seen a few startups who are looking to solve this problem by distributing the index. MyLiveSearch is one that comes to mind, although I think there are others. Avoiding the cost of building an index by spreading it across millions of client machines is an interesting approach, but I don’t think anyone’s come close yet.
April 15th, 2008 at 11:28 am
Diesel - marginal technology? Since when? Diesel accounts for approx half of cars in Europe?
April 15th, 2008 at 4:13 pm
Your analogy to the gasoline infrastructure is interesting, but makes too strong a statement about how search engines shape the Internet landscape. Certain aspects of pages (link structure, titles, etc.) are now optimized for search algorithms, but almost every page still contain a lot of unstructured text. By unlocking the meaning behind the text, Powerset hopes to enhance both the search and explore experiences of users.
I was playing around with an analogy of search engines being like roads and technologies like semantic search as improvements to the cars, but that doesn’t seem to capture the breakthrough features that we’re enabling with our semantic index.
I’m excited to see what your view will be after we launch!
-Mark Johnson, Powerset Product Manager
April 16th, 2008 at 12:24 am
Another interesting data point - Cuill apparently just raised $25m, bringing their total fundraising to at least $33m. According to TechCrunch, “Not a whole lot is known about Cuill except that it apparently can index the web at 1/10th the cost of Google.” Perhaps they’ve cleared the “building an index” problem.
April 17th, 2008 at 3:24 am
I think that changing habit is something that brand marketers excel at - being everywhere where the user is, and impressing them enough to use the search engine.
It’s not the quality of search - yet. You need critical mass before anyone starts saying “xxx search engine is better than yzz search engine”. For example, Yahoo and AOL used to have a large start on the market because they were the start pages.
However, Google gained alot of traction in the tech world as the best solution (as well as powering Yahoo and AOL’s search for a while, which boosted their brand. Yahoo had since played catchup (by buying both a natural search engine and the paid engine - and focused more on human editing (which is their strenfth) rather than just the algorithm itself. In blind testing, they’ve even won Google!
Regardless, once people started going “ga ga google”, the perception became that Google is the best, which slowly erodes market share. Yahoo and Microsoft simply can’t compete.
Additionally, I think that Google has gained a huge amount of traffic through their partnerships. Their first partnerships with AOL and Yahoo brought them to the forefront. However, their continuing relationship with Mozilla and Opera - offering Google as default search engines, and giving the open source foundation money, as well as powering sites search with google mini’s, has really made them a household name.
So anyone who wants to win the search engine war needs to have both “branding” - people seeing their search around, as well as rabid fans, along with “direct marketing” - allowing people to take that motivation and use it to test the search engine.
Here are a couple of companies that have unsuccessfully tried to get into the search door using direct marketing or branding:
1) A9.com - Amazon’s search engine. Amazon ran a promotion a while ago that offered anyone who used a9 a certain amount of times per month a discount on amazon products. This obviously didn’t go anywhere, as they haven’t continued it.
2) Microsoft live - They offered free prizes to people who tried the search in terms of software… that also failed horribly, as far as I know the only people I know who did that were geeks who were trying to game the system!
3) Ask.com - interesting idea, spectacular lack of skill in implementation, and terrible choice of medium. We all know about Ask.com’s brandning campaign across the UK. As soon as I saw I thought, these guys either have way too much money to blow, or they absolute morons (or their ad agency… or both). It turns out that they were absolute morons, and the $100 million dollars went to waste, and they are now focusing on the african American niche.
You can check out my full rant on what it takes to be a google killer on my blog…