Implicit and Explicit Personalization in Search
Starring 
Exalead
and 
SurfCanyon
What are the weaknesses of the major search engines that you are trying to overcome?
In 2007, we conducted a study with 1000 Internet users, and we learned that 1 in 5 users felt overwhelmed by the volume of results their searches returned, and that 1 in 2 users never ventured beyond the first page of results. Given the massive amount of information available on the Internet, it’s distressing to know so many users are abandoning their quests when all they need and more is no doubt available. One has to ask, “Is this hit-or-miss keyword approach the best choice for accessing the richness of the Web?” It’s an especially pertinent question given that on major engines the first page of results is often dominated by spam sites and irrelevant commercial pitches, all pushed to the top of the heap by armies of search engine marketing gurus.
Since its creation in 2000, Exalead has striven to provide it users with an alternative path for exploring the Internet, to protect Internet users’ freedom of choice. To keep the massive amount of information users face both on the Web and at the office accessible and intelligible, we offer ample yet utterly simple tools for efficiently navigating and refining results and enjoyably exploring content. Traditional search engines, on the other hand, don’t give users a choice beyond simply clicking < Next > to move on to the next page in an infinite laundry list. However, we’ve been highly pleased to see that our approach, which we call “Search by Serendipity,” is being taken up bit by bit by the heavyweights.
For all but the simplest navigational searches, too much time is spent trying to find the magic query that yields the desired information on the first page of search results. The search engine should actively work with the user to help them find what they need.
How do you define both explicit and implicit personalization?
With explicit personalization, the user in the driver’s seat. Within the limits of the available tools, users can customize and filter resources on a site in the manner that best suits their needs.
With implicit personalization, it is the site itself that adapts and conforms independently to the needs of the user. It can automatically adapt its resources to the needs of a user (whether individually or as part of a user group) based on the user’s profile and prior behavior.
Our search platform incorporates elements of both types of personalization. Through our public Web engine, we propose a wealth of explicit personalization options for our users, responding as effectively as we can to the heterogeneity of the Web and Web users. For B2C portals deploying our search engine on their own sites, we offer a number of implicit personalization tools as well. These types of tools are best suited to a more homogeneous audience and content base, with the expectations and needs of users being easier to identify-and hence leverage-in the quest for an improved user experience.
Speretta and Gauch wrote “Personalization is the process of presenting the right information to the right user at the right time.” Explicit and implicit personalization are two non-exclusive and complementary implementations.
Explicit personalization specifically asks the user to provide more information about their information need(s). This information is sometimes gathered through long-term general interest surveys (e.g. “what are your hobbies, what is your profession, where do you live?”). For specific information seeking tasks, explicit personalization sometimes asks the user to narrow the search via drill-down menus or by selecting information clusters. In classical information retrieval research, explicit relevance feedback is a technique where the searcher is asked to manually rate documents they have seen in order to more accurately fetch additional documents. Information directly gathered from the user is very powerful for tailoring a search session to the specific user.
Implicit personalization relies on the more ambiguous but more easily obtained history of user behavior to indirectly infer the user’s information need(s). Implicit personalization can be long term (e.g. a person’s hobbies can often be inferred from the web sites they visit repeatedly) or short term (a person’s immediate information need can be inferred from their current actions and context). Implicit personalization can also be collective (e.g. your friends like this so you may be interested in it). Although each user action is only weakly correlated with their long-term interests, the combination of multiple weak signals can lead to very powerful inferences. Properly interpreted, user behavior can lead to strong conclusions about the user’s short-term interests.
What are some popular examples?
Exalead.com is a good example of explicit personalization. All users are presented with the same result sets based on their query keywords, but the presentation makes extended use of keyword clustering to allow users to personalize these result sets based on their needs and interests. We deployed clustering in advance of Ask, and even Google, setting trends in this as well as other areas of Web search technology.
An example of implicit personalization is Last.fm. Last.fm shows that the implicit web is already a reality. Last.fm recommends new music to a user by independently analyzing the user’s existing library of favorite artists. Our purchases, our navigation histories and our search queries feed these types of recommendation engines and can help them refine the World Wide Web for us.
In the context of search, popular examples of explicit personalization are geo-targeting through IP addresses (e.g. if you search for “Ford” he advertisements will often be for local car dealers), related query suggestion (Yahoo! Search Suggest), categorical and hierarchical rill-down (Exalead, Clusty, Endeca, etc.), and explicit user profiling (when Netflix asks you to rank movies that you have seen).
Implicit personalization is used by Google (when you have activated the option) to provide a few search results that take into account your rowsing history. Amazon suggestions are based on implicit personalization although the suggestions are often humorous if you are like me and share an account with your spouse. Surf Canyon is the first company to provide real-time implicit personalization.
What are some of the major advantages and disadvantages of explicit personalization?
The freedom to explore, a uniform user experience. When resources are not filtered according to a user’s prior actions, then that user retains total liberty to explore and discover the complete universe of content available to them. A standardized, non-personalized presentation of resources also constitutes a shared, even egalitarian experience, with each click guiding users to the same results. On the other hand, leaving personalization options solely in the hands of the user can make it more difficult for him or her to fulfill their needs because the system cannot capitalize on a user’s prior actions to help divine their intentions and guide them through what is often an overwhelming amount of information.
The advantage of explicit personalization is that the information is unambiguous. When the user selects a related query or chooses a category among the possibilities presented, the search becomes much more accurate.
The largest hurdle for explicit personalization is that it requires additional work on the part of the user. Web users are overwhelmed by requests for our attention and often don’t want to exert additional effort even if it provides real benefit.
Another major disadvantage is that a user might not be fully cognizant of what they are looking for. If I am using a search engine with categorization, I might end up in the wrong category before I know enough to select the correct category. Even worse, any system that requires the user to explicitly profile themselves is susceptible to people describing who they want to be rather than who they are. As Jason Fry wrote in the Wall Street Journal, “…we lie — and never more effectively than when we’re lying to ourselves. My dissatisfaction with Amazon’s recommendations may have more to do with my view of myself than it does with Amazon’s engine: I fancy myself a reader of contemporary literature and history books, but I mostly buy “Star Wars” novels and “Curious George” books for my kid.”
What are some of the major advantages and disadvantages with implicit personalization?
This interesting approach points to one of the great paradoxes of the Web as revealed by our 2007 study: while 80% of the participants expressed a desire for personalized content, 50% evinced a strong resistance to divulging personal information in exchange.
We think implicit personalization is not well adapted for use in general Web search, if it is offered in isolation. The world of the Internet is infinite and deeply heterogeneous, and the behaviors of individual users are not really predictable. If I search on “leopard,” am I looking for a picture for my kid’s school report, or for information on the latest Mac OS?
On the other hand, the use of implicit personalization is very interesting in more constrained contexts. Our enterprise search software for portals (e-commerce, community sites, content portals, etc.) provide the intelligence needed to deploy implicit personalization and therefore to improve the user experience and augment site revenue.
Of course, implicit personalization raises privacy concerns. Users are rightly concerned as to whether the data they provide or their patterns of navigation for a particular site will be exploited in an unwelcome manner in another context.
The major advantage of implicit personalization is that it requires no additional effort on the part of the user. When implemented properly it can be very effective in providing accurate recommendations to the user.
The major disadvantage of implicit personalization is that it can sometimes make incorrect inferences, especially when the data is sparse or when the user context changes. My long-term interests may be at odds with my short term information needs. As an example, if I search for “Giants Stadium” and hail from San Francisco I am much more likely interested in the place where San Francisco Giants play (I have forgotten the stadium ame since they seem to have a new sponsor every year). Next fall, however, I might find myself in New York and interested in the parking at the Meadowlands.
Surf Canyon does implicit personalization in real time, and the sparse data problem is something we work to overcome. Fortunately, for the most difficult searches we often have the greatest amount of user behavior data and can therefore be of greatest benefit.
Which has the greatest potential to improve the search experience?
They both have great potential to improve the search experience, but only if they are always presented in a transparent manner, with the user retaining the right to choose one type or the other according to the usage context. Again, our approach to Web search rests on leaving choice in the hands of the user, and on respect for users’ privacy.
Both forms of personalization have great potential for improving the search experience, and even more so if used together! For the majority of queries, personalization in any form is unnecessary. The search engine has essentially replaced URLs and bookmarks for navigating to websites, and personalization does nothing for these navigational queries. If the user doesn’t end their search session with the first click, implicit Real-time personalization should begin working with the user to help them find what they need. At the same time, explicit personalization options should be made available to further help the search. Different Searches will require different tools. As the volume of web content grows, users will demand access to all of these tools.
May 21st, 2008 at 6:05 pm
THis is now on sphinn - go sphinn it!