As you know for your numbers to be compared to the Trec legal track it would be helpful and more accurate to run your test against the same data as it is the only way to do a proper apples to apples comparison.

Even if the data you used is similar to the Trec data it is not a valid comparison, which is why Trec participants all use the same baseline data for the competition. The numbers you site as far as I am concerned are not valid as they do not use the same data.

Your technology sounds very interesting but all the posts I have seen from your company seem to be simply look at me too, invoking the name of your “compeditors” to get hits rather than instructive posts about your technology and where it fits into the wider world.

The above post is an example, it really wasn’t that helpful.

Cheers,

Sam

In response to the concerns about higher computation costs, statistical semantic approaches, such as LSI, do indeed experience exponential growth in computation resources because they have to compare all the co-occurrences in an entire document base. On the other hand, Semantic NLP based upon linguistic approaches, does not have that problem. CognitionSearch, for example, indexes documents about half as fast as a typical pattern-matcher technology, and the function is linear as it scales. Its algorithms use bottom-up interpretation — word-by-word and sentence-by-sentence.

To answer the concerns about scientific measures, precision and recall are standard measures of Search engine performance. Precision is a measure of retrieval accuracy calculated by dividing the total number of relevant retrievals by the number of all retrievals generated by the Search. Recall is a measure of the extent to which relevant material in the total document base is found. It is calculated by dividing the number of relevant retrievals by the total number of potentially relevant retrievals in the document base.

Pattern-matching technologies perform with both low precision and low recall (typically under 20% for both ). The TREC (Text Retrieval Conference), sponsored by the National Institute of Standards and Technology (NIST), is a recognized source of precision/recall testing for various technologies, including pattern-matching and statistical approaches. In TREC’s legal track competition in 2007, there were 13 technologies participating. Their precision performance ranged from under 1% to 23% and their recall performance ranged from under 1% to 22%.

While Cognition did not participate in the TREC competition in 2007 (but is participating in 2008), it did conduct its own internal precision/recall tests on a wide variety of document bases (similar to the TREC data) and Websites. These included the National Library of Medicine’s MEDLINE™, the public domain Enron fraud case, the public domain Microsoft anti-trust case, the BBC World News Website (http://news.bbc.co.uk/), and the Global Issues Website (http://www.globalissues.com), among others. For each test, 50 queries that were considered likely to be asked by users of the data/Website were formulated and posed to a CognitionSearch Search function on the sites documents. Relevancy was judged for a sample of 20 or fewer retrievals and extrapolated. Cognition’s precision exceeded 90%. Recall was measured relatively. In other words, full recall was taken to be the total of all relevant retrievals returned by any of the Search engines used in the particular test. Cognition’s relative recall in these tests exceeded 90% relative recall.

In summary, by employing Semantic NLP technology, Search results will achieve significantly better precision and recall than pattern-matching or statistical approaches.

1 [TREC Conference 2007]

Thanks for mentioning hakia here. I just wanted to clarify one point. Our semantic resources are actually transparent. You can interact with a limited dataset at http://labs.hakia.com/hakia-lab-onto.html. There is more information about the underlying theory, Ontological Semantics, and its application to Web search here: http://www.ontologicalsemantics.com/.

Best,
Dr. Kiki Hempelmann, Chief Scientific Officer, hakia

The example query, “strike up an intimate friendship”, goes too far, IMHO. Even if the search engine understands the meanings of the words, the phrase is still too vague to be useful. The functionality implied here seems to skip a level that would be much more useful. If you could make a query like, “forums that do not charge a fee on which I can meet smart people who like to talk about personal relationships”, return relevant results, that would be better.