On the Fly Query Entity Decomposition Using Snippets

This authors are aware of the requirements of current web development. At this moment, an emerging ecosystem of real time applications is growing in importance, and well known algorithms are being adapted to this settings and new methods are being proposed.

In this new Web environment the context changes very quickly as the news spread in real time through web communities (e.g. Michael Jackson’s death or Iran elections news arrived more quickly through Twitter’s trend topics than through any news agency).

This changing web environment lead to the main requirement of our approach: being able to offer good results not only for unexpected queries, but also for queries which relate to drifting topics of interest.

As an example, the query icelandic volcano could have probably be segmented as [icelandic] [volcano] before April 2010; after the impact of the eruptions of Eyjafjallaj kull on air travel a much better segmentation could probably be [icelandic volcano].

We are not saying, by no means, that methods based on prior training are not fast or feasible for this task. They are extremely fast after being trained and they have been applied to query segmentation. For instance, [8] propose describe a method based on processing a large corpus in addition to Wikipedia titles, to pre-compute query segmentations for n-grams before any actual query was provided. Their method relies on result caching and, hence, they report a query segmentation rate of 500 queries per second.

It is the speed of topic drifting and how to handle that changing trends which we feel can’t be properly achieved by using methods relying on training data and heavy result caching. Because of this, our approach consist of performing query segmentation on-the-fly. Of course, a balance should be reached as the results must be provided in a reasonable amount of time, in fact, repeated queries could and should be cached in order to avoid duplicate computations.

Authors are also conscious that returning fresh results demands a light algorithm which may not achieve as good performance as previous approaches based on training data. However, we feel that a certain lack of precision could be tolerable in exchange for a larger adaptability to drifting and evolving topics.

Previous researchers have discussed the nature of the MWUs a query segmentation algorithm should be able to found.

[8] argues that, to be correct, entities must have a ’complete semantic meaning’. This criteria introduces some ambiguity about the semantic completeness of an entity that is difficult to handle by any algorithm. Additionally, given the multilinguality nature of the Web, determining semantic completeness in an automatic fashion for any given entity in any given language could be insurmountably hard.

Other authors (e.g. [Risvik2003] or [3]) are not such concerned about the meaning of the queries and, thus, they assume that any set of words highly related according to the chosen statistical measure would be a MWU. This approach better handles some issues –such as multilinguality– but it can also be noisy and nothing prevents the system of extracting collocations or idioms as MWUs (e.g. ”good morning” or ”birds of a feather”).

In this paper we will refer as Entities to those MWUs which the algorithm found to be statistically relevant.

Actual details about our method are provided below, at this moment, it is enough to say it is based on the use of snippets. Snippets are short extracts of web documents usually containing some keywords. They are commonly seen in result pages of search engines but we are open to also consider other sources of snippets (e.g. a Twitter message containing some of the keywords from the query could also be considered a snippet). However, for the purpose of this paper, we have only used snippets from three major search engines (Google⁵⁵5http://code.google.com/intl/es-ES/apis/ajaxsearch/documentation/#fonje, Bing⁶⁶6http://msdn.microsoft.com/en-us/library/dd251056.aspx and Yahoo!⁷⁷7http://developer.yahoo.com/search/boss/) exploiting their respective APIs. This way, we have been able to compare the performance of our method using different sources of snippets.

In addition to our own method we tested several statistical measures previously employed in MWU detection; namely, Mutual Information, SCP, Phi, Dice and LogLike. [4] devised a way to generalize those well-known measures from bigrams to n-grams of any arbitrary length and, hence, we have used those more general versions. However, due to space constraints we are not able to give any detail about them and, so, further details and formulas are to be found in the mentioned paper.

As we have already said we have devised another method based on the use of snippets. The underlying idea is really simple. First, all possible n-grams in the query are produced. Then, the frequency for each n-gram in the snippets is to be found and those n-grams below a certain threshold are discarded. Finally, the query is segmented by using the most frequent remaining n-grams. The aforementioned threshold limits the minimum number of snippets a n-gram should appear in; for instance, a 0.5 threshold means that those n-grams not appearing in 50

We tested four different values for this parameter (namely, 0.25, 0.5, 0.75 and 1) so we can study how it affects to the performance of our method.

In order to evaluate the different query segmentation methods we needed some previously segmented queries. Those queries could have been segmented by ”experts” or by actual search engine users.

A dataset segmented by experts should provide a consistent set of segmentations (e.g query new york travel guide would probably appear as [new york] [travel guides] in all its instances). However, according to [3] the level of agreement between experts is relatively low (between 57.6% and 60.8%).

In contrast, a dataset segmented by actual search engine users should provide real information on how the users expect the search engine to group concepts; but with a much higher hetereogenity in the available segmentations because different users could understand concepts in different ways and, thus, would stress different segments within the queries.

Fortunately enough, we were able to found one dataset of each kind. The expert-segmented query log is the one described by [3]. This dataset (from now on we will refer to it as aol-bergsma-wang-2007) is a subset of the AOL query log [9] where the queries were manually segmented by the original authors of [3].

The second dataset, the one comprising queries segmented by actual users, was obtained from Dogpile SearchSpy. This second dataset (from now on searchspy-2010) is only comprised of queries including paired double-quotes which enclose two or more terms.

In both datasets queries were lowercased and to obtain the snippets double-quotes were removed to avoid the search engine to exploit that information.

At a first glance, one could think of using the percentage of correctly segmented queries to measure the effectiveness of the different methods. However, this approach does not take in consideration the existence of different degrees of segmentation correctness. For instance, the query new york travel guides can be segmented in several different ways ([new york] [travel guides], [new york] travel guides, new [york travel guides], etc). We could think of [new york] [travel guides] as the best segmentation and consider all the other ones as wrong segmentations. However, [new york] travel guides doesn’t seem as wrong as new [york travel] guides or new york travel guides.

In addition to these varying degrees of quality, it must be noticed that segmentation is a user-dependent task and two different users –even experts, as shown by [3]– can provide different segmentations; hence, a binary decision does not seem the best choice.

Both issues drive us to look for a different evaluation method judging individual segmentation decision instead of queries as a whole item.

We choose to use the well-known measures of precision and recall but taking into account not the queries but the blanks between segments both in the reference and the segmentation provided by each algorithm. This was based on the work of [Makhoul1999] and is a pretty common approach when evaluating segmentation tasks. For instance, in the previous example query new york travel guides and assuming the correct segmentation is [new york] [travel guides], the segmentation [new york] travel guides have a precision of 1 and a recall of 0.5 while segmentation new [york travel] guides have both 0 precision and 0 recall.

As we have already said, all of the methods to be evaluated rely on snippets to produce results. For the study described in this paper the snippets are obtained from different major search engines. Modern Web search engines take into account the relative positions of keywords both in the queries and in the results and, thus, one could argue that segmentation performance is not actually a produce of the different evaluated methods but of the search engine employed. In order to measure the bias introduced by the search engine, and dispel any concern about the actual performance of the proposed technique, we run the algorithms in three different ”flavors”. Each flavor rearranged the queries in a different way before submitting it to the search engine to obtain the snippets.

The first one sends the queries after removing all the double-quotes; the second one sends the original queries, that is including double-quotes; and the third one not only removes the double-quotes but also reverses the order of the terms in order to remove any term collocation within the query that could be used as a ”clue” by the search engine. For instance, the query "new york" travel guides would be submitted by each of the different flavors as: new york travel guides, "new york" travel guides  and guides travel york new.

This way, we have a base case in which queries are submitted with no segmentation information but keeping keywords ordering, a best case where all the segmentations within the query are preserved, and a worst case where neither segmentations nor the keywords ordering are preserved.

By doing this it could be possible to appreciate, albeit somewhat indirectly, the impact term collocation could exert on the results and, consequently, on the snippets and the different methods performance. Thus, if term collocations were heavily used by search engines to produce results then the differences between flavors 1 and 2 should be minimal. Additionally, by reversing the order of the terms in the query all valid collocations are removed and spurious ones are introduced; this should mangle the search engine results and, hence, make much harder to find valid segmentations. Of course, if even under this hard circumstances the methods manage to find correct segmentations we should discard the hypothesis of the search engine doing the work of detecting entities and should instead accept that the algorithms are actually finding them by means of the information available in the snippets.

Table 1 shows the P, R, and F measures for each statistic measure and each snippet source used in the experiments.

When Yahoo! Boss or Bing are used as snippet sources, F measures vary between 0.6 to 0.77; this is a reasonable performance for a technique that relies on relatively small information (10 snippets of text as a maximum) which would make statistical algorithms to work in an unreliable way.

Quite surprisingly, using Google as the snippet source drops F measures to very low values (from 0.48 to 0.61). This is shown in table 2 where a difference of 0.15 is shown between Yahoo! Boss and Google.

The main reason for such differences, beyond the quality of the returned results, is the amount of queries from the query log that Google couldn’t find snippets for.

All of the snippet sources showed this lack of results for certain queries, but in the case of Google there were about 1,100 queries which didn’t have any associated results. In the case of Bing less than 500 queries obtain no results, while Yahoo! did not return results for less than 300 queries.

We could have removed those ”failed” queries but because the proposed system is highly dependent on the snippet source we feel it was important to measure performance using the whole query log.

Concerning to the statistical methods, the Entity Frequency achieves the better performance, despite being the simplest one, with a difference of 0.07 from Loglike, the second top-ranked algorithm.

In table 3 we can see the results of our method when running on aol-bergsma-wang-2007 query log.

For this query log, F measures achieves better performance (between 0.07 for Boss and 0.23 for Google) as we can see in table 4. The main reason for this better performance is that very few queries obtained no results in the snippet sources because [3] selected only those queries with at least one clicked result in the AOL query log.

If those queries without results are removed the differences between snippet sources are not significant, but, again, using Yahoo! Boss achieves the better performance.

Concerning the algorihms, Entity Frequency behaves better with this query log than with the previous one but a more important boost is the one achieved by other statistic measures such as Dice, Mutual Information or Loglike. In fact, Loglike is the algorithm which achieves the highest values for R and F measures, although Entity Frequency still shows the highest value for Precision.

As we have previously explained, it should be confirmed that the results are not due to the search engines performing query entity extraction and thus returning results for those entities.

Due to space constraints tables summarizing the results for what we have called best and worst flavors can’t be shown, but Table 5 shows the variation obtained in F measure for the best and worst cases as described in Section 4.3.

It can be noticed that forcing quotes in the queries led to an increase in the number of queries no obtaining results.As we have seen before, this lack of results introduces some bias on the snippets source performance so we are going to clean up the datasets of those queries with no results This way, variations on the snippets source performance will not be caused by the quality of the query but by the snippet source itself.

Although differences in best cases are not really significant (between 0.04 and 0.09) when we compare them against differences with worst cases we find that best cases are very different from the base case.

So, in short, by means of these experiments comparing best and worst case scenarios, we can safely assert that (1) our technique is actually performing query segmentation –certainly by means of the snippets– and (2) it is not inadvertently taking advantage of any underlying query segmentation or entity detection phase performed by the search engine.

In this paper we have revisited the challenges IR systems must face with regards to the lack of context for user goals. Such challenges are growing because of the currently emerging real-time scenario which requires new solutions to handle not only such a lack of context, but also the fast drift of the topics underlying a given query.

We have described how query entity extraction can be a useful tool to tackle with this problem, and we have proposed using short snippets to obtain –by means of simple statistical methods– the most relevant MWUs.

Such methods have been implemented and some experiments have been performed on two previously segmented query logs: one of them by a panel of experts and the other one by actual Web searchers. Preliminary results are pretty good (F-measure is about 0.8) and, thus, further research in this line should be done.

searchspy-10 query log reflects real users’ behavior with regards to query segmentation (let’s remember it comprises double-quoted queries as issued by the users). The main conclusion one can reach from the experiments conducted on this query log is that performance is highly dependent on the underlying snippet source.

Preliminary work using the Wikipedia search engine as a source of snippets has shown that Wikipedia can be a very good source for certain kind of queries but not that good for other ones.

Two algorithms outperform the others in both experiments: Loglike and Entity Frequency. Hence, it seems that final choice for our method is between these two statistical approaches.

On one hand, Loglike achieves the highest performance for all the experiments, its behavior is highly irregular and does not handle very well queries from ”actual” users (i.e. those from the searchspy-10 query log). On the other hand, Entity Frequency provides much more consistent results for both query logs achieving similar performance.

Besides, Entity Frequency has been studied with different thresholds, exhibiting different performance values –specially for precision and recall– and this is also a variable to take into account.