Towards Meaningful Statements in IR Evaluation
Mapping Evaluation Measures to Interval Scales

The representational theory of measurement [53, 87, 59] is one of the most developed approaches to measurement, suitable for many areas of science ranging to physics and engineering to psychology. The basic idea is that real world objects have attributes which constitute their relevant features and induce a set of relationship among them; the set of objects $E$ together with the relationships $R^{E}_{1},R^{E}_{2},\ldots$ among them comprise the so-called Empirical Relational System (ERS) $\mathbf{E}=\big{\langle}E,R^{E}_{1},R^{E}_{2},\ldots\big{\rangle}$. Then, we look for a mapping between the real word objects $E$ and numbers $N$ in such a way that the relationships $R^{E}_{1},R^{E}_{2},\ldots$ among the objects match with relationships $R^{N}_{1},R^{N}_{2},\ldots$ among numbers; the set of numbers $N$ together with the relationships $R^{N}_{1},R^{N}_{2},\ldots$ constitutes the so-called Numerical Relational System (NRS) $\mathbf{N}=\big{\langle}N,R^{N}_{1},R^{N}_{2},\ldots\big{\rangle}$.

More precisely, the representational theory of measurement seeks for an homomorphism $\phi$ which maps $E$ onto $N$ in such a way that $\forall R^{E}_{i},\,\forall e_{1},e_{2},\ldots,e_{k}\in E\mid(e_{1},e_{2},\ldots,e_{k})\in R^{E}_{i}$ it holds that $\exists\,n_{1}=\phi(e_{1}),n_{2}=\phi(e_{2}),\ldots n_{k}=\phi(e_{k})\in N\mid(n_{1},n_{2},\ldots,n_{k})\in R^{N}_{i}$. The homomorphism $\phi$ is called a scale of measurement. Note that, in general, we seek for an homomorphism and not an isomorphism because two different real word objects might be mapped into the same number.

The most typical example is length. Suppose the ERS $\mathbf{E}=\big{\langle}E,\succsim,\circ\big{\rangle}$ is a set of rods with an order relationship $\succsim$ among rods and a concatenation operation $\circ$ among them. If the attribute under examination is the length of a rod, we can map the ERS to the NRS $\mathbf{N}=\big{\langle}\mathbb{R}_{0}^{+},\geq,+\big{\rangle}$ such that $\forall e_{1},e_{2},e_{3}\in E$ it holds $e_{1}\succsim e_{2}\Leftrightarrow\phi(e_{1})\geq\phi(e_{2})$ and $e_{1}\circ e_{2}\sim e_{3}\Leftrightarrow\phi(e_{1})+\phi(e_{2})=\phi(e_{3})$, that is if a rod is longer than another one the number assigned to the first one is bigger than the number assigned to the second one and the concatenation of two rods corresponds to the sum of the two numbers assigned to them.

The core of the representational theory of measurements is to seek for a representation theorem and a uniqueness theorem for the scale of measurement in order to fully define it.

The representation theorem ensures that if the ERS satisfies given properties, it is possible to construct an homomorphism to a certain NRS. In the previous example, the representation theorem defines which properties the order relation $\succsim$ and the concatenation $\circ$ have to satisfy in order to construct a real-valued function $\phi$ which is order preserving and additive. It is important to underline that the representational theory of measurement seeks for “operations” among real word objects – e.g. we can put two rods side by side to order them or we can lay two rods end by end to concatenate them – and if these “operations” satisfy given properties they can be reflected into corresponding operations among numbers, where numbers are just a proxy of what happens among real world objects but are much more convenient to manipulate.

In general, given an ERS and an NRS, it is possible to create more than one homomorphism between them. For example, it is possible to express length by using meters or yards and both of them are legitimate scales for length. The uniqueness theorem is concerned with determining which are the permissible transformations $\phi\rightarrow\phi^{\prime}$ such that $\phi$ and $\phi^{\prime}$ are both homomorphisms of the given ERS into the same NRS. In our example, any transformation $\phi^{\prime}=\alpha\phi,\alpha>0$ is permissible for length. Therefore, the uniqueness theorem guarantees that the “structure” of a scale of measurement is invariant to changes in the numerical assignment, which preserve the relationships.

Stevens [85] introduced a classification of scales based on their permissible transformations, described below.

Stevens moved a step forward and linked the notion of scale with that of admissible statistical operations which can be carried out with that scale:

• •

  Nominal scale: the only allowable operation is counting number of items in each class, that is, in statistical terms, mode and frequency.

• •

  Ordinal scale: besides the operations already allowed for nominal scales, median, quantiles, and percentiles are appropriate, since there is a notion of ordering.

• •

  Interval scale besides the operations already allowed for ordinal scales, mean and standard deviation are allowable since they depend just on sum and subtraction⁵⁵5Note that when we talk about admissible operations, we mean operations between items of the scale. So, for example, a mean involves summing items of the scale, e.g. temperature, and this is possible on an interval scale. The fact that a mean also requires a division by the number $N$ of items added together is not in contrast with saying that only addition and subtraction are allowed, since $N$ is not an item of the scale..

• •

  Ratio scale: besides the operations already allowed for interval scales, geometric and harmonic mean, as well as coefficient of variation, are allowable since they depend on multiplication and division.

These prescriptions originated several debates over the decades. Lord [56, p. 751] argued that “since the numbers don’t remember where they come from, they always behave the same way, regardless” and so any operation should be allowed even on “football numbers”, i.e. a nominal scale; Gaito [42] reinforced this argument by distinguishing between the realm of the measurement theory, where Stevens’s restrictions should apply, and the realm of the statistical theory, where these restrictions should not be applied, since other assumptions, such as normal distribution of the data, are those actually needed. Townsend and Ashby [89] replied back showing cases where performing operations inadmissible for a given scale of measurement may mislead the conclusions drawn by statistical tests. O’Brien [68] discussed the type of errors introduced when using ordinal data for representing an underlying continuous variable, classifying them into pure transformation errors, pure categorization errors, pure grouping errors, and random measurement errors. Velleman and Wilkinson [95] summarized the previous debate and argumented that once you are in the numerical realm every operation is admissible among numbers. Recently, Scholten and Borsboom [80] made a case of flaws in the original Lord’s argument and, as a striking consequence, Lord’s experiment would not be a counterargument to Stevens’s restrictions but it would rather comply with them. In a very recent textbook, Sauro and Lewis [78] firmly supported Lord’s view, at least in the case of ordinal scales, but with the caveat to not make claims on the outcomes of a statistical test that violate the underlying scale. So, for example, if you are on ordinal scale and you detected a significant effect using a test which would require a ratio scale, you should not claim that that effect is twice as big as another effect but just that it is significant.

The above observation brings the debate back to the core issue of what we should pay attention to. Indeed, both Hand [45] and Michel [62, 63] argued that the problem is not what operations you can perform with numbers but what kind of inference you wish to make from those operations and how much such inference has to be indicative of what actually happens among real world objects. Already Adams et al. [2, pp. 99-100] explicitly stated that

Statistical operations on measurements of a given scale are not appropriate or inappropriate per se but only relative to the kinds of statements made about them. The criterion of appropriateness for a statement about a statistical operation is that the statement be empirically meaningful in the sense that its truth or falsity must be invariant under permissible transformations of the underlying scale

These statements opened the way to the development of a full (formal) theory of meaningfulness [29, 67, 69], which is a central concept to clearly shape and define the questions discussed above: according to the adopted measurement scales, what processing, manipulation, and analyses can be conducted and what can we tell about the conclusions drawn from such processing?

Note that the statement “A mouse weights more than an elephant” is meaningful even if it is clearly false; indeed, its truth value, i.e. false, does not change whatever weight scale you use (kilograms, pounds, and so on). Therefore, as anticipated above, meaningfulness is a distinct concept from the one of truth of a statement and it is somehow close to the notion of invariance in geometry, since the truth value of a statement stays the same independently from the permissible scales used to express it.

Siegel [83] and Senders [82] have discussed the implications of Stevens’ classification and permissible operations in the case of statistical inference and parametric and nonparametric statistical significance tests. We consider the following tests:

• •

  Sign Test [44] is a non parametric test which looks at the signs of the differences among two paired samples $x_{i}$ and $y_{i}$; the null hypothesis is that the median of the differences is zero.

  The sign test requires samples to be on an ordinal scale, since it needs to determine the sign of their difference or, equivalently, which one is greater. Note that the sign test discards the tied samples, i.e. when $x_{i}=y_{i}$.

• •

  Wilcoxon Rank Sum Test (or Mann-Whitney U Test) [104, 44] is a non parametric test which looks at the ranks of two paired samples $x_{i}$ and $y_{i}$; the null hypothesis is that the two samples have the same median.

  The Wilcoxon rank sum test requires samples to be on an ordinal scale, since it needs to order them for determining their rank.

• •

  Wilcoxon Signed Rank Test [104, 44] is a non parametric test which looks at the signs and ranks of the differences among two paired samples $x_{i}$ and $y_{i}$; the null hypothesis is that the median of the differences is zero.

  The Wilcoxon signed rank test requires samples to be on an interval scale, since it regards the ranks of the differences, for which intervals must be equi-spaced. Note that the Wilcoxon signed rank test discards the tied samples, i.e. when $x_{i}=y_{i}$.

• •

  Student’s t Test [86] is a parametric test for the null hypothesis that two paired samples $x_{i}$ and $y_{i}$ come from a normal distribution with same mean and unknown variance.

  The Student’s t test requires samples to be on an interval scale, since it needs to compute means and variances.

• •

  ANalysis Of VAriance (ANOVA) [37, 55] is a parametric test for the null hypothesis that $q$ samples come from a normal distribution with same mean and unknown variance.

  ANOVA requires samples to be on an interval scale, since it needs to compute means and variances.

• •

  Kruskal-Wallis Test [54, 44] is a nonparametric version of the one-way ANOVA for the null hypothesis that $q$ samples come from a distribution with same median. It is based on the ranks of the different samples and it can be considered as an extension of the Wilcoxon rank sum test to the comparison of multiple systems at the same time.

  The Kruskal-Wallis test requires samples to be on an ordinal scale, since it needs to order them for determining their rank.

• •

  Friedman Test [38, 39, 44] is a nonparametric version of the two-way ANOVA for the null hypothesis that the effects of the $q$ samples are the same. It is based on the ranks of the different samples.

  The Friedman test requires samples to be on an ordinal scale, since it needs to order them for determining their rank.

As in the case of Stevens’ permissible operations, defining which statistical significance tests should be permitted on the basis of the scale properties of the investigated variables raised a lot of discussion and controversy. Anderson [10], along the line of reasoning of Lord, argued that statistical significance tests should be used regardless of scale limitations. Gardner [43] summarizes much of the discussion up to that point, leaning towards not worrying too much about scale assumptions, and suggests that, if and when lack of compliance to measurement scale requirements biases the outcomes of significance tests, transformations can be applied to turn ordinal scales into more interval-like ones such as, for example, averaging the ranks of each score, as proposed by Gaito [41], or using a more complex set of rules, as developed by Abelson and Tukey [1]. Ware and Benson [102] replied to Gardner’s positions by further revising the pro and con arguments and concluding that parametric significance tests should be used only when dealing with interval and ratio scales while, in the case of ordinal scales, nonparametric significance tests should be adopted. Townsend and Ashby [89] further investigated the issue, highlighting some serious pitfalls you may fall in, when ignoring the scale assumptions.

We can summarise the discussion with the conclusions of Marcus-Roberts and Roberts [61, p. 391]:

The appropriateness of a statistical test of a hypothesis is just a matter of whether the population and sampling procedure satisfy the appropriate statistical model, and is not influenced by the properties of the measurement scale used. However, if we want to draw conclusions about a population which say something basic about the population, rather than something which is an accident of the particular scale of measurement used, then we should only test meaningful hypotheses, and meaningfulness is determined by the properties of the measurement scale in connection with the distribution of the population.

and Hand [45, p. 471]

Restrictions on statistical operations arising from scale type are more important in model fitting and hypothesis testing contexts than in model generation or hypothesis generation contexts.

Let us start our discussion by considering a not-exhaustive list of core IR areas where scales may matter.

The most common and basic operation we perform to understand whether a system $A$ is better than a system $B$ is to average their performance over a set of topics and compare these aggregate scores. According to the discussion so far this leads to meaningful statements only if IR evaluation measures are, at least, interval scales.

Topic difficulty [19] is another central theme in IR because of its importance for adapting the behaviour of a system to the topic at hand. Voorhees [98, 99], in the TREC Robust tracks, explored how to evaluate and compare systems designed to deal with difficult topics and proposed to use the geometric mean, instead of the arithmetic one, for Average Precision (AP) [15]. However, the use of a geometric mean further raises the requirements for the evaluation measures, even calling for a ratio scale.

Statistical significance testing has a long story of adoption and investigation in IR, from the early uses of t-test reported by Salton and Lesk [77], to the discussion on the compliance with the distribution assumptions of significance tests by van Rijsbergen [93], to advocating for a more wide-spread adoption of different types of significance tests by Hull [49], Savoy [79], Carterette [21], Sakai [72], to surveys on the current state of adoption of significance tests by Sakai [74]. Again, drawing meaningful inference depends on the appropriate use of parametric or nonparametric tests in accordance with the scale properties of the adopted IR evaluation measures.

Several authors have proposed the use of score transformation and standardisation techniques, such as z-score by Webber et al. [103] and other types of linear (and non-linear) transformations by Sakai [73], Urbano et al. [91], in order to compare performance across collections and to reduce the impact of few topics skewing the performance distribution. However, in order to ensure meaningful conclusions from these transformation, at least an interval scale would be required.

Despite so many aspects of IR evaluation which can be affected by the scale properties of evaluation measures and despite the deep scrutiny that the above techniques have received over the years, there has been much less attention to the implications of the scale assumptions on them.

Robertson [70] was the first to discuss the admissibility of the use of the geometric mean from the Stevens’s perspective in the context of the TREC Robust track. In particular, Robertson focused on the fact that Mean Average Precision (MAP) and Geometric Mean Average Precision (GMAP) may lead to different conclusions – e.g. blind feedback is beneficial according to MAP but detrimental according to GMAP – and which of them may hold more (intrinsic) validity. In this respect, Robertson [70, p. 80] observed that

If the interval assumption is not valid for the original measure nor for any specific transformation of it, then any monotonic transformation of the measure is just as good a measure as the untransformed version. If we believe that the interval assumption is good for the original measure, that would give the arithmetic mean some validity over and above the means of transformed versions. If, however, we believe that the interval assumption might be good for one of the transformed versions, we should perhaps favour the transformed version over the original. But if there is no particular reason to believe the interval assumption for any version, then all versions are equally valid. If they differ, it is because they measure different things.

Since both AP and the log-transformation of AP (implied by the geometric mean) are not interval scales, Robertson concluded that no preference could be granted to MAP or GMAP in terms of (intrinsic) validity of their findings. In this way Robertson takes a neutral stance with respect to the debate on whether certain operations should be permitted or not on the basis of the scale properties.

Note that Robertson somehow implicitly indicates transformations as a possible means to turn a not-interval scale into an interval one, as also supported by Gaito [41], Abelson and Tukey [1].

As a final remark, even if Robertson did not mention it explicitly, his reasoning seems to be loosely related the concept of meaningfulness when he says [p. 80]

Good robustness would be indicated if the conclusions looked the same whatever transformation we used; if we found it easy to find transformations which would substantially change the conclusions, then we might infer that our conclusions are sensitive to the interval assumption, and that the different transformations measure different things in ways that may be important to us

still keeping a neutral stance about what should or should not be done.

Fuhr [40] took a firm position and argued that Mean Reciprocal Rank (MRR) [84] should not be computed because: 1. in general, RR is just an ordinal scale and, according to Stevens means cannot be computed for a ordinal scales; 2. in particular, RR has some counter-intuitive behaviour. On the other hand, Sakai [75] has recently disagreed with Fuhr: 1. in general, on the fact that means should not be computed for an ordinal scale, using arguments similar to those discussed in Section 2.3; 2. in particular, on the use of RR which Sakai finds quite useful from a practical point of view.

Whatever stance you wish to take about whether (or not) operations should be constrained by scale properties, from the discussion so far, it clearly emerges that IR needs further and systematic investigation about the implications and impact of derogating from compliance with scale properties. Moreover, most of the above discussion is just about averaging values and does not tackles the implications for statistical significance testing. Finally, and more importantly, we completely lack a thorough discussion on and any adoption of the notion of meaningfulness in IR and this is quite striking for a discipline so strongly rooted in experimentation and so much based on inference.

Ferrante et al. [31, 32, 33] leveraged the representational theory of measurement for developing a formal theory of IR evaluation measures which allows us to determine the scale properties of an evaluation measure. In particular, they defined an ERS for system runs and used two basic operations – swap, i.e. swapping a relevant with a not-relevant document in a ranking, and replacement, i.e. substituting a relevant document with a not-relevant one – to study how runs are ordered. In this way, they demonstrated that there exists a partial order of runs where, when runs are comparable, all the measures agree on the same way of ordering them; however, when runs are not comparable, measures may disagree on how to order them. By using properties of the partial orders and theorems from the representational theory of measurement, they were able to define an interval scale measure $\phi$ and to check whether there is any linear transformation between such measure $\phi$ and IR evaluation measures, in order to determine if the latter are interval scales too.

In short, Ferrante et al. found that, for a single topic:

• •

  set-based evaluation measures:

  • –

    binary relevance: precision, recall, F-measure are interval scales;

  • –

    multi-graded relevance: Generalized Precision (gP) and Generalized Recall (gR) are interval scales only if the relevance degrees are on a ratio scale;

• •

  rank-based evaluation measures:

  • –

    binary relevance: Rank-Biased Precision (RBP) [65] is an interval scale only for $p=1/2$; Average Precision (AP) is not an interval scale;

  • –

    multi-graded relevance: Graded Rank-Biased Precision (gRBP) is an interval scale only for $p=G/(G+1)$, where $G$ is the normalized smallest gap between the gain of two consecutive relevance degrees, and the relevance degrees themselves are on a ratio scale; Discounted Cumulative Gain (DCG) [50] and Expected Reciprocal Rank (ERR) [23] are not interval scales.

Ferrante et al. [33] also studied what they called the induced total order, i.e. pretending that runs in the ERS are ordered by the actual values of a measure. Also in this case which is the most “favourable” to each measure, Ferrante et al. have shown that AP, RBP with $p\neq 1/2$ (and its multi-graded version), DCG, and ERR are not interval scales, because their values are not equi-spaced.

Figure 1 shows the Hasse diagram [26] which represents the partial order among all the runs of length $N=4$. In the figure, vertices are runs while edges represent the direct predecessor relation that is, if $r\prec s$, i.e. $r$ and $s$ are comparable, then $r$ is below $s$ in the diagram. Note that if $r$ and $s$ lie on the same horizontal level of the diagram, then they are incomparable; furthermore, elements on different levels may be incomparable as well. In the example $(1,1,0,1)\preceq(1,1,1,0)$, $(1,1,0,0)\preceq(1,1,1,0)$, and $(1,0,1,1)\preceq(1,1,1,0)$ are all comparable; therefore, all IR measures agree on these runs and order them in the same way. On the other hand, $(1,1,0,0)$ and $(1,0,1,1)$ are not comparable, as well as $(1,1,0,0)$ and $(0,1,1,1)$, and IR measures disagree on how to order them; as a consequence, measures will order these runs differently, producing different Rankings of Systems (RoS).

The difference in the RoS produced by evaluation measures is what is studied when performing a correlation analysis among measures, e.g. by using Kendall’s $\tau$ [52]; practical wisdom says that measures should be neither too much correlated – otherwise it practically makes no difference using one or the other – nor too few correlated – otherwise it may be an indicator of some “pathological” behaviour of a measure. Indeed, each evaluation measure embodies a different user model [20], i.e. a different way in which the user interacts with the ranked result list and derives gain from the retrieved documents, and the differences between the RoS produced by different evaluation measures, and as a consequence their Kendall’s $\tau$, may be considered as the tangible manifestation of such different user models. Note that the work by Ferrante et al. provides a formal explanation of what originates differences in Kendall’s $\tau$: for all the runs which are comparable in the Hasse diagram, Kendall’s $\tau$ between different measures is $1$, since all of them order these runs in the same way; for runs which are not comparable in the Hasse diagram, Kendall’s $\tau$ between different measures is less than $1$, since all of them order these runs differently; therefore, these not comparable runs are where user models differentiate themselves and can take a different stance.

However, these differences in the RoS are not causing IR evaluation measures to not be interval scales; they would just mean that IR evaluation measures are different scales. The real problem with IR evaluation measures is that their scores are not equi-spaced and thus they cannot be interval scales, as explained in Section 2.2. This issue is depicted in Figure 2 which shows how different measures – namely, Precision (and Recall⁶⁶6Note that in this specific case, since the length of the run $N=4$ and the recall base $RB=4$ are the same, Precision and Recall yield to the same scores.), AP, RR, RBP with $p\in\{0.3,0.5,0.8\}$, and DCG with log base $2$ – order and space the runs shown in the Hasse diagram of Figure 1.

We can observe that only Precision (Recall) and RBP with $p=0.5$ produce equi-spaced values, while all the other measures violate this assumption, required to obtain an interval scale; in other terms, Figure 2 visually represents the issue found by Ferrante et al. [33] even when using the induced total order. We can also note that all the measures agree only on the common comparable runs – i.e. $(0,0,0,0)\preceq(0,0,0,1)\preceq(0,0,1,0)$ and $(1,1,0,1)\preceq(1,1,1,0)\preceq(1,1,1,1)$ – but, as soon as incomparable runs come into play, they start to disagree on how to order them. Finally, looking at Figure 2 we can notice how IR measures behave differently in violating the equi-spacing assumption. RBP with $p\in\{0.3,0.8\}$ and DCG follows a somehow regular pattern, where scores are not equi-spaced but they are in some way evenly clustered and they are symmetric if you fold the figure along its middle horizontal axis; on the other hand, AP and RR follow a much more irregular and not symmetric pattern.

We can also note how these measures spread values in their range differently. Precision (and Recall) and DCG spread their values all over the possible range while this is not always the case with RBP. Indeed, RBP assumes runs of infinite length and normalizes by the $\frac{1}{1-p}$ factor. However, we deal with runs of limited length and the $\frac{1}{1-p}$ factor is an overestimation, the bigger the overestimation the bigger is the value of $p$ and the smaller is the length of the run – this is more clearly visible in the case of RBP with $p=0.8$ in Figure 2(f). Finally, AP, RBP with $p=0.3$, and RR, i.e. those measures farther from being interval scales, leave large portions of their possible range completely unused. In particular, AP leaves one quarter of its range unused, in the top part roughly corresponding to the first quartile of the possible values; RR leaves one half of its range unused, in the top part roughly corresponding to the first and second quartiles of the possible values; and, finally, RBP with $p=0.3$ leaves half of its range empty, in the middle part roughly corresponding to the second and third quartile of the possible values.

Why does it matter how much equi-spaced the values are and how they are spread over their range? Consider a random variable $X$ that takes values in the set $\{0,1,2,4,13\}$. Even if all these five values can be obtained with equal probability, i.e. the random variable is uniform, the mean and the median of the variable differ, being the mean equal to $4$ and the median to $2$. This shows how the lack of equi-spacing causes some sort of “imbalance” even in the case of a uniform variable, which may be an undesirable situation from the measurement point of view, at least if not explicitly considered and accounted for. Furthermore, when we compute $\mathbb{P}[X\in(x-\varepsilon,x+\varepsilon)]$, i.e. the probability that the value of $X$ is equal to $x$ with an error of at most $2\varepsilon$, this function is not constant all over the range but it is assumer greater for values around $\{0,1,2\}$ than for those around $\{4,13\}$. As a consequence, a similar accuracy in approximating the value of $X$ produces a different precision in the measurement depending on the value $x$ that we are considering. Note that in the present toy model it happens for $\varepsilon\geq 1$, but a suitable modification of the present model can produce the same behaviour for any $\varepsilon>0$ set in advance.

As a further example, let us consider a measure with a limited range of equi-spaced values. If we draw a set of random values taken from this range and consider its arithmetic mean, by the law of large numbers, we have that this mean converges to the middle point of the range interval. This property is independent from the distance among the subsequent values, i.e. the unit of measurement chosen. So we can use such a procedure – the convergence of the mean towards the middle of the range – in order to “calibrate” the measuring instrument, independently from the specific unit of measurement chosen. This is no more possible if we have values which are not equi-spaced.

The fact that IR evaluation measures, apart from Precision, Recall, and RBP with $p=0.5$, are not interval scales leads to the general issues with computing means, statistical tests, and meaningfulness discussed in Sections from 2.2 to 7.3 and shown in Examples 3 and 7. In addition, Examples 9 and 10 above show how the lack of equi-spacing may also lead to statements like “system A is better than B” (or viceversa) which are not always intuitive all over the scale.

The fact that Precision and Recall are interval scales makes addition and subtraction permissible operations and, as a consequence, computing arithmetic means permissible too. Therefore, it is safe to average performance of IR systems across topics when we use Precision and Recall. But is that really true?

As said, Ferrante et al. [33] have found an interval scale $\phi$, called Set-Based Total Order (SBTO), and have shown that both Precision and Recall are an affine transformation of this interval scale and thus also an affine transformation of each other. Ferrante et al. [34] have raised this question: if Precision and Recall are transformations of the same interval scale, they are ordinal scales too and they should rank systems in the same way. Therefore, if they produce the same RoS, Kendall’s $\tau$ correlation between them should be 1. So, why their Kendall’s $\tau$ correlation is $0.8588$, using the TREC 8 Ad-hoc data?

Let us consider how correlation analysis between evaluation measures works. Given two rankings $X$ and $Y$, their Kendall’s $\tau$ correlation is given by $\tau\big{(}X,Y\big{)}=\frac{P-Q}{\sqrt{\big{(}P+Q+T\big{)}\big{(}P+Q+U\big{)}}}$, where $P$ is the total number of concordant pairs (pairs that are ranked in the same order in both vectors), $Q$ the total number of discordant pairs (pairs that are ranked in opposite order in the two vectors), $T$ and $U$ are the number of ties, respectively, in the first and in the second ranking. $\tau\in[-1,1]$ where $\tau=1$ indicates two perfectly concordant rankings, i.e. in the same order, $\tau=-1$ indicates two fully discordant rankings, i.e. in opposite order, and $\tau=0$ means that 50% of the pairs are concordant and 50% discordant.

The typical way of performing correlation analysis is as follows: let $\phi_{1}$ and $\phi_{2}$ be two evaluation measures; in our case, $\phi_{1}$ is Precision and $\phi_{2}$ is Recall. Let $\Phi_{1}$ and $\Phi_{2}$ be two $T\times S$ matrices where each cell contains the performance on topic $i$ of system $j$ according to measures $\phi_{1}$ and $\phi_{2}$, respectively. Therefore, $\Phi_{1}$ and $\Phi_{2}$ represent the performance of $S$ different systems (columns) over $T$ topics (rows). Let $\mkern 1.5mu\overline{\mkern-1.5mu\Phi\mkern-1.5mu}\mkern 1.5mu_{1}$ and $\mkern 1.5mu\overline{\mkern-1.5mu\Phi\mkern-1.5mu}\mkern 1.5mu_{2}$ be the column-wise averages of the two matrices, i.e. the average of the performance of each system across the topics. If you sort systems by their score in $\mkern 1.5mu\overline{\mkern-1.5mu\Phi\mkern-1.5mu}\mkern 1.5mu_{1}$ and $\mkern 1.5mu\overline{\mkern-1.5mu\Phi\mkern-1.5mu}\mkern 1.5mu_{2}$, you obtain two RoS corresponding to $\mathrm{\phi}_{1}$ and $\mathrm{\phi}_{2}$, respectively, and you can compute Kendall’s $\tau$ correlation between these two RoS. This is the traditional way for computing the correlation between two evaluation measures and Ferrante et al. call it overall correlation, since it first computes the average performance across the topics and then it computes the correlation between evaluation measures. This approach leads to a Kendall’s $\tau$ correlation of $0.8588$ between Precision and Recall.

Ferrante et al. proposed a different way of computing the correlation, called topic-by-topic correlation, where, for each topic $i$, they consider the RoS on that topic corresponding to $\phi_{1}$ and the one corresponding to $\phi_{2}$, i.e. they consider the $i$-th rows of $\Phi_{1}$ and $\Phi_{2}$, respectively; they then compute Kendall’s $\tau$ correlation among the two RoS on that topic. Therefore, they end-up with a set of $T$ correlation values, one for each topic. Using, this way of computing correlation, Ferrante et al. found that Kendall’s $\tau$ correlation between Precision and Recall is always $1$ for all the topics and this was the result expected for two interval scales which order systems in the same way.

Therefore, if you consider each topic alone, Precision and Recall are just a transformation of the same interval scale, as Celsius and Fahrenheit are, and their Kendall’s $\tau$ correlation is $1$. However, if you first average across topics, which should be a permitted operation for interval scales, and then you compute Kendall’s $\tau$ correlation, it stops to be $1$. This was somehow surprising and unexpected. Indeed, as an example from another domain, if you take a matrix of scores in Celsius degrees and another one with the corresponding Fahrenheit degrees, their Kendall’s $\tau$ correlation is always $1$, either if you compute it row-by-row (i.e. our topic-by-topic correlation) or if you first average across rows and then compute it (i.e. our overall correlation).

Ferrante et al. [34, p. 305] explained this behaviour as due to the recall base:

Recall heavily depends on the recall base which changes for each topic and it is used to normalize the score for each topic; therefore, in a sense, recall on each topic changes the way it orders systems

We further investigate this issue in Section 3.4 below, where we provide details and demonstrations, but here we summarise the sense of our findings. The difference between overall and topic-by-topic correlation is basically due to the fact that we are using different interval scales for each topic. These scales are indeed transformations of one in the other for each topic and this is why topic-by-topic correlation is $1$; however, since we are changing scale from one topic to another, when average across topics we are mixing different scales and this is why the overall correlation is different from $1$.

The fact that we may be changing scale from topic to topic has very severe consequences. All the debate originated by Stevens’s permissible operations and the possibility of averaging only from interval scales onwards has always been based on the obvious assumption that the averaged values were all drawn from the same scale; no one has ever doubted that it is not possible to average values coming from different scales because this would be like mixing apples with oranges. So, what is the meaningfulness of typical statements like “System A is (on average) better than system B” when we are not only violating the interval scale assumptions but, even more seriously, we are mixing different scales? What about the meaningfulness of typical statements like “System A is significantly better than system B”? The debate between using parametric or nonparametric tests concerns how much you wish to comply with the interval scale assumptions but, undoubtedly, all the significance tests, when aggregating across values, expect them to be drawn from the same scale.

If we wish to make an analogy, it is like the difference between using mass and weight, being Precision similar to mass and Recall to weight. It would be somehow safe to average the mass of bodies coming from different planets but it would not to average their weight, due to the different gravity on the different planets. The recall base is what changes the gravity from planet/topic to planet/topic in the case of Recall.

However, even Precision is not completely “safe” because, when the length of the run changes, its scale changes as well. As a consequence we may end up using different scales from one run to another and this can happen not only across topics, as in the case of Recall, but also within topics, if we have two or more runs retrieving a different number of documents for that topic. This statements affects the evaluation of classical Boolean retrieval, where Precision and Recall are computed for the set of retrieved documents for each query, followed by averaging over all queries. So we have to conclude that this procedure is seriously flawed. Luckily, in most of today’s evaluations, the length of the run has a much smaller effect because, in typical TREC settings, almost all the runs retrieve 1,000 documents for each topic and just few of them retrieve less documents; this effect would also (practically) disappear when you consider Precision at lower cut-offs, like P@10, when it is almost guaranteed that all the runs retrieve 10 documents.

Summing up, independently from an evaluation measures being an interval scale or not, the recall base (greatly) and the length of the run (less) cause the scale to change from topic to topic and/or from run to run. This makes averaging across topics, as well as other forms of aggregation used in significance tests, problematic at best. We show how and why this happens in the case of Precision (Section 3.4.1) and Recall (Section 3.4.2), which are the simplest measures you can think of, since they change either the length of the run or the recall base alone. We also consider the more complex case of the F-measure (Section 3.4.3), which changes both the run length and the recall base at the same time. Therefore, we hypothesise that these issues may be even more severe in the case of more complex evaluation measures, like AP and others, which are not even interval scales and mix recall base and run length with rank position and various forms of utility accumulation and stopping behaviours.

Finally, also the way in which we interpret the results of correlation analysis may be impacted. Indeed, we typically attribute differences in correlation values to the different user models embedded by evaluation measures. The rule-of-thumb by Voorhees [96, 97] is that an overall correlation above $0.9$ means that two evaluation measures are practically equivalent, an overall correlation between $0.8$ and $0.9$ means that two measures are similar, while dropping below $0.8$ indicates that measures are departing more and more. Therefore a correlation of $0.8588$ would suggest that Precision and Recall share some commonalities but they differ enough due to their user models, still not being pathologically different. However, we (now) know that they are just the transformation of the same interval scale and that this correlation value is just an artifact of mixing different scales across topics rather than an intrinsic difference in the user models of Precision and Recall.

As discussed above, Ferrante et al. [33] have demonstrated that Precision, Recall, and F-measure are interval scales when you fix the length of the run $N$ and the recall base $RB$, i.e. they are an homomorphism with respect to the same ordering of runs in the ERS. However, if we mix together runs with different bounded lengths and/or different bounded recall bases, Precision, Recall and F-measure are no more interval scales, they are no more an affine transformation of each other and they even order the runs in different ways. Clearly, this is a severe issue when you need to average (or compute any other aggregate) across different topics or runs with different lengths.

Let us consider the universe set $S[N,K]$ which contains all the runs of any possible length $n$, less than or equal to $N$, and with respect to all the possible recall bases $RB$, less than or equal to $K$. To avoid trivial cases, we consider always $N$ and $K$ greater than or equal to $1$. A run in $S[N,K]$ is represented by a triple $[r,n,RB]$, where $r$ indicates the number of relevant documents retrieved by the run, $n$ is the length of the run, and $RB$ is the recall base, i.e. the total number of relevant documents for a topic. Note that, for each run in $S[N,K]$ it holds $n\leq N$ and $RB\leq K$ by construction, but we also have $r\leq(n\wedge RB)$, where $x\wedge y=\min\{x,y\}$, i.e. there is a (implicit) dependence on the recall base when it comes to the number of relevant retrieved documents.

We define $S_{n,RB}$ as the set which contains all the runs with the same length $n$ and with respect to the same recall base $RB$. Therefore, we can express the universe set $S[N,K]$ as the union of such sets, namely

$S_{n,RB}$ models the typical case of runs all with the same length for a given topic (or for a set of topics which have the same recall base). This is exactly the case for which Ferrante et al. [33] have demonstrated that Precision, Recall and F-Measure are interval scales and an affine transformation of each other. However, this holds for each $S_{n,RB}$ separately while the issue we discuss in this section is what happens when you mix different $S_{n,RB}$, i.e. when you go towards $S[N,K]$.