Dimensional Analysis in Statistical Modelling

One tenet of working with measured quantities is that units in an expression or equation must “match up”; relationships among measurable quantities require dimensional homogeneity. To check the validity of comparative statements about two quantities, $X_{1}$ and $X_{2}$, such as $X_{1}=X_{2}$, $X_{1}<X_{2}$ or $X_{1}>X_{2}$, $X_{1}$ and $X_{2}$ must be the same dimension, such as time. To add $X_{1}$ to $X_{2}$, $X_{1}$ and $X_{2}$ must not only be the same dimension but must also be on the same scale and expressed in the same units of measurement.

To discuss this explicitly, we use a standard notation (Joint Commitee on Guides in Metrology, 2012) and write a measured quantity $X$ as $X=\{X\}[X]$, where $\{X\}$ is the numerical part of $X$. $[X]$ may be read as the dimension of $X$ e.g. $[X]=L$ for length, for instance, or the units of $X$ on the chosen scale of measurement, e.g. $[X]=cm$. The latter by its nature means that the dimension is $L$. If $[X]=[1]$, then we say that $X$ is unitless or dimensionless. We define 1 or any number to be unitless, i.e., $1={1}[1]$, unless stated explicitly.

To develop an algebra for measured quantities, for a function $f$ we must say what we mean by $\{f(X)\}$ (usually easy) and $[f(X)]$ (sometimes challenging). The path is clear for $f$ a simple function. For example, consider $f(X)=X^{2}$. Clearly we must have $X^{2}=\{X\}^{2}[X]^{2}$, yielding, say, (3 inches)${}^{2}=9$ inches². But what if $f$ is a more complex function? This issue will be discussed in general in Subsection 4.2 and in detail for $f(x)=\ln(x)$ in Subsection 4.4.

For simple functions, the manipulation of both numbers and units is governed by an algebra of rules referred to as quantity calculus (Taylor, 2018). This set of rules states that $x$ and $y$

• •

  can be added, subtracted or compared if and only if $[x]=[y]$;

• •

  can always be multiplied to get $xy=\{xy\}[xy]$ where $\{xy\}=\{x\}\{y\}$ and $[xy]=[x][y]$;

• •

  can always be divided when $\{x\}\neq 0$ to get $y/x=\{y/x\}[y/x]$ where $\{y/x\}=\{y\}/\{x\}$ and $[y/x]=[y]/[x]$;

and that

• •

  $x$ can be raised to a power that is a rational fraction $\gamma$, provided that the result is not an imaginary number, to get $x^{\gamma}=\{x\}^{\gamma}[x]^{\gamma}$.

Thus it makes sense to square-root transform ozone $O_{3}=\{O_{3}\}$ parts per million (ppm) as $\{O_{3}\}^{1/2}$ ppm^1/2 since ozone is measured on a ratio-scale with a true origin of $0$ and hence must be non–negative (Dou, Le and Zidek, 2007). These rules can be applied iteratively a finite number of times to get expressions that are combinations of products of quantities raised to powers, along with sums and rational functions of such expressions.

This subsection concludes with an example that demonstrates the use of DH  and quantity calculus.

We see that there may be several ways to nondimensionalize a model. Another method, widely used in the physical sciences, involves always normalizing by the standard units specified by the Système International d’Unités (SIU), units such as meters or kilograms. So when the dimensions of a non–negative quantity $X$ like absolute temperature have an associated SIU of $Q_{0}=\{1\}[Q_{0}]$, $X$ can be converted to a unitless quantity by first expressing $X$ in SIUs and then by using quantity calculus to rescale it as $X/Q_{0}$. The next example provides an important illustration of the application of the standardized unit approach.

The choice of scale restricts the choice of units of measurement, and these units dictate the type of model that may be used. However, comparing the size of two quantities on a ratio-scale must be made using their ratio, not their difference, whereas the opposite is true on an interval-scale where differences must be used.

Thus we need to study scales in the context of model building and hence in the context of quantity calculus. In his celebrated paper, Stevens (1946) starts by proposing four major scales for measurements or observations: categorical, ordinal, interval and ratio. This taxonomy is based on the notion of permissible transformations as is the work of our Section 5. However, our work is aimed at modelling while Stevens’s work is aimed at statistical analysis. Stevens defines permissible transformations as follows. He allows permutations as the transformations of data on all four scales, allows strictly increasing transformations of data on the ordinal, ratio and interval-scales, allows scalar transformations ($f(x)=ax$) of data on the ratio and interval-scales and allows linear transformations ($f(x)=ax+b$) of data on the interval-scale.

Stevens created his taxonomy as a basis for classifying the family of all statistical procedures for their applicability in any given situation (Stevens, 1951). And for instance, Luce (1959) points out that, for measurements made on a ratio-scale, the geometric mean would be appropriate for estimating the central tendency of a population distribution (Velleman and Wilkinson, 1993). In contrast, when measurements are made on an interval-scale the arithmetic mean would be appropriate. The work of Stevens seems to be well-accepted in the social sciences, with Ward (2017) calling his work monumental. But Steven’s work is not widely recognized in statistics. Velleman and Wilkinson (1993) reviews the work of Stevens with an eye on potential applications in the then emerging area in statistics of artificial intelligence (AI), hoping to automate data analysis. They claim that “Unfortunately, the use of Stevens’s categories in selecting or recommending statistical analysis methods is inappropriate and can often be wrong.” They describe alternative scale taxonomies for statistics that have been proposed, notably by Mosteller and Tukey (1977). A common concern centers on the inadequacies of an automaton to select the statistical method for an AI application. Even the choice of scale itself will depend on the nature of the inquiry and thus is something to be determined by humans. For example, length might be observed on the relatively uninformative ordinal scale $\{short,\leavevmode\nobreak\ medium\leavevmode\nobreak\ ,long\}$, were it sufficient for the intended goal of a scientific inquiry, rather than on the seemingly more natural ratio-scale $(0,\infty)$.

The interval-scale of real numbers allows for the taking of sums, differences, products, ratios, and integer powers of values observed on that scale. Rational powers of nonnegative values are also allowed although irrational powers lead into the domain of transcendental functions and difficulties of interpretation. The same operations are allowed for a ratio-scale of real numbers provided that the differences are non-negative. So superficially, these two scales seem nearly the same.

But there is a substantial qualitative difference between ratio- and interval-scales, so ignoring the importance of scale when building models can result in challenges in interpretation. The issue has to do with the meaning of the $0$ on a ratio-scale. The next hypothetical example illustrates the point.

This subsection concerns a perhaps unconscious switch in a statistical analysis from a ratio scale, which lies on $[0,\infty)$, to an interval-scale, which lies on $(-\infty,\infty)$.

A function $u$, which describes the relationship among quantities $X_{1},\dots,X_{p}$ as

may be a sequence of transformations or operations involving the $X_{i}$’s, possibly combined with parameters. We know how to calculate the resulting units of measurement when $u$ consists of a finite sequence of permissible algebraic operations. The function consisting of the concatenation of such a sequence may formally be defined as a root of a polynomial equation that must satisfy the requirement of dimensional homogeneity (other desirable properties of $u$ along with methods for determining an allowable $u$ are discussed in Section 5). Such a function is called algebraic.

But $u$ may also involve non-algebraic operations leading to non-algebraic functions called transcendental (because they “transcend” an algebraic construction). Examples in the univariate case ($p=1)$ are $\sin(X)$ and $\cosh(X)$ and, for a given nonnegative constant $\alpha$, $\alpha^{X}$ and $\log_{\alpha}{(X)}$. The formal definition of a non-algebraic function does not explicitly say whether or not such a function can be applied to quantities with units of measurement. Bridgman (1931) sidesteps this issue by arguing that it is mute since valid representations of natural phenomena can always be nondimensionalized (see Subsection 5.1). But the current Wikipedia entry on the subject states “transcendental functions are notable because they make sense only when their argument is dimensionless” (Wikipedia, 2020). The next subsection explores the much used class of Box-Cox family (Box and Cox, 1964) that includes transcendental functions.

Frequently in statistical modelling, a transformation is used to extend the domain of applicability of a procedure that assumes normally distributed measurements (De Oliveira, Kedem and Short, 1997). That transformation may also be seen as a formal part of statistical model building that facilitates maximum likelihood estimation of a single parameter (Draper and Cox, 1969). The Box-Cox (BC) transformations constitute an important class of such transformation and therefore a standard tool in the statistical scientist’s toolbox.

In its simplest form, a member of this family of transformations has the form of a function $bc(X)=X^{\lambda}$ for a real-valued parameter $\lambda\in(-\infty,\infty)$. Here $X$ would need to lie in $[0,\infty)$, a ratio-scale, to avoid potential imaginary numbers. However, in practice interval-scales are sometimes allowed, a positive constant being added to avoid negative realizations of $X$. This ad hoc procedure thus validates the use of a Gaussian distribution to approximate the sampling distribution for $X$.

Since $X$ is measured on a ratio–scale, for any two points on that scale, $bc(x_{2}/x_{1})\leavevmode\nobreak\ =\leavevmode\nobreak\ bc(x_{2})/bc(x_{1})$ while the scale is equivariant under multiplicative transformations, i.e., $bc(ax)\leavevmode\nobreak\ =\leavevmode\nobreak\ a^{\lambda}\leavevmode\nobreak\ bc(x)$ for any point on that scale. Finally $bc(X)>0$ so that the result of the transformation also lies on a ratio–scale, even when its intended goal is an approximately Gaussian distribution for the (transformed) response.

Box and Cox (1964) actually state their transformation as

that moves the origin of the ratio–scale from $0$ to $-1$ . It is readily seen that unless $\lambda$ is a rational number, $bc_{\lambda},\leavevmode\nobreak\ \lambda\neq 0$ will be transcendental. That fact would be inconsequential in practice in as much as a modeller would only ever use a rational number for $\lambda$ . Or at least that would be the case except that the BC class has been extended to include $\lambda=0$ by admitting for membership $\lim_{\lambda\rightarrow 0}bc_{\lambda}(X)=\ln(X),\leavevmode\nobreak\ X>0$ .

On closer inspection, we see that for validity, in Equation (4.2), the $1$ needs to be replaced by $(1[X])^{\lambda}$ to include units of measurement. Then for $\lambda\neq 0$ the transformation becomes

As $\lambda\rightarrow 0$, the only tenable limit would seem to be

not $\ln{(X})$. In other words, in taking logarithms in the above example, the authors may have unconsciously nondimensionalized the measurements. Taking antilogarithms would then return $\{X\}$, not $X$.

Equation 4.3 thus tells us the Box–Cox transformation may have the unintended consequence of transforming not only the numerical value of the measurements but also its units of measurement, which become $[X]^{\lambda}$. This makes a statistical model difficult to interpret. For example imagine the challenge of a model with a random response in units of $mm^{1/100}$. And since the transformation is nonlinear, returning to the original scales of the data would be difficult. For example, unbiased estimates would become biased and their standard errors could be hard to approximate.

The importance of the logarithmic transformation in statistical and scientific modelling, and issues that have arisen about its lack of units, leads next to a special subsection devoted to it.

The section begins with Buckingham’s simple motivating example.

This example of nondimensionalizing by finding one expression, $\pi$, as in Equation (5.1) can be extended to cases where we must nondimensionalize by finding several $\pi$ quantities. This extension is formalized in Buckingham’s Pi-theorem. Here is a formal statement (in slightly simplified form) as stated by Buckingham (1914) and discussed in a modern style in Bluman and Cole (1974).

The theorem is proven constructively, so we can find $\pi_{1},\ldots,\pi_{q}$ and $U$. We first determine the $m$ fundamental dimensions used in $X_{1},\ldots,X_{p}$. We then use the quantities $X_{1},\ldots,X_{p}$ to construct two sets of variables: a set of $m$ primary variables also called repeating variables and a set of $q$ secondary variables, which are nondimensional. For example, if $X_{1}$ is the length of a box and $X_{2}$ is the height and $X_{3}$ is the width, then there is $m=1$ fundamental dimension, the generic length denoted $L$. We can choose $X_{1}$ as the primary variable and use $X_{1}$ to define two new variables $\pi_{1}=X_{2}/X_{1}$ and $\pi_{2}=X_{3}/X_{1}$. These new variables, called secondary variables, are dimensionless. Buckingham’s theorem states the algebraic equation relating $X_{1},X_{2}$ and $X_{3}$ can be re-written as an equation involving only $\pi_{1}$ and $\pi_{2}$. Note that we could have also chosen either $X_{2}$ or $X_{3}$ as the primary variable.

A famous application of Buckingham’s theorem concerns the discovery of the Reynold’s number in fluid dynamics, which is discussed in Gibbings (2011). For brevity we include that example in Appendix B.

A link between Buckingham’s approach and statistical modelling was recognized in the paper of Albrecht et al. (2013) and commented on in Lin and Shen (2013). But its link with the statistical invariance principal seems to have been first identified in the thesis of Shen (2015). This connection provides a valuable approach for the statistical modelling of scientific phenomena. Shen builds a regression model starting with Buckingham’s approach and thereby a nondimensionalized relationship amongst the variables of interest. We propose a different approach in Section 6. We present Shen’s illustrative example next.

We now describe an alternative to the approach of Buckingham (1914) due to Bridgman (1931). At around the same time that Edgar Buckingham was working on his Pi-theorem, Percy William Bridgman was giving lectures at Harvard on the topic of nondimensionalization that were incorporated in a book whose first edition was published by Yale University Press in 1922. The second edition came out in 1931 (Gibbings, 2011). Bridgman thanks Buckingham for his papers but notes their approaches differ. And so they do. For a start, Bridgman asserts his disagreement with the position that seems to underlie Buckingham’s work that “a dimensional formula has some esoteric significance connected with the ‘ultimate nature’ of things.” Thus those that espouse that point of view must “find the true dimensions and when they are found, it is expected that something new will be suggested about the physical properties of the system.” Instead, Bridgman takes measurement itself as the starting point in modelling and even the collection of data: “Having obtained a sufficient area of numbers by which the different quantities are measured, we search for relations between these numbers, and if we are skillful and fortunate, we find relations which can be expressed in mathematical form.” He then seeks to characterize a measured quantity as either primary, that is, the product of direct measurement, or secondary, that is, computed from the measurements of primary quantities, as, for instance, velocity is computed from the primary quantities of length and time. Finally he sees the basic scientific issue as that of characterizing one quantity in terms of the others as in our explication of Buckingham’s work above in terms of the function $u$.

Bridgman considers the functional relationship between secondary and primary measurements under what statistical scientists might call “equivariance” under multiplicative changes of scale in the primary units. He proves that the functional relationship must be based on monomials with possible fractional exponents, not unlike the form of the $\pi$-functions above. Thus Bridgman is able to re-derive Buckingham’s $\pi$ formula, albeit with the added assumption that $u$ is differentiable with respect to its arguments.

Nondimensionalization seems more difficult outside of the domain of the physical sciences. For example, the dimensions of quantities such as utility cannot be characterized by a ratio-scale. And the choice of the primary dimensions is not generally so clear, although Baiocchi (2012) does provide an example in macroeconomics where time $[T]$, money $[\$]$, goods $[R]$ and utility $[U]$ may together be sufficient to characterize all other quantities.

Bridgman’s results on allowable laws were limited to laws involving quantities measured on ratio-scales. A substantial body of work has been devoted to extending these results to laws involving quantities measured on nonratio-scales, beginning with the seminal paper of Luce (1959). To quote the paper by Aczél, Roberts and Rosenbaum (1986), which contains an extensive review of that work, “Luce shows that the general form of a ‘scientific law’ is greatly restricted by knowledge of the ‘admissible transformations’ of the dependent and independent variables.” It seems puzzling that this principle has been recognized little if at all in statistical science. This may be due to the fact fact that little attention is paid to such things as dimensions and units of measurement.

The substantial body of research that followed Luce’s publication covers a variety of scales, e.g. ordinal, among other things. Curiously that body of work largely ignores the work of Buckingham in favor of Bridgman even though the former preceded the latter. Also ignored is the work on statistical invariance described, which goes back to G. Hunt and C. Stein in 1946 in unpublished but well-known work that led to optimum statistical tests of hypotheses.

To describe this important work by Luce, we re-express Equation (5.2) as

for some function $u^{*}$ and thereby define a class of all possible laws that could relate $X_{p}$ to the predictors $X_{1},\dots,X_{p-1}$, before turning to a purely data-based empirical assessment of the possible $u^{*}$’s. Luce requires that $u^{*}$ satisfy an invariance condition. Specifically, he makes the strong assumption that the scale of each $X_{i},\leavevmode\nobreak\ i=1,\leavevmode\nobreak\ \dots,\leavevmode\nobreak\ p-1$, is susceptible to a transformation $T_{i}\in{\cal F}_{i}$, i.e. $X_{i}\rightarrow T_{i}(X_{i})$ for some sets of possible transformations ${\cal F}_{1},\dots,{\cal F}_{p-1}$. Furthermore he assumes that the $X_{i}$’s are transformed independently of one another; no structural constraints are imposed. Luce assumes the existence of a function $D$ such that

for all possible transformations and choices of $X_{i},\leavevmode\nobreak\ i=1,\leavevmode\nobreak\ \dots,\leavevmode\nobreak\ p-1$. He determines that under these conditions, if each $X_{i},\leavevmode\nobreak\ i=1,\leavevmode\nobreak\ \dots,\leavevmode\nobreak\ p$, lies on a ratio-scale then

where the $\alpha_{i}$’s are nondimensional constants. This is Bridgman’s result, albeit proved by Luce without assuming differentiability of $u^{*}$. If on the other hand some of the $X_{i}$’s, $i=1,\leavevmode\nobreak\ \dots,\leavevmode\nobreak\ p-1$, are on a ratio-scale while others are on an interval-scale and $X_{p}$ is on an interval-scale, then Luce proves $u^{*}$ cannot exist except in the case where $p=2$ and $X_{1}$ is on an interval-scale.

However, as noted by Aczél, Roberts and Rosenbaum (1986), the assumption in Equation (5.5) of the independence of the transformations $T_{i}$ seems unduly strong for many situations, and weakening that assumption expands the number of possibilities for the form of $u^{*}$. Further work culminated in that of Paganoni (1987). While this work was for $X_{i}$’s in a general vector space, for simplicity we present it here in our context, where $X_{i}\in{\rm I\!R}$, $i=1,\ldots,p$. Let ${\cal X}$ and ${\cal{P}}$ be nonempty subsets of ${\rm I\!R}^{p-1}$ and ${\cal R}$ a set of $(p-1)$ by $(p-1)$ real-valued matrices. Suppose that

1. 1.

   $\textbf{x}+\textbf{p}\in{\cal X}$ for all $\textbf{x}\in{\cal X}$ and $\textbf{p}\in{\cal P}$;

2. 2.

   the identity matrix is in ${\cal R}$ and, for all $R\in{\cal R}$ and all ${\textbf{x}}\in{\cal X}$, $R{\textbf{x}}\in{\cal X}$;

3. 3.

   if ${\cal P}\neq\{0\}$, then $\lambda R\in{\cal R}$ for all $R\in{\cal R}$ and all $\lambda>0$.

Suppose also that the function $u^{*}$ in Equation (5.4) satisfies

for all $\textbf{R}\in\mathcal{R}$, $\textbf{x}\in{\cal X}$ and $\textbf{p}\in{\cal P}$ for some positive-valued function $\alpha$ and real-valued function $\beta$. Paganoni then determines the possible forms of $\alpha$ and $\beta$.

This subsection reviews the theory of transformation groups and the statistical invariance principle, a topic that has a rich history (Eaton, 1983). These are needed for extending the Buckingham Pi-theory. That need is recognized by Shen and Lin (2019), although their applications concern ratio-scales and physical models. To introduce these groups, for simplicity, in both this section and the next, we will focus on how groups transform the sample space. Later, in Sections 6.3 and 6.5, we will use the same concepts for the full general theory of statistical invariance and generalized statistical invariance.

Each $\omega\in\Omega$ has a vector of measurable attributes $X=X(\omega)\in{\cal X}$:

A sample of $\omega$’s is to be drawn according to a probability distribution $P$ on $\Omega$, with $P$ inducing a probability distribution on $X$. Buckingham’s theory (see Subsection 5.1) aims at relating $\omega$’s attributes through a model like that in Equation (5.4). Our extension of that theory below will be stochastic in nature and assign $X_{p}$ the special role of predictand.

A sample of size $n$ yields a sample of observations $X_{ij}$ represented by ${\bf X}^{p\times n}\in{\cal X}^{n}$. The statistical invariance principle posits that randomized statistical decision rules that determine actions should be invariant under $1:1$ transformations by members $g$ of an algebraic group of transformations $G$. That is, any pair of points ${\bf x},{\bf x}^{\prime}\in{\cal X}^{n}$ are considered equivalent for statistical inference if and only if ${\bf x}=g({\bf x}^{\prime})$ for some $g\in G$. This equivalence is denoted ${\bf x}\sim{\bf x}^{\prime}$. By definition, the equivalence classes formed by $G$ are disjoint and exhaustive so we can index them by a parameter $\gamma\in\Gamma$ and let ${\cal X}^{n}_{\gamma},\leavevmode\nobreak\ \gamma\in\Gamma$, represent an equivalence class. The ${\cal X}^{n}_{\gamma},\leavevmode\nobreak\ \gamma\in\Gamma$, are referred to as orbits, which could be indexed by a set of points, $\{{\bf x}_{\gamma},\leavevmode\nobreak\ \gamma\in\Gamma\}$. If the set of points satisfies some regularity conditions, then it is called a cross-section, denoted ${\cal X}^{n}/G$, and its existence is studied by Wijsman (1967). Assuming a cross-section does exist, we may write

In other words, any point ${\bf x}\in{\cal X}^{n}$ is represented by $(g,{\bf x}_{\gamma})$ for appropriately chosen $g$ and ${\bf x}_{\gamma}$.

The statistical invariance principle states that a statistical decision rule must be invariant, that is, the rule must take the same value for all points in a single orbit $\gamma$. Maximal invariant functions play a special role in statistics. The function $M$ is invariant if its value is constant on each orbit. Further, $M$ is a maximal invariant if it takes different values on each orbit.

The following example shows the statistical invariance principle in action.