Detection and Amelioration of Social Engineering Vulnerability in Contingency Table Data using an Orthogonalised Log-linear Analysis

Social engineering attacks on cyber infrastructure constitute a large and growing economic problem. As the comprehensive literature review in [1] reveals, the term ‘social engineering’ has been applied to a wide variety of illegitimate methods of gaining advantage ranging from simple breaches of physical security to sophisticated phishing campaigns [2]. However the central feature of social engineering that emerges is the exploitation of human vulnerability through some form of interaction between attacker and the attack target.

This is the case even where an unauthorised person gains physical entry to secure facilities by following legitimate personnel through a controlled access point (‘tailgating’) or enlisting the aid of an authorised person by pretending to be struggling with a heavy load [4] [3]. Here, the attacker exploits the reluctance of people to confront a potentially awkward situation or the desire to be helpful which illustrates the general observation that humans are the weakest link in any security system [4].

Phishing is also becoming a major threat to cyber security. From the discussion in [2] it is clear that there is a close relationship with social engineering, with some researchers describing phishing as a form of social engineering whereas others see phishing as employing social engineering. Regardless of the details, a unifying thread is the exploitation of human vulnerability for the creation of some form of deception for nefarious purposes.

Methods of defending against social engineering attacks are discussed in [4], [8] and [6]. A clearly defined and well documented security policy is regarded as essential in protecting information infrastructure including robust user authentication [4]. Two factor authentication is widely deployed and believed to be much more impervious to attack than simple passwords and PINS. However two factor authentication is itself vulnerable to social engineering [9] [10] by essentially tricking the victim into providing the verification code to the social engineer using the type of technique described in [9] and [10].

This illustrates the inescapable fact that the central goal of social engineering is to evade whatever security mechanisms are deployed to protect the target system by exploiting human vulnerabilities [8] [6] [5]. Grimes [9] observes, from an industry perspective, that “$\cdots$ even the very best technical controls will allow some amount of social engineering to get by your defenses. What will save you then is good security awareness training.” Indeed [8] cites case studies which “$\cdots$ demonstrate that security awareness is the crucial and most effective tool in the fight against social engineering attacks $\cdots$”.

Security awareness training is discussed from various points of view in [8], [6], [11] and [9] with the caution that this needs to be ongoing to overcome the natural tendency to forget or habituate [8]. Resistance training, “$\cdots$ aimed at making employees resilient against persuasion techniques that a social engineer may employ” [4] are also advocated by some, particularly where the interaction is in the form of a dialog.

Extreme types of security awareness are proposals, discussed in [6] and [11], for a system to monitor the emotional state of, for example, call centre operators, to ensure they are in the correct frame of mind to resist social engineering. However we regard this type of intrusive approach as draconian and likely to be ineffective because it would generate resentment and encourage evasion by the operator as well as being disruptive when the monitor decreed the operator to be in the wrong frame of mind.

A more benign and, we suggest, more effective approach is to provide potential victims with an automated assistant which monitors the interaction with an attacker to detect indications that a social engineering attempt may be underway. For example, Bhakta and Harris [11] describe a system for analysing each line in a text based interaction to detect instances of a predefined ‘topic black list’ warning the potential victim and perhaps taking preemptive action to prevent a security violation. Similarly, in this paper we accept that the vulnerability of human operators to social engineering is inevitable and describe a system to assist them is assessing the overall reliability of the information they are being presented with.

The centrality of human vulnerability in social engineering is emphasised by the philosophically well grounded definition proposed in [1] Section IV:

Continuing research in social engineering requires a sound intellectual foundation, beginning with a consistent ontology which then provides the basis for the underlying conceptual scheme. This, in turn, enables the empirical knowledge base to be ordered by constructing more specialised taxonomies.

Previous work on developing conceptual models and taxonomies is reviewed in [6]. Recently, Wang et. al. [12] have proposed a comprehensive foundation for social engineering research comprising an ontology and conceptual model with consequent taxonomies. Their conceptual model has eleven components, [12] Fig. 5, with each of these components having their own taxonomies populated by the ontology.

Defining a set of formal relations between the components of the conceptual model leads to a knowledge graph in which the vertices represent the elements of the ontology and the edges represent the relations between the conceptual model components whose taxonomies include the elements. All of this is machine readable so that the knowledge graph can be constructed computationally from empirical data on actual social engineering attack scenarios and then analysed to reveal general features of attacks.

For example the vertex degree, i.e. the number of edges associated with a vertex, is an indicator of how frequently an ontological element features in attacks. A relatively high degree can reveal a particular human or system vulnerability so that an alarm can be generated when it occurs during an attack or special attention can be paid to it in security awareness training [12].

Human vulnerability is examined in detail in [6] which proposes an “I-E Model of Human Weakness” as a basis for investigating social engineering attacks and potential defences. Here, ‘I’ denotes internal which refers to the psychological characteristics of potential victims whereas ‘E’ denotes external referring to the social environment in which manipulation of the victim occurs. Whereas [6] uses the term ‘socio - psychology’ as a general description of this class of model, we prefer the term ‘psycho - social’ because it is consistent with the description of similar issues in the health sciences [7] where the internal-external dichotomy is also employed, although the social environment there is somewhat different.

Because “$\cdots$ the success of social engineering relies heavily on the information gathered $\cdots$” [12] one of the conceptual model components is ‘social engineering information’. The importance of information is also stressed in [4] and [2], where ‘information gathering’ is listed as the first step in the attack cycle, and in [5] where ‘information requirements analysis’ is described as one of the prerequisites of an attack.

While we agree with the generality of Definition I.1, in this paper we focus on that class of cyber security breach where the social engineer manipulates the target of an attack into revealing confidential information to be used in perpetrating some type of fraud. We distinguish between the target of the social engineering attack, on the one hand, and the victim of the subsequent fraud on the other. Whereas the target is a single human entity, the victim may be a corporate entity or the many customers of that corporate entity.

In a typical phishing attack the target is also the victim [2]. However in another class of attack, to which call centres in particular are susceptible, the social engineer attempts to persuade the target that the social engineer is, or represents, the intended victim by entering into some form of dialog. The immediate goal is to convince the target that the attacker is entitled to privileged information, information which can range from a simple account password to commercial or national secrets. If convinced, the target reveals the information to the attacker who then uses it to perpetrate some form of fraud against the victim. Identity theft, a growing problem, is a particular case of this form of impersonation.

Clearly the social engineer has to create a false identity based on some form of profile which will establish the credibility of the social engineer as the source of the information on which the attack is based. The wider question of assessing credibility in social networks has been surveyed in [13] which asserts that credibility is a synonym for believability and trustworthiness although the authors concede the multidimensional nature of credibility in the social network context. Multidimensionality is also stressed in [15] where trustworthiness and expertise are described as primary markers of source credibility. The basis of judgements of source credibility made by Facebook users is examined in [20] in four dimensions, one of which is sincerity. Judgements in this dimension were made in part on the basis of the source’s profile indicating the importance of that profile being sufficiently detailed to be convincing.

Source credibility has been examined in a variety of contexts, e.g. investigations of the evaluation of real and fake news articles [21] and the adoption of information in on-line communities [22]. In both cases, the theoretical context is provided by Dual-Process models of human information processing. Whereas the Heuristic-Systematic variant of the Dual-Process model is employed in [22], the work in [21] is based on another variant, the Elaboration Likelihood Model, originally developed to investigate persuasion.

The investigation of victimisation in phishing attacks in [23] is also based on the Heuristic-Systematic Model (HSM) where information processing occurs in two modes. In the heuristic mode a message is evaluated quickly on the basis of a set of available cues, one of which is source credibility, whereas in the systematic mode the evaluation involves cognitive processes and is consequently much slower and more deliberative.

These two modes are not mutually exclusive and have been demonstrated to operate concurrently [22] [23] such that one can moderate the other i.e. one can reinforce the other (referred to as additivity) or one can attenuate the effects of the other. In particular, source credibility has been demonstrated to reinforce the results of systematic processing [22] and, conversely, systematic processing can attenuate the effect of heuristic cues such as source credibility.

In the type of phishing attack considered here, the social engineer presents an argument aimed at persuading the target to perform some action - supplying confidential information or logging into a fake website for example. The target invokes some form of heuristic-systematic process to evaluate the argument. However if systematic processing dominates, the flaws in the argument are likely to be exposed [23] so it is in the social engineer’s interest to suppress systematic processing in favour of heuristic processing by manipulation of the target. One method of achieving this is to place the target under time pressure which suppresses systematic processing [23] [22].

An important aspect of the HSM is that it recognises that people will terminate the process when they feel comfortable with the judgement that has been evolving. In the model this is referred to as reaching the individual’s sufficiency threshold which depends on the prevailing circumstances. Lowering the sufficiency threshold will result in the individual favouring the heuristic over the systematic mode.

In [23], it is proposed that the success of an attack depends, at least in part, on using an attack pretext to suppress the target’s sufficiency threshold. Pretexting [23] [1] is where the social engineer, in preparing the attack, creates a scenario which provides the informational context within which the attack takes place, a context which can be also designed to reinforce the perception of source credibility. Indeed we regard the pretext as a core component of the type of dialog based phishing attack considered in this paper.

However we believe that a more detailed account of the relationship between the pretext and the sufficiency threshold can be developed from the Risk Information and Seeking Processing (RISP) model of information science which incorporates the Heuristic-Systematic Model. The RISP Model has been widely researched in the information sciences as an explanation for how and why people seek additional information in the context of risk (see [24] for a review and discussion). While developing the full relationship between social engineering and RISP is beyond the scope of this paper, the essential idea for our purposes is as follows.

Firstly we observe that risk is inherent in a dialog based phishing attack and that risk is perceived at some level by the target. One of the main components of RISP is information insufficiency which is defined as the difference between an individual’s current knowledge level and their sufficiency threshold. A higher level of information insufficiency, all else being equal, will drive the individual towards higher levels of information seeking and processing [25] and, consequently, to bias their processing mode towards systematic processing.

In light of this we propose that pretexting, rather than lowering the sufficiency threshold as hypothesised in [23], instead raises the perceived level of current knowledge relevant to the attack. The pretext thus decreases the level of information insufficiency thereby reducing the perceived need to seek additional information by interaction with the attacker. Furthermore a carefully crafted pretext can lower the level to the point where the heuristics will override any doubts that may arise from systematic processing of the pretext. Consequently pretexting complements source credibility by ensuring the target is more strongly influenced by source credibility and other heuristics than by the merits of the arguments presented by the social engineer.

All of these factors lead us to define a model of a socially engineered phishing (SEP) attack, which we refer to as a SEP attack model, as:

Note that all three components are fake but presumably sufficiently plausible that the target is willing to engage with the attacker.

Credibility generally increases with familiarity [15] which implies that a completely unfamiliar pretext will be regarded with considerable skepticism by the target suggesting the engagement of some level of systematic processing. Provisionally accepting the pretext implies that at least some of the components of the pretext are held by the target to be true. Because a high level of systematic processing would require that the large proportion of the components be held to be true, suppressing the sufficiency threshold reduces the proportion of components that the target needs to recognise as true in order to accept the pretext.

The social engineer is then free to invent the remaining components but with the constraint that consistency is important for credibility [15] so that inconsistency in the information presented to the target results in a greater reliance on systematic processing [22]. This clearly conflicts with the original aim of ensuring the dominance of heuristic processing. Consequently, the fake components of the pretext or message must be chosen to appear consistent with the components that the social engineer can assume the target will regard as true on the basis of the social engineer’s research into the target’s background.

All of this emphasises the critical role played by information as was stressed in Section I-B. It becomes clear that a social engineer needs to mount a significant effort to painstakingly acquire the information required for a successful attack. The nature of the information is such that the acquisition process can be indirect and complex so that social engineers “$\cdots$ will tie little pieces of information they have acquired over time, decipher cues and signals given to them by multiple employees, and then connect the pieces of the jigsaw puzzle to unearth the information they have been after” [8].

Previous work has focussed on the exploitation of whatever factual information the social engineer can extract directly from available sources. However we argue that the ‘pieces of the jigsaw puzzle’ acquired by a social engineer will in general be incomplete and perhaps inconsistent so that ‘unearth(ing) the information’ will require significantly more that merely assembling a collection of facts. This is particularly the case with the type of personal detail that will lend a strong sense of authenticity to the pretext.

Specifically, we base this paper on our assertion that information acquisition can be enhanced by inferring additional information from, not only that already acquired but, importantly, contextual information as well. Rarely, however, will this be a straightforward matter of deductive inference leading to certainty. Instead the inferences will be probabilistic in nature.

Probabilistic inference techniques are finding increasing application in database security research particularly in characterising and detecting network based attacks. In fact in a database inference attack, inference procedures are an intrinsic component of the attack methodology itself and developing effective countermeasures against inference attacks is an established aspect of database management research [16]. The underlying concept is that of an ‘inference channel’ which is generated by the patterns of association in the data which support inferences.

Of greater relevance to this paper are a number of techniques which have been developed to infer unknown (latent) attribute or demographic values from a known set derived from social network data. For example a machine learning method based on feature vectors is described in [26] whereas probabilistic graph theoretic techniques based on Markov Random Field concepts are proposed in [27] and [28].

Underlying the probabilistic approaches is the characterisation of the set of attributes as random variables described by a multivariate probability distribution which is estimated, at least implicitly, from the known values. However in both [27] and [28] the computational problem is made tractable by using binary variables and by adopting a pairwise Markov Random Field Model. The latter places quite strong constraints on the form of the multivariate distribution restricting it to a product of univariate and bivariate ‘potential’ functions [29]. In addition, the assortativity [27] of the associated social network defines a neighbourhood structure which further restricts the form of distribution with consequent restrictions on the conditional independencies between the attributes [29].

Even with their simplifying assumptions these are not simple, straightforward procedures either conceptually or computationally. To be clear, we are not suggesting that social engineers in general have the capacity to utilise these techniques to infer missing information even where the data was available. Instead we develop the argument below that the social engineer will make these inferences analogically based on a broad understanding of the socioeconomic context and human nature.

To formalise these ideas we use the terminology of Clarke [30], which distinguishes between an entity and an identity. An entity is something in the world, not necessarily human, which can present many different identities depending on the particular role that entity is playing in the relevant context. Whereas the social engineer’s intended victim is an entity, it is that entity’s role in transacting with the organisation represented by the human target that is the social engineer’s focus.

Both the social engineer and the target are dealing with the particular identity presented by the entity comprising that subset of the entities attributes required for the role. Identities are distinguished by the values that those attributes have in each particular case, those values providing the basis for identification.

For identification purposes, the target will have access to a record associated with the victim in the form of a digital persona which Clarke defines as “(t)he collection of data stored in a record (which) is designed to be rich enough to provide the record-holder with an adequate image of the represented entity or identity” [30]. Bearing in mind that the digital persona is, by design, a limited view of the victim, the key aspect of this is the social engineer having a sufficiently plausible profile of the victim to convince the target that the social engineer is indeed the victim.

Unfortunately this is made increasingly easy by the eagerness of people in general to place personal details in specific profiles on the various forms of social media. However, even hiding identities behind handles is an increasingly flimsy defence [1] so that a social engineer is increasingly able to acquire key elements of the identity relevant to the attack.

An important component of the social engineer’s skill set is collecting these identity elements from a variety of public sources. Nevertheless, in general, publicly available information will be insufficient for plausibility, requiring supplementary information to flesh out the persona giving the target the sense that they are indeed communicating with the victim. For this to succeed the supplementary information need not be accurate but merely consistent with the digital persona.

One of the key proposals of this paper is that this supplementary information can be derived by the social engineer, in part by inference from already obtained information perhaps by formal statistical techniques or, more than likely, by using general background knowledge of existing socio-economic patterns. Indeed we assume that astute social engineers are adept at observing and exploiting patterns in the socio-economic events surrounding them. Experimental psychology has demonstrated a clear connection between explanation, on the one hand, and learning and inference on the other [31][32] so we assume, in addition, that those patterns which have an explanatory context, will predominate.

The implication is that the social engineer makes inferences in an explanatory framework. Based on the close psychological connection between explanation and analogy [33], we believe that this is best categorised as a form of analogical inference. For example a young person living in a country town is in an analogous situation to many young people living in many country towns. It can then be inferred that a particular young person is very likely to be unemployed because the economies of many country towns are contracting offering limited employment opportunities to young people as is well known.

The essential point is that it is this analogous explanatory framework which lends plausibility, in the mind of the target, to the inferred information used in perpetrating the fraud rather than the formal statistical properties of the publicly available information. Consequently we do not require social engineers to have access to a formal database, nor to have the skills to analyse the data even if they did. Nevertheless the social engineer will be well aware informally of those statistical properties which will form part of the motivation for drawing the inferences depending on the perceived level of statistical relevance.

The notion of statistical relevance in its various forms has played a role in the philosophical analysis of explanation in a scientific context since [34]. In experimental psychology the relationship between explanatory power and statistical relevance has been demonstrated using simple measures,[35]. At a more theoretical level, Schupbach and Sprenger [36] characterise statistical relevance in terms of the degree to which one proposition H (e.g. a hypothesis) makes a second proposition E (e.g. empirical evidence) less surprising. They then use this in a Bayesian context as part of their definition of a measure of explanatory power which Schupbach [37] has demonstrated successfully accounts for the judgements of explanatory power made by subjects in a series of psychological experiments.

All of this suggests that astute social engineers will generally base their analogical inferences on informal assessments of statistical relevance derived from observation of socio-economic patterns. However, instead of attempting to describe and analyse analogical inferences, we focus on the more easily defined task of analysing the underlying probabilistic structure that the psychological evidence suggests reflects and supports analogical inference. While the explanatory aspects of the type of analogical inference referred to here have a strong causal flavour, we will regard probabilistic information as purely associative having no sense of causality [38]. However we recognise that this creates the potential for false positives.

By analysing the probabilistic structure of the publicly available information we can anticipate the drawing of analogical inferences and a key outcome of our research is an indicator of the degree to which a dataset is susceptible to inference. This indicator can be exploited to design countermeasures that reveal, and warn of, the patterns of association that underly the analogical inference, alerting potential targets to the possibility that information that they might be presented with is unreliable.

We have discussed the critical role played by information in Sections I-B and II-B with an essential part of the attack preparation being the careful assembly of as much detail about the attack victim and context as is feasible [12] [4] [2] [5]. This information is used the construct the pretext which is an essential component of a social engineering phishing attack. The Heuristic-Systematic (HS) model of information processing developed in the Information Sciences has previously been used in [23] as a basis for analysing the interaction between attacker and target.

However, because the attack occurs in the context of risk, in this paper we extend that analysis by adopting the Risk Information Seeking and Processing (RISP) model [25] which incorporates the HS model. An important component of the RISP model is information insufficiency which acts as the primary motivator for seeking and processing further information. In Section II-B we invoke the RISP model to argue the importance of a sufficiently convincing pretext to reduce the target’s information insufficiency level to the point where the target avoids the systematic mode of processing. The target then does not subject the available information to the more detailed analysis which might raise doubts in the target’s mind about the veracity of the attacker’s claims.

Our underlying thesis is that the details revealed by the attacker’s careful assembly of existing information are generally insufficient for building a convincing pretext. However they can be sufficient for the social engineer to infer probabilistically enough additional details to persuade the target that the claims being made in the pretext are valid if the target processes the claims heuristically. Consequently a seemingly convincing pretext may be less substantial than it appears because it is based partly on inferred attribute values. This raises the question of whether this situation can be detected and, if so, can the degree to which a pretext is dependent on inference be estimated. We seek to answer this question in this paper.

Our primary goal, then, is to devise a means of advising potential targets, particularly in informal database environments, that certain combinations of attributes might be unreliable for identification because the values of some are inferable from others. In other words the social engineer is likely to have inferred some attribute values thus appearing to be the victim who of course would know those values. We believe that our approach of using indications of inference as the basis for advising targets of the need for caution has not previously been reported in the literature.

Note that this is quite distinct from describing methods for actually drawing inferences, using, for example, the conceptually and computationally complex processes described in [26], [27], and [28]. Instead, in Section III-A below we develop a method of estimating the potential of a data set for inference to be performed without actually computing the inferences thus avoiding much of the complexity.

In the following we proceed from the observation that inference relies on relationships between the attributes where, in this case, the relationships are not only explanatory, thus supporting analogical inference, but are also manifested statistically enabling complementary probabilistic inference. The principal theoretical focus of the paper, then, is on the identification of attributes where unknown attribute values can be estimated by probabilistic inference from some database using the known values of other attributes.

Like [17], and [19] we assume the existence of a database derived from some defined population from which can be extracted a multidimensional contingency table ${\mathcal{T}}$ in which the cells of the table are indexed by an ordered index vector $\mathbf{I}$. Each cell contains the number of members of the population with the combination of attribute values referenced by the index so the table can be conceptualised as an unnormalised discrete multivariate probability distribution. However, instead of using the posterior joint distribution [19] directly or the posterior marginal distribution [17], we employ a log linear transformation of the prior distribution to analyse the probabilistic relations between subsets of attributes which give rise to the structure in the table.

Because inferences are drawn from prior information we focus on the contingency table equivalent of a conditional probability distribution which we refer to as a conditional subtable. We will assume inference is immediate in the sense that the conditional distribution is dominated by, i.e. concentrated on, a particular combination of attribute values. These can then be taken as the unknown values with high likelihood.

We develop a novel computational procedure for detecting particular subsets of attributes which have conditional subtables dominated by a small set of values without having to analyse each individual subtable. The basis of this procedure is the observation that a uniform distribution is completely uninformative and that, in order for probabilistic inference to be viable, a distribution must exhibit significant deviation from uniformity. This is because of some property of whatever mechanism is generating the cell contents - a mechanism that causes some cells to be significantly more probable than others.

In other words these cells are salient and we measure salience as a deviation from uniformity. Roughly speaking we argue that a high level of salience in a subset of attributes results in the multivariate conditional subtable of those attributes being concentrated on just some small number of the attribute levels in the subtable. This does not require that the dominant cells have any topological relationship. In particular it does not require that they be neighbours so that standard measures based, for example, on second moments, are not suitable for our purpose.

In Section III-B we develop the vector space theory underlying our analysis utilising the orthogonal log linear transformation introduced by Dahinden et. al. [39]. Importantly this is based on projecting the vector representing the logarithm of the table onto a subspace orthogonal to the uniform vector. After analysing the orthogonalised log linear design matrix we state and prove Theorem III.1 in Section III-D. This novel result enables us to reduce a large contingency table to one involving a subset of attributes using geometrical means as a form of marginalisation while preserving salience as much as possible.

Theorem III.1 also provides the basis for our definition of a novel indicator, Probabilistic Salience, of the degree to which statistical data can support an inference that particular attributes have specific values (Definition III.2). We describe a means of evaluating that indicator from the data and demonstrate how it can identify those subsets of attributes in a contingency table which are most vulnerable to inferencing . Then, in Section III-E, we show why Probabilistic Salience does indeed measure the degree to which a subset of attributes is dominated by a relatively small number of members.

Most of Section III-B1 as well as the rest of Section III-B report the results of research which, to the best of our knowledge, is original. We believe that this work also makes a significant contribution to the contingency table literature because we were unable to find anything there that investigated this type of problem.

We envisage a novel system which employs Probabilistic Salience to generate a warning when some of the attribute values being claimed by a social engineer might have been inferred from others without having to classify them. The warning would take account of the level of any formal authentication provided and could be as simple as a red, amber, green traffic light indication. While this system would be most easily implemented in a text based dialog we see nothing in principle which would prevent speech recognition being used in a verbal dialog. Such a system could complement the topic blacklist system described in [11] as well as that referred to in [12] and might form a component of a more comprehensive intelligent assistant for combatting social engineering attacks in real time.

The central objective of our proposed probabilistic salience based warning system is to increase the target’s information insufficiency level by inducing the target to derate the inferred attributes thereby reducing the target’s current knowledge level. If the inferred content is large enough, the information insufficiency level will increase to the point where the target transitions to an information processing mode dominated by systematic processing.

In this mode the target could be expected to take a more deliberative and analytic approach to evaluating the attacker’s claims and expose the inevitable flaws and inconsistencies leading to a rejection of the attack. It is not intended that the system directly intervene in the attack by, for example, preemptively terminating the contact.

Investigating the conditions under which probabilistic inferencing can be performed, and assessing the potential for probabilistic inferences to be drawn is, to the best of our knowledge, a novel approach which this paper brings to the understanding of social engineering. Furthermore this novel approach is able to draw upon the understanding of human information processing being developed in the information sciences to begin constructing techniques for actively counteracting social engineering. We suggest that this is an example of a potentially fertile area of research applying the characteristics of human information processing in the context of risk to the development of effective countermeasures to social engineering.

Finally in Section V we propose a potential application of Probabilistic Salience and Theorem III.1 in what we refer to as ‘de-personalisation’ of a data set by analogy with the de-identification techniques used in data privacy. The objective here is to inhibit probabilistic inference by selectively reducing the data’s Probabilistic Salience without excessively compromising its utility.

The general problem area is that of $N$ dimensional contingency tables [40] formed from an ordered set of categorical variables, i.e. attributes, $C=\{C_{N-1},\cdots C_{0}\}$. Each variable consists of a set of values or levels such that, for example, the $k$th variable, $C_{k}$, comprises the $M_{k}$ levels $\{a_{k1},a_{k2}\cdots a_{kM_{k}}\}$. There are $M_{T}=\prod_{k=1}^{N}M_{k}$ cells in the table, each being identified by an ordered index set

where $\mathfrak{i}_{k}\in\{0\cdots M_{k}-1\}$ so that each attribute $C_{k}$ is directly associated with the index $\mathfrak{i}_{k}$. Each cell contains the number of members of the population which exhibit the joint occurrence of the attribute levels. However, for notational convenience in what follows, there will be the same number of levels in each attribute, $M$, so that $M_{T}=M^{N}$.

We regard the table as a geometric entity, in this case a hypercube of cells. To provide flexibility in associating an $N$ element vector of attribute values with the indices of the corresponding cell, we adopt the convention of representing the $k$th ‘axis’ of the hypercube by the $N$ element standard basis vector $\bf{e}_{k}$ in which the $k$th element is ‘$1$’ with the remainder zero. This enables the set of attributes to be referenced by the index matrix

If the levels of the attributes designating the $\ell$th cell are collected in the vector $\mathbf{a}_{\ell}=\langle a_{\ell_{N\!-\!1}}\cdots\mathbf{a}_{\ell_{0}}\rangle$, $\mathbf{a}_{\ell_{j}}\in\{0\cdots M_{k}-1\}$, the indices of the cell are given by

The index matrix construct (1) also provides a mechanism for dealing with subtables. If $C_{\ell_{k}}\cdots C_{\ell_{1}}$ is some subset $\mathbf{\mathsf{c}}^{\ell}$ of $k$ attributes, the attribute index vector $\mathbf{i}^{s}_{\mathbf{\mathsf{c}}}$ is

where there is no implication that the integers $\{\ell_{k}\cdots\ell_{1}\}$ are contiguous. Using $\mathbf{i}_{\mathbf{\mathsf{c}}}$ as a key provides the subtable index matrix

The contingency table is assumed to be a member of an ensemble of similar tables with the common characteristic that they represent the same population so that their respective entries sum to the population size say $N_{T}$. More formally this is equivalent to assuming that each table is drawn from a multinomial distribution although we will not make use of this fact. These tables can be represented as vectors by ordering the table indices according to some rule. Here we choose a lexicographic ordering in which the index, $\mathfrak{i}_{0}$ varies the most rapidly followed by $\mathfrak{i}_{1}$ and so on resulting in a table vector, $\mathcal{T}\in\mathfrak{R}^{M_{T}}$.

Much of the literature on contingency tables is concerned with the estimation of the cell contents from incomplete data. However, to avoid being distracted by estimation procedures, important though they are, we will assume that the data in the cells is a reliable representation of the population. The only complication is presented by zero entries in one or more cells but we will take a pragmatic approach here and replace zeros by ones by applying an affine transformation to the table with minimal impact on the conclusions for reasons that will become apparent below.

The log linear transformation is based on the cell by cell logarithmic transformation of a contingency table vector ${\mathcal{T}}=[\rho_{0},\cdots\rho_{M_{T}}]^{\mathsf{T}}$ to generate a new table $\mathbf{T}=[\gamma_{0}\cdots\gamma_{M_{T}}]^{\mathsf{T}}$ where $\gamma=\log\rho$. In its general form the log linear transformation is ([41])

where the $\xi$ functions are parameterised by a subset of the attributes.

In what follows it is essential to impose some form of ordering on the set of subsets $\mathbf{\mathsf{c}}$ i.e. on the power set of $C$, $\mathbb{P}(C)$. In the ordering adopted here, the first term has $|\mathbf{\mathsf{c}}|=0$, i.e. the null set representing the constant term. Next comes the group of subsets with $|\mathbf{\mathsf{c}}|=1$ representing the main effects , then those with $|\mathbf{\mathsf{c}}|=2$ and so on. The terms with $|\mathbf{\mathsf{c}}|\geq 2$ are referred to as ‘interaction terms’ because they represent probabilistic interactions between the members of the subset [41][29], beginning with first order interactions between pairs of attributes. Within each group the subsets are ordered following the lexicographic principle applied to their member’s indices. For example where $|{\mathbf{\mathsf{c}}}|=2$, i.e. first order interactions, the ordering is

Clearly, defining the $\xi$ functions, which are in the form of an $M_{T}$ dimensional vector, is a critical task in constructing the expansion. The key development on which the process described here is constructed has been introduced by Dahinden et. al. [39] where they express each $\xi$ function as a linear combination of a set of basis vectors which span the subspace containing it. There is one basis vector for each particular combination of the levels associated with a subset of attributes. The number of such basis vectors, i.e. the dimension of the subspace, is determined by the number of possible configurations of the levels which is $M^{|\mathbf{\mathsf{c}}|}$. Each $\xi$ vector can then be obtained from the data itself by determining the coefficients of the linear combination.

In matrix form, the log-linear model is

where ${\bf X}$ is a design matrix and ${\boldsymbol{\alpha}}$ is a vector of coefficients of the form

The basis vectors of a particular $\xi$ function then form a subset of the column vectors of $X$ i.e. a submatrix of $X$ where each submatrix represents one of the subsets, $\mathsf{c}$, of attributes involved in a particular level of interaction. Dahinden et. al. label the submatrices so generated as $X^{C_{k1},C_{k2}\cdots C_{k|{\boldsymbol{\mathsf{c}}}|}}=X^{\mathsf{c}_{k}}$ where the elements of each submatrix are either zero or one. The algorithm for generating the basis vectors is given in [39] but the essential idea is described below. However, because the detailed form of the design matrix is a critical part of our development, we examine its construction in more detail in Section III-C.

To proceed, it is necessary to introduce some additional notation. Firstly let $k$ be the interaction index, $k=|\boldsymbol{\mathsf{c}}|$, so that $k=0$ indexes the constant term in (4), $k=1$ the main effects and so on. For each value of $k$, the number of submatrices of the form $X^{\boldsymbol{\mathsf{c}}_{i}}$ is $N_{k}=\binom{N}{k}$ and the number of columns in each submatrix is $M^{k}$, one column for each of the possible combinations of levels in the set of $k$ attributes. Keep in mind that each cell in the table ${\mathcal{T}}$ is denoted by a specific index vector $\boldsymbol{\mathfrak{i}}=\langle\mathfrak{i}_{N-1}\cdots\mathfrak{i}_{j}\cdots\mathfrak{i}_{0}\rangle,\;\mathfrak{i}_{j}\in\{0\cdots M-1\}$ so that the $r$th component of the table vector $\mathbf{T}$ is denoted by

where $(\mathbf{R})_{M}$ is the $M$-ary number $\boldsymbol{\mathfrak{i}}$ expressed in radix $M$ form. Note that this is simply the operation of counting through the elements of the table vector with radix $M$.

Consider one specific value of $k$ and one specific submatrix, $X^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}$, where $\boldsymbol{\mathsf{c}}^{\ell}_{k}\in\boldsymbol{\mathsf{c}}_{k}\subseteq\mathbb{P}(C)\ s.t.\ |\boldsymbol{\mathsf{c}}_{k}|=k$ and $\ell\in\boldsymbol{\ell}=\{1\cdots\binom{N}{k}\}$ which specifies the subsets $\boldsymbol{\mathsf{c}}^{\ell}_{k}$ of $\boldsymbol{\mathsf{c}}_{k}$. For each value of $\ell$, the attributes are designated by the ordered vector of attribute numbers $\mathbf{i}^{s_{\ell}}_{k}=\langle i_{\ell_{k}}\cdots i_{\ell_{1}}\rangle$ so that e.g. $C_{i_{\ell_{k}}}$ is the leftmost attribute in the subset. Then each column of the submatrix represents one particular combination of the levels of the subset of attributes designated by $\mathbf{i}^{s_{\ell}}_{k}$. Note that there is no implication that the $k$ indices are contiguous which, in general, they are not as indicated by the example (5).

If the levels of the attributes are $a\in\{0\cdots M-1\}$, the vector $\mathbf{a}^{k,\ell}=\langle a_{\ell_{k}}\cdots a_{\ell_{1}}\rangle$, represents one combination of values of the attributes $\boldsymbol{\mathsf{c}}^{\ell}_{k}$. The $\nu$th column of the submatrix is then associated with a particular $\mathbf{a}^{k,\ell}_{j}$ as well as $\mathbf{i}^{s_{\ell}}_{k}$. The remaining ${(N-k)}$ table axes form the vector $\mathbf{i}^{g_{\ell}}_{k}$ with corresponding attributes vector $\mathbf{g}^{k,\ell}=\langle g_{\ell_{N-k}}\cdots g_{\ell_{1}}\rangle$. Using (7), the elements of the $\nu$th column vector having $\mathbf{i}^{s_{\ell}}_{k}$ and $\mathbf{a}^{k,\ell}_{j}$ in common, $\{x\}_{g}$, are designated by the table index vector, from (2) and (3)

where the elements of $\mathbf{g}^{k,\ell}$ range over all $M^{N-k}$ combinations of the attribute levels.

To define the basis vector recognise that the available information does not allow the elements of $\{x\}_{g}$ to be distinguished. Consequently the $\nu$th column vector becomes a basis vector by setting the elements of $\{x\}_{g}$ to unity with the remaining elements zero.

The lexicographic ordering is a bijective mapping, so that if there is a one at a particular position in the $\nu$th column of the submatrix $X^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}$, there cannot be a one in the corresponding position in any of the remaining $M^{k}-1$ columns. Consequently, the columns of the submatrix in question, i.e. the basis vectors, are orthogonal. Furthermore, because the total number of ones in the submatrix is $M^{k}M^{N-k}=M^{N}$, the sum $\sum_{j=1}^{M^{k}}{X_{j}^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}}$ is equal to the unit vector $\widehat{X}^{0}$.

The full matrix then has

columns so that (6) is very underdetermined. Conventionally the approach to this problem is to impose ‘identifiability’ by removing the null space of $X$ using, for example, the Moore-Penrose pseudo inverse. However this approach results in losing the explicit connection between the coefficient vector and the interaction terms.

Fortunately Darhinden et. al., [39], have derived an orthogonal form of $X$ which leads to a coefficient vector that can be related to the interaction terms in (4). The essential idea is as follows. Using a modified version of the notation in [39], the vector space, $\widehat{X}^{0}$ spanned by the constant vector $k=0$ has dimension 1. Then the vector space $\widehat{X}^{C_{\ell}}$, spanned by the columns of $X^{C_{\ell}}$, is composed of $\widehat{X}^{C_{\ell}}_{-}=\widehat{X}^{0}$ and its orthogonal complement, $\widehat{X}^{C_{\ell}}_{\perp}$, in $\widehat{X}^{C_{\ell}}$ the orthogonal complement having dimension $M-1$. The space, $\widehat{X}^{C_{\ell_{1}}C_{\ell_{2}}}$, spanned by $X^{C_{\ell_{1}}C_{\ell_{2}}}$ is composed of ${\widehat{X}}^{C_{\ell_{1}}C_{\ell_{2}}}_{-}=\{\widehat{X}^{C_{\ell_{1}}},\widehat{X}^{C_{\ell_{2}}}\}$ and its orthogonal complement, ${\widehat{X}}^{C_{\ell_{1}}C_{\ell_{2}}}_{\perp}$ in ${\widehat{X}}^{C_{\ell_{1}}C_{\ell_{2}}}$ which has dimension $M^{2}-2(M-1)-1=(M-1)^{2}$.

For the $k$th interaction index, there are $\binom{N}{k}$ spaces of the form $\widehat{X}^{C_{\ell_{1}}\cdots C_{\ell_{k}}}$ where each of the $\binom{N}{k}$ sets $\{\ell_{1}\cdots\ell_{k}\}$ is a distinct subset of the index set $\{1\cdots N\}$. The space spanned by the columns of $X^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}$, $\widehat{X}^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}$, is composed of the subspace

and its orthogonal complement in $\widehat{X}^{C_{\ell_{1}}\cdots C_{\ell_{k}}}$, $\widehat{X}^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}_{\perp}$. This orthogonal complement has dimension

Beginning with the constant vector, the orthogonalised form of $\bf{X}$, ${\overline{\bf X}}$, is derived by successively projecting the columns of the submatrix $X^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}$ onto the orthogonal complement $\widehat{X}^{\boldsymbol{\mathsf{c}}^{\ell}_{k}}_{\perp}$. Then ${\overline{\bf X}}$ is constructed from the constant vector and the columns of the submatrices following the ordering described above. Consequently the number of columns is

so that ${\overline{\bf X}}$ is a square matrix.

To explore the overall form of the design matrix $\overline{X}$ we return to the construction of the original form $X$ and the discussion following (7). The columns of the submatrix $X^{C_{i_{k}},\cdots C_{i_{1}}}$ are numbered left to right by the radix $M$ number $\nu=(a_{k},\cdots a_{1})_{M}$ i.e. in the first column of the submatrix, $a_{k}=0\cdots a_{1}=0$ whereas in the second column $a_{k}=0\cdots a_{2}=0$ and $a_{1}=1$. There are therefore $M^{k}$ columns in the submatrix where each is generated from the preceding column by counting in radix $M$.

Equation (20) can be expressed in the alternative form

Now consider the orthogonalisation process. With $\mathtt{X}^{i}$ denoting a column of the submatrix $X^{C_{i}}$, the first step obtains the projection of the column vector onto the orthogonal complement of the normalised constant vector $\overline{X}^{0}$ as

Applying (22) to the form (21) gives

Note that because (22) operates on rows only and the projection is onto the uniform vector, the hierarchical sequence of patterns in (21) is preserved in (23).

Using the case $k=2$ with $a_{1}=a_{2}=0$ to illustrate the process, the orthogonalisation represented by (22) to (23) results in $\mathtt{X}^{i_{2}i_{1}}_{\perp}$. Then, the projection of this onto the orthogonal complement of the subspace formed by $\overline{\mathtt{X}}^{i_{2}}_{\perp}$ and $X^{0}$ is obtained by again applying (22) resulting in $\mathtt{X}^{i_{2}i_{1}}_{\perp\perp}$. Finally, projecting this onto the orthogonal complement of the subspace formed by $\overline{\mathtt{X}}^{i_{1}}_{\perp}$, $\overline{\mathtt{X}}^{i_{2}}_{\perp}$ and $X^{0}$ yields

The form of (24) and numerical calculations suggest the $k$ attribute generalisation

For $\mathbf{a}_{k}=\langle 0,\cdots 1\rangle$ the column vector $\mathtt{X}^{i_{k}\cdots i_{1}}_{\perp\cdots\perp}(0,\cdots 1)$ lies in the orthogonal complement of the uniform vector and $\mathtt{X}^{i_{k}\cdots i_{1}}_{\perp\cdots\perp}|(0,\cdots 0)$. Explicit derivation of the $k=2$ case and numerical calculations suggest that the $j$th second level module has the form

Note that this has the same modular structure as (25) and again the first level modules sum to zero. Indeed the structures are sufficiently similar that the above orthogonality arguments can be invoked to show that (27) is orthogonal to (25) for all $k$ as well as to the corresponding $\mathtt{X}^{i_{k}^{\prime}\cdots i_{2}i_{1}}_{\perp\cdots\perp}(0\cdots 1)$ for $k^{\prime}\neq k$.

Because the origin of this critical modularity is (23), the orthogonalisation process applied to all of the remaining independent column vectors of $X^{\boldsymbol{\mathsf{c}}_{k}^{\ell}}$ will result in the orthogonalised submatrice ${\overline{X}}^{\boldsymbol{\mathsf{c}}_{k}^{\ell}}$ having the same modular structure as (25) and (27).

Each term of the log linear expansion (4), $\xi_{\boldsymbol{\mathsf{c}_{k}^{\ell}}}$, involves the subset of $k$ attributes $\mathsf{c}_{k}^{\ell}$ which raises the question of how to handle the remaining $N\!-\!k$. While marginalisation is the common approach, the nonlinear $\log$ transformation makes the result difficult to interpret. Instead we adopt the alternative approach in which the members of $\mathsf{c}$ determine a subtable conditioned on the remaining attributes.

Recall from Section III-B that the $k$ attributes associated with the submatrix ${\overline{X}}^{{\boldsymbol{\mathsf{c}}_{k}^{\ell}}}$ are designated by $\mathbf{i}^{s_{\ell}}_{k}$ with the remaining $N\!-\!k$ attributes designated by the vector $\mathbf{i}^{g_{\ell}}_{k}$. Via (7), the column vectors of the submatrix have exactly the same indexing structure as the contingency table $\mathbf{T}$, so, from (8), we can interpret an index vector of the form $\boldsymbol{\mathfrak{i}}$ as indexing a subtable designated by $\mathbf{g}^{k,\ell}$. This leads to: