Statistical analysis of the Indus script using $n$-grams

The civilisation that flourished from 7000 BCE to 1500 BCE in the valley of the river Indus and its surroundings was the first urban civilisation in the Indian subcontinent. At its peak, in the period between 2600 BCE and 1900 BCE, it covered approximately a million square kilometers [1], making it the largest urban civilisation of the ancient world. The remains of the civilisation were first found in Harappa and, following historical convention, is called the Harappan civilisation. The terms Harappan civilisation and Indus civilisation are used interchangeably in this paper.

The Indus people used a system of signs, which has mainly survived on seals (see Fig. 1 for an example), pottery, and other artifacts made of durable materials such as stone, terracotta, copper. The sign system is yet to be deciphered. The use of these signs is usually in short texts, numbering not more than $14$ signs in a single line of text. Around $400$ distinct signs have been identified [2, 3], though some researchers identify up to $676$ distinct signs [4]. The total texts number to about $3000$. The lack of decipherment of the sign system is attributed to the paucity of the working material, the absence of any bilingual texts that might aid in decipherment, and a lack of definite knowledge of the underlying language, assuming the script was used to write language. The sign system of the Indus remains controversial, with contested claims of decipherment, and even the suggestion that it does not encode language [5]. Here we shall use the terms sign system and script interchangeably to refer to the Indus script.

The main methodological difficulty in attempting any interpretation of the script is that, due to the paucity of information on the context of the writing, one is perforce required to make an a priori assumption regarding the information the script is meant to convey. Thus, the range of opinion on what the script encodes varies from an Indo-Aryan language [6], a Dravidian language [3], a purely numerical system [7], to non-linguistic symbols [5]. There is no consensus on any of the above interpretations.

A line of research, which does not require such a priori assumptions, is the statistical approach. At its heart lies the notion of recognising patterns through counting. While such an approach cannot shed light on the semantics of the script, it can reveal important features of its syntax. Research using the statistical approach was initiated by a Soviet team in 1965, further developed by a Finnish team in 1969 (for review of various attempts see [8, 9, 10]), continued by Siromoney [11] in the 1980s and followed up more recently by some of us [12, 13, 14].

In this article, we bring to bear the tools of $n$-gram modelling for a thorough statistical analysis of sequences in the Indus script. An $n$-gram is a sequence of $n$ units, or tokens. $N$-gram models provide a very useful method of modelling sequences¹¹1The examples of English language used in the paper to explain various concepts related to n-gram modelling are given for the ease of comprehension, as a representative of the general class of sequences. These linguistic examples do not imply that any linguistic assumption has been made for the Indus script., whether they be the sequence of letters or words in a natural language, the base pairs in the genetic code, or the notes in a musical score. This generality is possible because $n$-gram models are indifferent to the semantic content of the units or tokens making up the sequence (the words, the letters, the base pairs or the notes) but, nonetheless, reveal the syntax, if any, that the sequences follow. The $n$-gram approach, then, forms the ideal framework in which the Indus script can be analysed without making any a priori assumptions.

Our work in the paper uses a bigram model, that is an $n$-gram model with $n=2$, to model the sequence of signs in the Indus script. Our analysis reveals that the sign system of the Indus is structured, with specific text beginner and text ender signs, that there is directionality, strong correlations in the order of the signs, and groups of signs which appear to have the same syntactic function. In Section 2, we provide details of the corpus used in this study and present results of an empirical statistical analysis of the corpus. The empirical analysis shows that a small number of signs account for the majority of the usage, a fact we are able to capture by fitting the frequency distribution of signs to the well-known Zipf-Mandelbrot law. We note in passing, that word frequency distributions in natural languages also follow this law. In Section 3 we provide a summary of the theory of $n$-gram models, paying special attention to the problem of unseen $n$-grams. We describe how this problem is resolved using smoothing and backoff algorithms and demonstrate how this modifies the statistics obtained from the empirical analysis. Our main results, summarised above, are presented in Section 4 using a bigram model. The Indus script corpora contains many instances of texts in which signs are damaged beyond recognition or otherwise illegible. The bigram model can be used to suggest a set of signs, ranked by probability, for the restoration of the damaged or illegible signs. In Section 5 we outline the theory behind this method and provide examples on known texts from the corpus. In the final section, we discuss future work based on the $n$-gram approach and end with a summary of the present work.

Any work on the Indus script, statistical or otherwise, must begin with a corpus. Three major corpora of the Indus texts, by Mahadevan [2], Parpola [3] and Wells [4] are available. We use the electronic concordance of Mahadevan, henceforth referred to as M77, which records $417$ unique signs²²2The serial number of the signs used in this paper is as given by Mahadevan in his concordance (M77). As a convention followed in the present paper, the texts depicted by pictures are to be read from right to left, whereas the texts represented by just strings of sign numbers are to be read from left to right (see M77 for discussion on direction of texts). in $3573$ lines of $2906$ texts. In order to remove ambiguity, an Extended Basic Unique Data Set (EBUDS) is created by removing from the concordance all texts containing lost, damaged, illegible, or doubtfully read parts [12]. Texts spread across multiple lines are also removed. For texts that occur more than once, only a single copy is retained. Variations due to the archaeological context of the sites, stratigraphy, and type of object on which the texts are inscribed are, at present, not taken into account in the interests of retaining a reasonable sample size.

The reasons for discarding texts which are spread over multiple lines are twofold. First, it is not clear if each line of a multi-line text is to be treated as a single text having a continuity of sequence, or if it is to be regarded as separate text. Second, assuming a continuity of sequence, the order in which the texts are to be read across lines remains ambiguous [2].

The pruned EBUDS dataset contains $1548$ texts. In EBUDS, $40$ signs out of $417$ present in the sign list of M77 do not make their appearance. Out of these removed $40$ signs, one sign ($374$) appears $9$ times, one sign ($237$) appears $8$ times, two signs ($282$, $390$) appear $3$ times, three signs ($324$, $376$, $378$) appear twice and thirty-three signs appear only once in M77. Fig. 2 compares the distribution of texts lengths in various datasets and shows the effect of the removal of texts on the final dataset (EBUDS). We have already shown in our earlier work [12], that the relative frequency distribution of signs in EBUDS is comparable to M77 and hence EBUDS is a true representation of M77, with a reduction in total sign occurrences, but not in the percentage of total sign occurrences.

We first present the results of an empirical statistical analysis of the EBUDS corpus. Fig. 3 shows the frequency distribution of signs in EBUDS. The sign corresponding to $342$ in M77 (), is the most frequent sign, followed by signs $99$ (), $267$(), and $59$ (). The relative frequencies have no significant change in the M77 and EBUDS corpora.

The main purpose of an $n$-gram model is to capture the correlations that exist between tokens $s_{1},\ldots,s_{N}$ in a sequence ${\mathcal{S}}_{N}$. Thus, conditional probabilities form the core of an $n$-gram model. Specifically, for a string ${\mathcal{S}}_{N}=s_{1}s_{2}\ldots s_{N}$ the $n$-gram model is a specification of conditional probabilities of the form $P(s_{N}|s_{1}s_{2}\ldots s_{N-1})$, quantifying the probability that the previous $N-1$ signs of the sub string ${\mathcal{S}}_{N-1}=s_{1}s_{2}\ldots s_{N-1}$ is followed by the sign $s_{N}$. Given the $n$-gram conditional probability, and the relation between joint and conditional probabilities $P(ab)=P(b|a)P(a)$, the probability of the string ${\mathcal{S}}_{N}$ can be written as,

Recursively applying $P(ab)=P(b|a)P(a)$ to the leftmost terms, we obtain the probability as a product over conditional probabilities

In the above, it is understood that $\mathcal{S}_{0}$ is a special token indicating the start of the string. Note that the above expression is an identity that follows from the basic rules of probability and contains no approximations. As an example, the probability of a string of length three, $s_{1}s_{2}s_{3}$, is given as a product of trigram, bigram and unigram probabilities

Clearly, for an $n$-gram model to be tractable, only a finite number of such probabilities can be retained. In the simplest bigram model, all correlations beyond the preceding sign are discarded, so that

In a trigram model, all correlations beyond two preceding signs are discarded, so that

In a general $n$-gram model, all correlations beyond the $(n-1)$ preceding signs are discarded. An $n$-gram model can then be thought of as an $(n-1)^{th}$ order Markov chain in a state space consisting of the signs $s_{i}$. The introduction of the Markov assumption is the main approximation in $n$-gram models. $N$-grams were first used by Markov to analyse the probability of a consonant following a vowel in Pushkin’s Eugene Onegin. Shannon applied word $n$-grams to model sentences in English text and with $n=6$ obtained sentences which have a remarkable syntactic similarity to English. Since then, $n$-gram models have found wide use in many fields where sequences are to be analysed, including bioinformatics, speech processing and music.

The theory and applications of $n$-grams are described in several excellent textbooks [15, 17] and need not be repeated here. We summarise below our method of obtaining probabilities from counts, the use of smoothing and backoff to account for unseen $n$-grams, the measures we use for evaluating the $n$-gram model, and finally the tests we use to assign a statistical significance to the correlations.

$N$-gram probabilities are obtained from counts. For single signs, the counts are unambiguous. However, for sign pairs, it is possible that a rare combination may not be present in a small sized corpus. The count, and the resulting probability estimate for that sign pair, then is zero. However, in the absence of reasons to the contrary, common sense dictates that no sign pair should have a strictly zero probability. This intuition is quantified by various rules which remove probability weight from seen sign pairs and distributes it to sign pairs which have never been seen, but are nonetheless not impossible. Common amongst such “smoothing” rules are Laplace’s add-one rule, a method developed by Turing and Good in breaking the Enigma code, and a more recent algorithm due to Witten and Bell [18]. Here, we use the Witten-Bell algorithm to smooth our $n$-gram models. In Fig. 6 we show the estimate of the probability of a sign being followed by sign $2$, $P(b|2)$ before smoothing and after Witten-Bell smoothing. In above panel, the only non-zero probabilities are those corresponding to signs $12,14,162,$ and $176$. These probabilities sum to one, indicating that other sign pairs are impossible. The Witten-Bell smoothing algorithm restores a finite, but small probability to the unseen sign pairs, ensuring again that all probabilities sum to one. Apart from being a more reasonable way of estimating probabilities from counts, it also ensures that the resulting Markov chain is ergodic³³3A Markov chain is ergodic if it is possible to reach every state from every state (not necessarily in one move).. An ergodic Markov chain is essential in such applications, since otherwise, probabilities of all strings containing unseen $n$-grams vanish.

Smoothing is not the only way of correcting for unseen $n$-grams. Another method, called backoff uses probabilities of $(n-1)$-grams to estimate $n$-gram probabilities. Thus, the probability of unseen trigrams can be estimated from that of seen bigrams and unigrams. Here, we used the Katz backoff algorithm to estimate bigram, trigram and quadrigram probabilities when appropriate.

The estimation of $n$-gram probabilities from $n$-gram counts is called learning. A learned $n$-gram model can then be tested to see if it produces $n$-grams in the corpus. To avoid circularity, the corpus is usually divided into a training set, from which the probabilities are estimated, and a test set, on which the model is evaluated. There are several standard measures for evaluating the goodness of an $n$-gram model. Here, we use a standard measure, the perplexity, which is related to the information theoretic cross-entropy. We also do a cross-validation test using standard procedures.

Finally, we need tests of association to ascertain the significance of sign pairs which appear more or less frequently than what would be predicted by the Bernoulli scheme model. For this, we use a log-likelihood test, testing the null hypothesis that there is no association between sign pairs.

In the next section, we provide results obtained from an $n$-gram model of the Indus script. The analysis was done using the SRILM [19] and NSP [20] toolkits.

In this section we present the $n$-gram probablities obtained from the EBUDS corpus and draw inferences on the structure of the script from an analysis of the $n$-gram probabilities.

We introduce, following usual practice, two additional tokens <s> and </s> which indicate the beginning and end of a text respectively. By convention, the unigram probability for the start token is unity, $P(\texttt{<s>})=1$, since every text must begin with <s>. The probability of sign $a$ being a beginner is then $P(\texttt{<s>}a)=(P(a|\texttt{<s>})$, since $P(\texttt{<s>})=1$. The probability of sign $a$ being an ender is $P(a\texttt{</s>})$. It is also assumed, as usual, that $n$-gram probabilities do not depend on the location of the signs in a text, or more formally, that the Markov chain is stationary.

In any $n$-gram study, a maximum value of $n$ has to be chosen in the interest of tractability, beyond which correlations are discarded. This can be done in an empirical fashion, balancing the needs of accuracy and computational complexity, using measures from information theory which discriminate between $n$-grams models with increasing $n$ [15, 17], or by more sophisticated methods like the Akaike information criterion which directly provides an optimal value for $n$ [21].

In previous work, some of us have shown that bigram and trigram frequencies in the EBUDS corpus differ significantly from frequencies expected from a Bernoulli scheme [12]. The small size of the corpus limits the ability to assess significance of quadrigrams and beyond, when using the method in [12]. In our subsequent work [13] it has been shown that $88\%$ of the texts of length 5 and above can be segmented using frequent unigrams, bigrams, trigrams and quadrigrams and complete texts of length 2, 3 and 4. Moreover, frequent bigrams or texts of length 2 alone account for $52\%$ of the segmented corpus. Thus the bulk of the corpus can be segmented with $n$-grams with $n$ not exceeding $4$, and almost half the corpus can be segmented into bigrams alone. This indicates that correlations beyond quadrigrams have relatively less significance.

Here, we use an information theoretic measure, the perplexity, which we explain in detail below, to estimate an optimal value for $n$. Summarising the results of this analysis, which is presented in Table. III, we find that the perplexity monotonically decreases as $n$ ranges from $1$ to $3$ (corresponding to unigram, bigram and trigram correlations), but then saturates beyond $n=4$ (corresponding to quadrigram and higher correlations). This confirms the observation made in [13] that pentagram and higher correlations are of relatively less significance in the EBUDS corpus. From the differential reduction in perplexity with increase in model order, it is clear that the most significant correlations are due to bigrams, with somewhat modest trigram correlations, and almost negligible quadrigram correlations. Our subsequent analysis is, therefore, wholly based on bigrams. The main conclusions that we draw in this paper on the structure of the script are expected to remain broadly unaltered with the inclusion of trigram and quadrigram correlations. We expect a modest improvement in the predictive power of the $n$-gram model when trigram and quadrigram correlations are included. The role of higher-order correlations will be more fully explored in forthcoming work.

An important practical use of the bigram model is to restore signs which are not legible in the corpus due to damage or other reasons. The basic idea is to use the bigram model to evaluate the probability of a suggested restoration, and chose the restoration with the highest probability. For example, consider the three sign text $\mathcal{S}_{3}=s_{1}s_{x}s_{3}$ in which the middle sign $s_{x}$ is illegible. We use the bigram model to evaluate the probability of the string for different choices of $s_{x}$ by

The most probable sign with which to restore the text is, then, the maximum of this probability over all possible signs $s_{x}$. Since there are $417$ possible signs, this can be accomplished by a direct enumeration. When the number of illegible signs is more, the space over which the maximisation needs to be done grows rapidly. With $p$ illegible signs, there are $417^{p}$ values from which to pick a maximum. In such instances, instead of a direct search, a dynamic programming algorithm may be applied. Here, we use the well-known Viterbi algorithm[15] for finding the most probable state sequence for a given observation in a hidden Markov model, suitably modified to make it applicable to a Markov chain, to find most probable sequence of signs. Our results for sign restorations are summarised in Table  V. We list the original text, a randomly chosen deletion for that text, the most probable restoration, and the next probable restorations obtained using the bigram model. We see that in all cases, the bigram model is successful in reproducing the deleted sign. This gives us confidence that the bigram model can be used to suggest restorations of illegible signs in various corpora. Table  VI gives few examples of how the model can be used for restoration of doubtfully read signs in the texts of M77 corpus.

We can also use the model to find most probable texts of various lengths. In Table  VIII we reproduce the most probable texts of length $4,5$ and $6$ as predicted by the bigram model. It is quite remarkable that there are exact instances of the most probable texts of length $4,5$ and slight variant of most probable text of length $6$ in the corpus.

We have already discussed perplexity in section 4. Here we discuss another method to evaluate the performance of the model called cross validation. In cross validation, the corpus is divided into a training set, from which the probabilities are estimated, and a test set, on which the model is evaluated. The measure of goodness of the model calculated is sensitivity which is defined as

where $TP$ is the count of true positives and $FN$ is the count of false negatives. The ratio of training to test set size used is $80$ : $20$. We divide EBUDS into $5$ equal parts i.e. $P1$, $P2$, $P3$, $P4$ and $P5$ each being one-fifth of the corpus. We start with first part, selecting that to be the test set and concatenate the remaining parts to form the first training set. We use the training set to learn the parameters of the model and use the test set to evaluate the goodness of the model. We drop out signs randomly from the test set and ask the model to fill in the dropped signs. We then count the number of true positives, that is, the number of times the predicted signs matches with the signs under 90% area of the probability curve; otherwise, they are considered false negatives. This is repeated about 100 times with the first test set and training set and we plot the histogram of the sensitivity for these $100$ runs.

The cross validation test described above is repeated by taking each of the five equal parts of EBUDS as test set and concatenating remaining parts as training sets. The results are shown in Table VII and Fig. 11. As can be seen from the plots the sensitivity of the model considering all signs under 90% area of the cumulative probability curve as true positives is 74%.

We conclude that a bigram model, equivalent to a first-order Markov chain, can be used to capture the correlations of signs in the Indus texts and gives important insights into the structure of Indus writing. Our study has revealed that the script carries syntax, with well-defined text beginners and text enders, directionality and sign order, and strong correlations between signs. Information theoretic measures indicate that the correlations are not completely rigid. This indicates that the script can certainly be considered as a formal language, but it remains to be seen if these features imply an underlying natural language. The tests from which we draw this conclusion are summarised in Table IX. The bigram model can be used to suggest probable restorations of illegible signs from a corpus.

To the best of our knowledge, our work represents the first use of the methods of n-gram analysis to an undeciphered scipt. We believe stochastic methods hold considerable promise in elucidating syntax in undeciphered scripts. We will present further results on the syntax of Indus script, based on stochastic methods, in our forthcoming work.

We wish to acknowledge the research grant of the Jamsetji Tata Trust that enabled us to do this work. We wish to acknowledge the generous assistance of Indus Research Centre of the Roja Muthiah Research Library. We sincerely thank Prof. P. P. Divakaran for enabling a very fruitful collaboration between TIFR and IMSc. R. Adhikari gratefully acknowledges Pinaki Mitra for first arousing his interest in probability theory and the study of language. Rajesh Rao would like to acknowledge the support of the National Science Foundation (NSF) and the Packard Foundation. We are grateful to Harappa.com for their kind permission to use the picture of an Indus seal in the paper.