On the Optimality of the Kautz-Singleton Construction in Probabilistic Group Testing

The objective of group testing is to efficiently identify a small set of $d$ defective items in a large population of size $N$ by performing binary tests on groups of items, as opposed to testing the items individually. A positive test outcome indicates that the group contains at least one defective item. A negative test outcome indicates that all the items in the group are non-defective. When $d$ is much smaller than $N$, the defectives can be identified with far fewer than $N$ tests.

The original group testing framework was developed in 1943 by Robert Dorfman [1]. Back then, group testing was devised to identify which WWII draftees were infected with syphilis, without having to test them individually. In Dorfman’s application, items represented draftees and tests represented actual blood tests. Over the years, group testing has found numerous applications in fields spanning biology [2], medicine [3], machine learning [4], data analysis [5], signal processing [6], and wireless multiple-access communications [7, 8, 9, 10].

For any $t\times N$ matrix ${M}$, we use ${M}_{i}$ to refer to its $i$’th column and ${M}_{ij}$ to refer to its $(i,j)$’th entry. The support of a column $M_{i}$ is the set of coordinates where $M_{i}$ has nonzero entries. For an integer $m\geq 1$, we denote the set $\{1,\ldots,m\}$ by $[m]$. The Hamming weight of a column of $M$ will be simply referred to as the weight of the column.

We consider a model where there is a random defective set $S$ of size $d$ among the items $[N]$. We define $\mathcal{S}$ as the set of all possible defective sets, i.e., the set of $\binom{N}{d}$ subsets of $[N]$ of cardinality $d$ and we let $S$ be uniformly distributed over $\mathcal{S}$.³³3This assumption is not critical. Our results carry over to the setting where the defective items are sampled with replacement. The goal is to determine $S$ from the binary measurement vector $Y$ of size $t$ taking the form

where $t\times N$ measurement matrix $M$ indicates which items are included in the test, i.e., ${M}_{ij}=1$ if the item $j$ is participated in test $i$, $v\in\{0,1\}^{t}$ is a noise term, and $\oplus$ denotes modulo-2 addition. In words, the measurement vector $Y$ is the Boolean OR combination of the columns of the measurement matrix $M$ corresponding to the defective items in a possible noisy fashion. We are interested in both the noiseless and noisy variants of the model in (1). In the noiseless case, we simply consider $v=0$, i.e., $Y=\bigvee_{i\in S}M_{i}$. Note that the randomness in the measurement vector $Y$ is only due to the random defective set in this case. On the other hand, in the noisy case we consider $v\sim\textnormal{Bernoulli}(p)$ for some fixed constant $p\in(0,0.5)$, i.e., each measurement is independently flipped with probability $p$.

Given $M$ and $Y$, a decoding procedure forms an estimate $\hat{S}$ of $S$. The performance measure we consider in this paper is the exact recovery where the average probability of error is given by

and is taken over the realizations of $S$ and $v$ (in the noisy case). The goal is to minimize the total number of tests $t$ while achieving a vanishing probability of error, i.e., satisfying $P_{e}\rightarrow 0$.

We begin with the noiseless case ($v=0$ in (1)). The next theorem shows the optimality of the Kautz-Singleton construction with properly chosen parameters $n$ and $q$.

The proof of the above theorem can be found in Appendix -A. We note that the Kautz-Singleton construction in Theorem 1 has $t=nq=\Theta(d\log N)$ tests, therefore, achieving the order-optimal result in the probabilistic group testing problem in the noiseless case. It is further possible to extend this result to the noisy setting where we consider $v\sim\textnormal{Bernoulli}(p)$ for some fixed constant $p\in(0,0.5)$, i.e., each measurement is independently flipped with probability $p$. Our next theorem shows the optimality of the Kautz-Singleton construction in this case.

The proof of the above theorem can be found in Appendix -B. Similar to the noiseless setting, the Kautz-Singleton construction provides a strongly explicit construction achieving optimal number of tests $t=\Theta(d\log N)$ in the noisy case.

Given that the Kautz-Singleton construction achieves a vanishing probability of error with $t=\Theta(d\log N)$ order-optimal number of tests, a natural question of interest is how large the constant is and how the performance of this construction compares to random designs for given finite values of $d$ and $N$. To illustrate the empirical performance of the Kautz-Singleton construction in the noiseless case, we provide simulation results in Figure 2 and 3 for different choices of $N$ and $d$ and compare the results to random designs considered in the literature. We used the code in [31] (see [32] for the associated article) for the Kautz-Singleton construction. For comparison, we take two randomized constructions from the literature, namely the Bernoulli design (see [17]) and the near-constant column weight design studied in [18]. We use the cover decoder for decoding. The simulation results illustrate that the Kautz-Singleton construction achieves better success probability for the same number of tests, which suggests that the implied constant for the Kautz-Singleton construction may be better than those for these random designs; we note that similar empirical findings were observed in [32]. Since the Kautz-Singleton construction additionally has an explicit and simple structure, this construction may be a good choice for designing measurement matrices for probabilistic group testing in practice.

While the cover decoder, which has a decoding complexity of $O(tN)$, might be reasonable for certain applications, there is a recent research effort towards low-complexity decoding schemes due to the emerging applications involving massive datasets [33, 34, 26, 35, 36]. The target is a decoding complexity of $\textnormal{poly}(t)$. This is an exponential improvement in the running time over the cover decoder for moderate values of $d$. For the model we consider in this work (i.e., exact recovery of the defective set with vanishing probability of error), there is no known efficiently decodable scheme with optimal $t=\Theta(d\log N)$ tests to the best of our knowledge. The work [35] presented a randomized scheme which identifies all the defective items with high probability with $O(d\log d\log N)$ tests and time complexity $O(d\log d\log N)$. Another recent result, [36], introduced an algorithm which requires $O(d\log d\log N)$ tests with $O(d(\log^{2}d+\log N))$ decoding complexity. Note that the decoding complexity reduces to $O(d\log N)$ when $d=O(\textnormal{poly}(\log N))$ which is order-optimal (and sub-linear in the number of tests), although the number of tests is not. In both [35] and [36], the number of tests is away from the optimal number of tests by a factor of $\log d$.

We can convert the strongly explicit constructions in Theorem 1 and 2 into strongly explicit constructions that are also efficiently decodable by using a recursive technique introduced in [26] where the authors construct efficiently decodable error-tolerant list disjunct matrices. For the sake of completeness, we next discuss the main idea applied to our case.

The cover decoder goes through the columns of $M$ and decides whether the corresponding item is defective or not. This results in decoding complexity $O(tN)$. Assume we were given a superset $S^{\prime}$ such that $S^{\prime}$ is guaranteed to include the defective set $S$, i.e. $S\subseteq S^{\prime}$, then the cover decoder could run in time $O(t\cdot|S^{\prime}|)$ over the columns corresponding to $S^{\prime}$, which depending on the size of $S^{\prime}$ could result in significantly lower complexity. It turns out that we can construct this small set $S^{\prime}$ recursively.

Suppose that we have access to an efficiently decodable $t_{1}(d,\sqrt{N},\epsilon/4,p)\times\sqrt{N}$ matrix $M^{(1)}$ which can be used to detect at most $d$ defectives among $\sqrt{N}$ items with probability of error $P_{e}\leq{\epsilon/4}$ when the noise parameter is $p$ by using $t_{1}({d},\sqrt{N},{\epsilon/4},{p})$ tests. We construct two $t_{1}(d,\sqrt{N},\epsilon/4,p)\times N$ matrices $M^{(F)}$ and $M^{(L)}$ using $M^{(1)}$ as follows. For $i\in[N]$, the $i$’th column of $M^{(F)}$ is equal to $j$’th column of $M^{(1)}$ if the first $\frac{1}{2}\log N$ bits in the binary representation of $i$ are given by the binary representation of $j$ for $j\in[\sqrt{N}]$. Similarly, for $i\in[N]$, the $i$’th columns of $M^{(L)}$ is equal to the $j$’th column of $M^{(1)}$ if the last $\frac{1}{2}\log N$ bits in the binary representation of $i$ are given by the binary representation of $j$ for $j\in[\sqrt{N}]$.

The final matrix matrix $M$ is constructed by vertically stacking $M^{(F)}$, $M^{(L)}$ and a $t_{2}(d,N,\epsilon/2,p)\times N$ matrix $M^{(2)}$ which is not necessarily efficiently decodable (e.g., the Kautz-Singleton construction). As before, $t_{2}(d,N,\epsilon/2,p)$ is the number of tests for $M^{(2)}$, which we assume can be used to detect $d$ defectives among $N$ items with probability of error $P_{e}\leq{\epsilon/2}$ when the noise parameter is $p$. The decoding works as follows. We obtain the measurement vectors $Y^{(F)}$, $Y^{(L)}$, and $Y^{(2)}$ given by $Y^{(F)}=\bigvee_{i\in S}M^{(F)}_{i}\oplus v^{(F)}$, $Y^{(L)}=\bigvee_{i\in S}M^{(L)}_{i}\oplus v^{(L)}$, and $Y^{(2)}=\bigvee_{i\in S}M^{(2)}_{i}\oplus v^{(2)}$ respectively where $v^{(F)}$, $v^{(L)}$, and $v^{(2)}$ are the noise terms corrupting the corresponding measurements. We next apply the decoding algorithm for $M^{(1)}$ to $Y^{(F)}$ and $Y^{(L)}$ to obtain the estimate sets $\hat{S}^{(F)}$ and $\hat{S}^{(L)}$ respectively. Note that the sets $\hat{S}^{(F)}$ and $\hat{S}^{(L)}$ can decode the first and last $\frac{1}{2}\log N$-bits of the defective items respectively with probability at least $1-\epsilon/2$ by the union bound. Therefore, we can construct the set $S^{\prime}=\hat{S}^{(F)}\times\hat{S}^{(L)}$ where $\times$ denotes the Cartesian product and obtain a super set that contains the defective set $S$ with error probability at most $\epsilon/2$. We further note that since $|\hat{S}^{(F)}|\leq d$ and $|\hat{S}^{(L)}|\leq d$, we have $|S^{\prime}|\leq d^{2}$. We finally apply the naive cover decoder to $M^{(2)}$ by running it over the set $S^{\prime}$ to compute the final estimate $\hat{S}$ which can be done in additional $O(t_{2}\cdot d^{2})$ time. Note that by the union bound the probability of error is bounded by $\epsilon$. Figure 4 illustrates the main idea with the example of $d=2$ and $N=16$. We provide this decoding scheme in Algorithm 1 for the special case $N=d^{2^{i}}$ for some non-negative integer $i$ although the results hold in the general case and no extra assumption beyond $d=\Omega(\log^{2}N)$ is needed. The next theorem is the result of applying this idea recursively.

The proof of the above theorem can be found in Appendix -C. We note that with only $\log\log_{d}N$ extra factor in the number of tests, the decoding complexity can be brought to the desired $O(\textnormal{poly}(t))$ complexity. We further note that the number of tests becomes order-optimal in the regime $d=\Theta(N^{\alpha})$ for some $\alpha\in(0,1)$. In Table I we provide the results presented in this work along with the related results in the literature.

The literature on the non-adaptive group testing framework includes both explicit and random test designs. We refer the reader to [12] for a survey. In combinatorial group testing, the famous construction introduced by Kautz and Singleton [23] achieves $t=O(d^{2}\log_{d}^{2}N)$ tests matching the best known lower bound $\min\{N,\Omega(d^{2}\log_{d}N)\}$ [21, 22] in the regime where $d=\Theta(N^{\alpha})$ for some $\alpha\in(0,1)$. However, this strongly explicit construction is suboptimal in the regime where $d=O(\textnormal{poly}(\log N))$. An explicit construction achieving $t=O(d^{2}\log N)$ was introduced by Porat and Rothschild in [24]. While $t=O(d^{2}\log N)$ is the best known achievability result in combinatorial group testing framework, there is no strongly explicit construction matching it to the best of our knowledge. Regarding efficient decoding, recently Indyk, Ngo and Rudra [34] introduced a randomized construction with $t=O(d^{2}\log(N))$ tests that could be decoded in time $\textnormal{poly}(t)$. Furthermore, the construction in [34] can be derandomized in the regime $d=O(\log N/\log\log N)$. Later, Ngo, Porat and Rudra [26] removed the constraint on $d$ and provided an explicit construction that can be decoded in time $\textnormal{poly}(t)$. The main idea of [34] was to consider list-disjunct matrices; a similar idea was considered by Cheraghchi in [33], which obtained explicit constructions of non-adaptive group testing schemes that handle noisy tests and return a list of defectives that may include false positives.

There are various schemes relaxing the zero-error criteria in the group testing problem. For instance, the model mentioned above, where the decoder always outputs a small super-set of the defective items, was studied in [33, 38, 39, 40]. These constructions have efficient ($\textnormal{poly}(t)$-time) decoding algorithms, and so can be used alongside constructions without sublinear time decoding algorithms to speed up decoding. Another framework where the goal is to recover at least a $(1-\epsilon)$-fraction (for any arbitrarily small $\epsilon>0$) of the defective set with high probability was studied in [35] where the authors provided a scheme with order-optimal $O(d\log N)$ tests and the computational complexity. There are also different versions of the group testing problem in which a test can have more than two outcomes [41, 42] or can be threshold based [43, 44, 45]. More recently, sparse group testing frameworks for both combinatorial and probabilistic settings were studied in [46, 47, 48].

When the defective set is assumed to be uniformly random, it is known that $t=\Theta(d\log N)$ is order-optimal for achieving the exact recovery of the defective set with vanishing probability of error (which is the model considered in this work) in the broad regime $d=O(N^{\alpha})$ for some $\alpha\in(0,1)$ using random designs and information-theoretical tools [37, 16]. These results also include the noisy variants of the group testing problem. Efficient recovery algorithms with nearly optimal number of tests were introduced recently in [35] and [36]. Regarding deterministic constructions of almost disjunct matrices, recently Mazumdar [25] introduced an analysis connecting the group testing properties with the average Hamming distance between the columns of the measurement matrix and obtained (strongly) explicit constructions with $t=O(d\log^{2}N/\log d)$ tests. While this result is order-optimal in the regime where $d=\Theta(N^{\alpha})$ for some $\alpha\in(0,1)$, it is suboptimal for moderate values of $d$ (e.g., $d=O(\textnormal{poly}(\log N))$). The performance of the Kautz-Singleton construction in the random model has been studied empirically [32], but we are not aware of any theoretical analysis of it beyond what follows immediately from the distance of Reed-Solomon codes. To the best of our knowledge there is no known explicit/strongly explicit construction achieving $t=\Theta(d\log N)$ tests in general for the noiseless/noisy version of the probabilistic group testing problem.

In this work, we showed that the Kautz-Singleton construction is order-optimal in the noiseless and noisy variants of the probabilistic group testing problem. To the best of our knowledge, this is the first (strongly) explicit construction achieving order-optimal number of tests in the probabilistic group testing setting for poly-logarithmic (in $N$) values of $d$. We provided a novel analysis departing from the classical approaches in the literature that use combinatorial properties of the underlying code. We instead directly explored the probability of a non-defective item being covered by a random defective set using the properties of Reed-Solomon codes in our analysis. Furthermore, by using a recursive technique, we converted the Kautz-Singleton construction into a construction that is also efficiently decodable with only a log-log factor increase in number of tests which provides interesting tradeoffs compared to the existing results in the literature.

There are a number of nontrivial extensions to our work. Firstly, it would be interesting to extend these results to the regime $d=o(\log^{2}N)$. Another interesting line of work would be to find a deterministic/randomized construction achieving order-optimal $t=\Theta(d\log N)$ tests and is also efficiently decodable.