A Local Approach to Studying the Time and Space Complexity of Deterministic and Nondeterministic Decision Trees

Decision trees and decision rules are intensively studied and used in different areas of computer science. The questions important for the theory of decision trees and rules include relations between decision trees and decision rule systems, time-space tradeoff for decision trees, and time-space tradeoff for decision rule systems.

In this paper, instead of decision rule systems we study nondeterministic decision trees. These trees can be considered as representations of decision rule systems that sometimes have less space complexity than the original rule systems. We study problems over infinite binary information systems and divide the set of all infinite binary information systems into three complexity classes depending on the worst case time and space complexity of deterministic and nondeterministic decision trees solving problems. This allows us to identify nontrivial relationships between deterministic decision trees and decision rule systems represented by nondeterministic decision trees. For each complexity class, we study issues related to time-space trade-off for deterministic and nondeterministic decision trees.

Decision trees [1, 2, 6, 22, 26, 32] and systems of decision rules [4, 5, 7, 11, 25, 29, 30, 31, 35] are widely used as classifiers to predict a decision for a new object, as a means of knowledge representation, and as algorithms for solving problems of fault diagnosis, computational geometry, combinatorial optimization, etc.

Decision trees and rules are among the most interpretable models for classifying and representing knowledge [15]. In order to better understand decision trees, we should not only minimize the number of their nodes, but also the depth of decision trees to avoid the consideration of long conjunctions of conditions corresponding to long paths in these trees. Similarly, for decision rule systems, we should minimize both the total length of rules and the maximum length of a rule in the system. When we consider decision trees and decision rule systems as algorithms (usually sequential for decision trees and parallel for decision rule systems), we should have in mind the same bi-criteria optimization problems to minimize space and time complexity of these algorithms.

In this paper, we represent systems of decision rules as nondeterministic decision trees to compress them and to emphasize the possibility of processing different decision rules in parallel. We consider deterministic and nondeterministic decision trees as algorithms and study their space and time complexity, paying particular attention to time and space complexity relationships. Examples of deterministic and nondeterministic decision trees computing Boolean function $x_{1}\wedge x_{2}$ can be found in Fig. 1.

Infinite systems of attributes and decision trees over these systems have been intensively studied, especially systems of linear and algebraic attributes and the corresponding linear [8, 9, 16] and algebraic decision trees [3, 12, 13, 14, 36, 37, 38]. Years ago, one of the authors initiated the study of decision trees over arbitrary infinite systems of attributes [17, 18, 19, 20, 21]. In this paper, we study decision trees over arbitrary infinite systems of binary attributes represented in the form of infinite binary information systems.

General information system introduced by Pawlak [28] consists of a universe (a set of objects) and a set of attributes (functions with finite image) defined on the universe. An information system is called infinite, if both its universe and the set of attributes are infinite. An information system is called binary if each of its attributes has values from the set $\{0,1\}$.

Any problem over an information system is described by a finite number of attributes that divide the universe into domains in which these attributes have fixed values. A decision is attached to each domain. For a given object from the universe, it is required to find the decision attached to the domain containing this object.

As algorithms solving these problems, deterministic and nondeterministic decision trees are studied. As time complexity of a decision tree, we consider its depth, i.e., the maximum number of nodes labeled with attributes in a path from the root to a terminal node. As space complexity of a decision tree, we consider the number of its nodes.

There are two approaches to the study of infinite information systems: local when in decision trees solving a problem we can use only attributes from the problem description, and global when in the decision trees solving a problem we can use arbitrary attributes from the considered information system. In this paper, we study decision trees in the framework of the local approach.

To the best of our knowledge, time-space trade-offs for decision trees over infinite information systems were not studied in the framework of the local approach prior to the present paper except for its conference version [10], which does not contain proofs. The paper [24] was the first one in which the time-space trade-offs for decision trees over infinite information systems were studied in the framework of the global approach.

Results obtained in [24] are different from the results obtained in the present paper. Apart from the difference in approach, in [24], the set of all infinite information systems is divided not into three but into five families and the criteria for the behavior of functions characterizing the minimum depth of the deterministic and nondeterministic decision tree are completely different in comparison with the present paper. However, many of the definitions and results of the two papers appear similar. In the present paper, we use some auxiliary statements proved in [24] and adapt some proofs from [24] to the case of the local approach.

Based on the results obtained in the present paper and in [22, 26], we describe possible types of behavior of four functions $h_{U}^{ld},h_{U}^{la},L_{U}^{ld},L_{U}^{la}$ that characterize worst case time and space complexity of deterministic and nondeterministic decision trees over an infinite binary information system $U$ (index $l$ refers to the local approach). Decision trees solving a problem can use only attributes from the problem description.

The function $h_{U}^{ld}$ characterizes the growth in the worst case of the minimum depth of a deterministic decision tree solving a problem with the growth of the number of attributes in the problem description. The function $h_{U}^{ld}$ is either grows as a logarithm or linearly.

The function $h_{U}^{la}$ characterizes the growth in the worst case of the minimum depth of a nondeterministic decision tree solving a problem with the growth of the number of attributes in the problem description. The function $h_{U}^{la}$ is either bounded from above by a constant or grows linearly.

The function $L_{U}^{ld}$ characterizes the growth in the worst case of the minimum number of nodes in a deterministic decision tree solving a problem with the growth of the number of attributes in the problem description. The function $L_{U}^{ld}$ has either polynomial or exponential growth.

The function $L_{U}^{la}$ characterizes the growth in the worst case of the minimum number of nodes in a nondeterministic decision tree solving a problem with the growth of the number of attributes in the problem description. The function $L_{U}^{la}$ has either polynomial or exponential growth.

Each of the functions $h_{U}^{ld},h_{U}^{la},L_{U}^{ld},L_{U}^{la}$ has two types of behavior. The tuple $(h_{U}^{ld},h_{U}^{la},L_{U}^{ld},L_{U}^{la})$ has three types of behavior. All these types are described in the paper and each type is illustrated by an example.

There are three complexity classes of infinite binary information systems corresponding to the three possible types of the tuple $(h_{U}^{ld},h_{U}^{la},L_{U}^{ld},L_{U}^{la})$. For each class, we study joint behavior of time and space complexity of decision trees. The obtained results are related to time-space trade-off for deterministic and nondeterministic decision trees.

A pair of functions $(\varphi,\psi)$ is called a boundary $ld$-pair of the information system $U$ if, for any problem over $U$, there exists a deterministic decision tree over $z$, which solves this problem and for which the depth is at most $\varphi(n)$ and the number of nodes is at most $\psi(n)$, where $n$ is the number of attributes in the problem description. An information system $U$ is called $ld$-reachable if the pair $(h_{U}^{ld},L_{U}^{ld})$ is a boundary $ld$-pair of the system $U$. For nondeterministic decision trees, the notions of a boundary $la$-pair of an information system and $la$-reachable information system are defined in a similar way. For deterministic decision trees, the best situation is when the considered information system is $ld$-reachable: for any boundary $ld$-pair $(\varphi,\psi)$ for an information system $U$ and any natural $n$, $\varphi(n)\geq h_{U}^{ld}(n)$ and $\psi(n)\geq L_{U}^{ld}(n)$. For nondeterministic decision trees, the best situation is when the information system is $la$-reachable.

For all complexity classes, all information systems from the class are $ld$-reachable. For two out of the three complexity classes, all information systems from the class are $la$-reachable. For the remaining class, all information systems from the class are not $la$-reachable. For all information systems $U$ that are not $la$-reachable, we find nontrivial boundary $la$-pairs, which are sufficiently close to $(h_{U}^{la},L_{U}^{la})$.

The rest of the paper is organized as follows: Section 2 contains main results, Sections 3-5 – proofs of these results, and Section 6 – short conclusions.

Let $A$ be an infinite set and $F$ be an infinite set of functions that are defined on $A$ and have values from the set $\{0,1\}$. The pair $U=(A,F)$ is called an infinite binary information system [28], the elements of the set $A$ are called objects, and the functions from $F$ are called attributes. The set $A$ is called sometimes the universe of the information system $U$.

A problem over $U$ is a tuple of the form $z=(\nu,f_{1},\ldots,f_{n})$, where $\nu:\{0,1\}^{n}\rightarrow\mathbb{N}$, $\mathbb{N}$ is the set of natural numbers $\{1,2,\ldots\}$, and $f_{1},\ldots,f_{n}\in F$. We do not require attributes $f_{1},\ldots,f_{n}$ to be pairwise distinct. The problem $z$ consists in finding the value of the function $z(x)=\nu(f_{1}(x),\ldots,f_{n}(x))$ for a given object $a\in A$. The value $\dim z=n$ is called the dimension of the problem $z$.

Various problems of combinatorial optimization, pattern recognition, fault diagnosis, probabilistic reasoning, computational geometry, etc., can be represented in this form.

As algorithms for problem solving we consider decision trees. A decision tree over the information system $U$ is a directed tree with a root in which the root and edges leaving the root are not labeled, each terminal node is labeled with a number from $\mathbb{N}$, each working node (which is neither the root nor a terminal node) is labeled with an attribute from $F$, and each edge leaving a working node is labeled with a number from the set $\{0,1\}$. A decision tree is called deterministic if only one edge leaves the root and edges leaving an arbitrary working node are labeled with different numbers.

Let $\Gamma$ be a decision tree over $U$ and

be a directed path from the root $v_{0}$ to a terminal node $v_{m+1}$ of $\Gamma$ (we call such path complete). Define a subset $A(\xi)$ of the set $A$ as follows. If $m=0$, then $A(\xi)=A$. Let $m>0$ and, for $i=1,\ldots,m$, the node $v_{i}$ be labeled with the attribute $f_{j_{i}}$ and the edge $d_{i}$ be labeled with the number $\delta_{i}$. Then

The depth of the decision tree $\Gamma$ is the maximum number of working nodes in a complete path of $\Gamma$. Denote by $h(\Gamma)$ the depth of $\Gamma$ and by $L(\Gamma)$ – the number of nodes in $\Gamma$.

A decision tree over the information system $U$ is called a decision tree over the problem $z=(\nu,f_{1},\ldots,f_{n})$ if each working node of $\Gamma$ is labeled with an attribute from the set $\{f_{1},\ldots,f_{n}\}$.

The decision tree $\Gamma$ over $z$ solves the problem $z$ nondeterministically if, for any object $a\in A$, there exists a complete path $\xi$ of $\Gamma$ such that $a\in A(\xi)$ and, for each $a\in A$ and each complete path $\xi$ such that $a\in A(\xi)$, the terminal node of $\xi$ is labeled with the number $z(a)$ (in this case, we can say that $\Gamma$ is a nondeterministic decision tree solving the problem $z$). In particular, if the decision tree $\Gamma$ solves the problem $z$ nondeterministically, then, for each complete path $\xi$ of $\Gamma$, either the set $A(\xi)$ is empty or the function $z(x)$ is constant on the set $A(\xi)$. The decision tree $\Gamma$ over $z$ solves the problem $z$ deterministically if $\Gamma$ is a deterministic decision tree, which solves the problem $z$ nondeterministically (in this case, we can say that $\Gamma$ is a deterministic decision tree solving the problem $z$).

Let $P(U)$ be the set of all problems over $U$. For a problem $z$ from $P(U)$, let $h_{U}^{ld}(z)$ be the minimum depth of a decision tree over $z$ solving the problem $z$ deterministically, $h_{U}^{la}(z)$ be the minimum depth of a decision tree over $z$ solving the problem $z$ nondeterministically, $L_{U}^{ld}(z)$ be the minimum number of nodes in a decision tree over $z$ solving the problem $z$ deterministically, and $L_{U}^{la}(z)$ be the minimum number of nodes in a decision tree over $z$ solving the problem $z$ nondeterministically.

We consider four functions defined on the set $\mathbb{N}$ in the following way: $h_{U}^{ld}(n)=\max$ $h_{U}^{ld}(z)$, $h_{U}^{la}(n)=\max$ $h_{U}^{la}(z)$, $L_{U}^{ld}(n)=\max$ $L_{U}^{ld}(z)$, and $L_{U}^{la}(n)=\max$ $L_{U}^{la}(z)$, where the maximum is taken among all problems $z$ over $U$ with $\dim z\leq n$. These functions describe how the minimum depth and the minimum number of nodes of deterministic and nondeterministic decision trees solving problems are growing in the worst case with the growth of problem dimension. To describe possible types of behavior of these four functions, we need to define some properties of infinite binary information systems.

We now consider two examples of infinite binary information systems that satisfy the condition of reduction. These examples are close to ones considered in Section 3.4 of the book [1].

We now consider examples of infinite binary information systems with finite I-dimension and with infinite I-dimension. More examples can be found in Lemmas 7-9.

We now consider four statements that describe possible types of behavior of functions $h_{U}^{ld}(n)$, $h_{U}^{la}(n)$, $L_{U}^{ld}(n)$, and $L_{U}^{la}(n)$. The next statement follows immediately from Theorem 4.3 [22].

The next statement follows immediately from Theorem 8.2 [26].

Let $U$ be an infinite binary information system. Proposition 1 allows us to correspond to the function $h_{U}^{ld}(n)$ its type of behavior from the set $\{\mathrm{LOG},\mathrm{LIN}\}$. Proposition 2 allows us to correspond to the function $h_{U}^{la}(n)$ its type of behavior from the set $\{\mathrm{CON},\mathrm{LIN}\}$. Propositions 3 and 4 allow us to correspond to each of the functions $L_{U}^{ld}(n)$ and $L_{U}^{la}(n)$ its type of behavior from the set $\{\mathrm{POL},\mathrm{EXP}\}$. A tuple obtained from the tuple

by replacing functions with their types of behavior is called the local type of the information system $U$. We now describe all possible local types of infinite binary information systems.

For $i=1,2,3$, we denote by $W_{i}^{l}$ the class of all infinite binary information systems, whose local type coincides with the $i$th row of Table 1. We now study for each of these complexity classes joint behavior of the depth and number of nodes in decision trees solving problems.

For a given infinite binary information system $U$, we will consider pairs of functions $(\varphi,\psi)$ such that, for any problem $z$ over $U$, there exists a deterministic decision tree over $z$ solving $z$ with the depth at most $\varphi(\dim z)$ and the number of nodes at most $\psi(\dim z)$. We will study such pairs and will call them boundary $ld$-pairs.

We are interested in finding boundary $ld$-pairs with functions that grow as slowly as possible. It is clear that, for any boundary $ld$-pair $(\varphi,\psi)$ of the information system $U$, the following inequalities hold: $\varphi(n)\geq h_{U}^{ld}(n)$ and $\psi(n)\geq L_{U}^{ld}(n)$. So the best possible situation is when $(h_{U}^{ld},L_{U}^{ld})$ is a boundary $ld$-pair of $U$.

We now consider similar notions for nondeterministic decision trees: the notion of boundary $la$-pair and the notion of $la$-reachable information system.

Note that for deterministic decision trees, the best situation is when the considered information system is $ld$-reachable and for nondeterministic decision trees – when the information system is $la$-reachable.

Each information system from the classes $W_{1}^{l},W_{2}^{l}$, and $W_{3}^{l}$ is $ld$-reachable. Each information system from the classes $W_{2}^{l}$ and $W_{3}^{l}$ is $la$-reachable. Each information system from the class $W_{1}^{l}$ is not $la$-reachable. For all information systems $U$, which are not $la$-reachable, we find nontrivial boundary $la$-pairs that are sufficiently close to $(h_{U}^{la},L_{U}^{la})$.

The obtained results are related to time-space trade-off for deterministic and nondeterministic decision trees. Details can be found in the following three theorems.

Table 2 summarizes Theorems 1-4. The first column contains the name of the complexity class. The next four columns describe the local type of information systems from this class. The last two columns “$ld$-pairs” and “$la$-pairs” contain information about boundary $ld$-pairs and boundary $la$-pairs for information systems from the considered class: “$ld$-reachable” means that all information systems from the class are $ld$-reachable, “$la$-reachable” means that all information systems from the class are $la$-reachable, Th. 2 (b) is a link to the corresponding statement Theorem 2 (b).

In this section, we consider a number of auxiliary statements and prove the two mentioned propositions.

Let $\Gamma$ be a decision tree over an infinite binary information system $U=(A,F)$ and $d$ be an edge of $\Gamma$ entering a node $w$. We denote by $\Gamma(d)$ a subtree of $\Gamma$, whose root is the node $w$. We say that a complete path $\xi$ of $\Gamma$ is realizable if $A(\xi)\neq\emptyset$.

Let $U$ be an infinite binary information system, $\Gamma$ be a decision tree over $U$, and $d$ be an edge of $\Gamma$. The subtree $\Gamma(d)$ will be called full if there exist edges $d_{1},\ldots,d_{m}$ in $\Gamma(d)$ such that the removal of these edges and subtrees $\Gamma(d_{1}),\ldots,\Gamma(d_{m})$ transforms the subtree $\Gamma(d)$ into a tree $G$ such that each terminal node of $G$ is a terminal node of $\Gamma$, and exactly two edges labeled with the numbers $0$ and $1$ respectively leave each working node of $G$.

We now consider two statements about classes of decision trees proved in [24]. Let $\Gamma$ be a decision tree. We denote by $L_{t}(\Gamma)$ the number of terminal nodes in $\Gamma$ and by $L_{w}(\Gamma)$ the number of working nodes in $\Gamma$. It is clear that $L(\Gamma)=1+L_{t}(\Gamma)+L_{w}(\Gamma)$.

Let $U$ be an infinite binary information systems. We denote by $G_{d}(U)$ the set of all deterministic decision trees over $U$ and by $G_{d}^{2}(U)$ the set of all decision trees from $G_{d}(U)$ such that each working node of the tree has two leaving edges.

We denote by $G_{a}^{f}(U)$ the set of all decision trees $\Gamma$ over $U$ that satisfy the following conditions: (i) if a working node of $\Gamma$ has $m$ leaving edges $d_{1},\ldots,d_{m}$ labeled with the same number and $m\geq 2$, then the subtrees $\Gamma(d_{1}),\ldots,\Gamma(d_{m})$ are not full, and (ii) if the root of $\Gamma$ has $m$ leaving edges $d_{1},\ldots,d_{m}$ and $m\geq 2$, then the subtrees $\Gamma(d_{1}),\ldots,\Gamma(d_{m})$ are not full. One can show that $G_{d}^{2}(U)\subseteq G_{d}(U)\subseteq G_{a}^{f}(U)$.

Let $U=(A,F)$ be an infinite binary information system. For $f_{1},\ldots,f_{n}\in F$, we denote by $N_{U}(f_{1},\ldots,f_{n})$ the number of $n$-tuples $(\delta_{1},\ldots,\delta_{n})\in\{0,1\}^{n}$ for which the system of equations

has a solution from $A$. For $n\in\mathbb{N}$, denote

It is clear that, for any $m,n\in\mathbb{N}$, if $m\leq n$ then $N_{U}(m)\leq N_{U}(n)$.

The next statement follows directly from Lemmas 5.1 and 5.2 [22] and the evident inequality $N_{U}(n)\leq 2^{n}$, which is true for any infinite binary information system $U$. The proof of Lemma 5.1 from [22] is based on Theorems 4.6 and 4.7 from the same monograph that are similar to results obtained in [33, 34].

We now prove Propositions 3 and 4.

First, we prove five auxiliary statements.

For each row of Table 1, we consider an example of infinite binary information system, whose local type coincides with this row.

For any $i\in\mathbb{N}$, we define two functions $p_{i}:\mathbb{N}\rightarrow\{0,1\}$ and $l_{i}:\mathbb{N}\rightarrow\{0,1\}$. Let $j\in\mathbb{N}$. Then $p_{i}(j)=1$ if and only if $j=i$, and $l_{i}(j)=1$ if and only if $j>i$.

Define an information system $U_{1}=(A_{1},F_{1})$ as follows: $A_{1}=\mathbb{N}$ and $F_{1}=\{l_{i}:i\in\mathbb{N}\}$.