Accurate Summary-based Cardinality Estimation Through the Lens of Cardinality Estimation Graphs

We begin by giving an overview of the Markov tables estimator (Aboulnaga et al., 2001), which was used to estimate the cardinalities of paths in XML documents. A Markov table of length $h\geq 2$ stores the cardinality of each path in an XML document’s element tree up to length $h$ and uses these to make predications for the cardinalities of longer paths. Table 1 shows a subset of the entries in an example Markov table for $h=2$ for our running example dataset shown in Figure 2. The formula to estimate a 3-path using a Markov table with $h=2$ is to multiply the cardinality of the leftmost 2-path with the consecutive 2-path divided by the cardinality of the common edge. For example, consider the query $Q_{3p}=\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}$ against the dataset in Figure 2. The formula for $Q_{3p}$ would be: $|\xrightarrow{A}\xrightarrow{B}|\times(|\xrightarrow{B}\xrightarrow{C}|/|\xrightarrow{B}|)$. Observe that this formula is inspired by the Bayesian probability rule that $Pr(ABC)=Pr(AB)Pr(C|AB)$ but makes a conditional independence assumption between $A$ and $C$, in which case the Bayesian formula would simplify to $Pr(ABC)=Pr(AB)Pr(C|B)$. For $Pr(AB)$ the formula uses the true cardinality $|\xrightarrow{A}\xrightarrow{B}$$|$. For $Pr(C|B)$ the formula makes a uniformity assumption that the number of $C$ edges that each $B$ edge extends to is equal for each $B$ edge and is $r=|\xrightarrow{B}\xrightarrow{C}|/|\xrightarrow{B}|$. Equivalently, this can be seen as an “average degree” assumption that on average the $C$-degree of nodes in the $\xrightarrow{B}\xrightarrow{C}$ paths is $r$. The result of this formula is $4\times\frac{3}{2}=6$, which underestimates the true cardinality of 7. The graph summaries (Maduko et al., 2008) for RDF databases and the graph catalogue estimator (Mhedhbi and Salihoglu, 2019) for property graphs have extended the contents of what is stored in Markov tables, respectively, to other acyclic joins, such as stars, and small cycles, such as triangles, but use the same uniformity and conditional independence assumptions.

We next represent such estimators using a CEG that we call $CEG_{O}$. This will help us describe the space of possible estimations that can be made with these estimators. We assume that the given query $Q$ is connected. $CEG_{O}$ consists of the following:

• $\bullet$

  Vertices: For each connected subset of relations $S\subseteq\mathcal{R}$ of $Q$, we have a vertex in $CEG_{O}$ with label $S$. This represents the sub-query $\bowtie_{R_{i}\in S}R_{i}$.

• $\bullet$

  Edges: Consider two vertices with labels $S$ and $S^{\prime}$ s.t., $S\subset S^{\prime}$. Let $\mathcal{D}$, for difference be $S^{\prime}\setminus S$, and let $\mathcal{E}\supset\mathcal{D}$, for extension be a join query in the Markov table, and let $\mathcal{I}$, for intersection, be $\mathcal{E}\cap S$. If $\mathcal{E}$ and $\mathcal{I}$ exist in the Markov table, then there is an edge with weight $\frac{|\mathcal{E}|}{|\mathcal{I}|}$ from $S$ to $S^{\prime}$ in $CEG_{O}$.

When making estimates, we will apply two basic rules from prior work that limit the paths considered in $CEG_{O}$. First is that if the Markov table contains size-$h$ joins, the formulas use size $h$ joins in the numerators in the formula. For example, if $h=3$, we do not try to estimate the cardinality of a sub-query $\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}$ by a formula $\xrightarrow{A}\xrightarrow{B}\times\frac{\xrightarrow{B}\xrightarrow{C}}{\xrightarrow{B}}$ because we store the true cardinality of $\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}$ in the Markov table. Second, for cyclic graph queries, which was covered in reference (Mhedhbi and Salihoglu, 2019), an additional early cycle closing rule is used in the reference when generating formulas. In CEG formulation this translates to the rule that if $S$ can extend to multiple $S^{\prime}$s and some of them contain additional cycles that are not in $S$, then only such outgoing edges of $S$ to such $S^{\prime}$ are considered in finding paths.

Even when the previous rules are applied to limit the number of paths considered in a CEG, in general there may be multiple $(\emptyset,Q)$ paths that lead to different estimates. Consider the $CEG_{O}$ shown in Figure 4 which uses a Markov table of size 2. There are 36 $(\emptyset,Q)$ paths leading to 7 different estimates. Two examples are:

• $\bullet$

  $|\xrightarrow{A}\xrightarrow{B}|\times\frac{|\xrightarrow{B}\xrightarrow{C}|}{|\xrightarrow{B}|}\times\frac{|\xrightarrow{B}\xrightarrow{D}|}{|\xrightarrow{B}|}\times\frac{|\xrightarrow{B}\xrightarrow{E}|}{|\xrightarrow{B}|}=52.5$

• $\bullet$

  $|\xrightarrow{A}\xrightarrow{B}|\times\frac{|\xrightarrow{B}\xrightarrow{C}|}{|\xrightarrow{B}|}\times\frac{|\xleftarrow{C}\xrightarrow{D}|}{|\xrightarrow{C}|}\times\frac{|\xleftarrow{D}\xrightarrow{E}|}{|\xrightarrow{D}|}=57.6$

Similarly, consider the fork query $Q_{5f}$ in Figure 1 and a Markov table with up to 3-size joins. The $CEG_{O}$ of $Q_{5f}$ is shown in Figure 3, which contains multiple paths leading to 2 different estimates:

• $\bullet$

  $|\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}|\times\frac{|\xleftarrow{C}\ext@arrow 0359\arrowfill@\mathrel{\mathchoice{\raise 1.16336pt\hbox to0.0pt{$\displaystyle\relbar$\hss}\lower 1.16336pt\hbox{$\displaystyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\textstyle\relbar$\hss}\lower 1.16336pt\hbox{$\textstyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\scriptstyle\relbar$\hss}\lower 1.16336pt\hbox{$\scriptstyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\scriptscriptstyle\relbar$\hss}\lower 1.16336pt\hbox{$\scriptscriptstyle\relbar$}}}\mathrel{\mathchoice{\raise 1.16336pt\hbox to0.0pt{$\displaystyle\relbar$\hss}\lower 1.16336pt\hbox{$\displaystyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\textstyle\relbar$\hss}\lower 1.16336pt\hbox{$\textstyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\scriptstyle\relbar$\hss}\lower 1.16336pt\hbox{$\scriptstyle\relbar$}}{\raise 1.16336pt\hbox to0.0pt{$\scriptscriptstyle\relbar$\hss}\lower 1.16336pt\hbox{$\scriptscriptstyle\relbar$}}}\rightrightarrows{E}{D}|}{|\xrightarrow{C}|}$

• $\bullet$

  $|\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}|\times\frac{|\xrightarrow{A}\xrightarrow{B}\xrightarrow{D}|}{|\xrightarrow{A}\xrightarrow{B}|}\times\frac{|\xrightarrow{A}\xrightarrow{B}\xrightarrow{E}|}{|\xrightarrow{A}\xrightarrow{B}|}$

Both formulas start by using $|\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}|$. Then, the first “short-hop” formula makes one fewer conditional independence assumption than the “long-hop” formula, which is an advantage. In contrast, the first estimate also makes a uniformity assumption that conditions on a smaller-size join. We can expect this assumption to be less accurate than the two assumptions made in the long-hop estimate, which condition on 2-size joins. In general, these two formulas can lead to different estimates.

For many queries, there can be many more than 2 different estimates. Therefore, any optimistic estimator implementation needs to make choices about which formulas to use, which corresponds to picking paths in $CEG_{O}$. Prior optimistic estimators have either left these choices unspecified or described procedures that implicitly pick a specific path yet without acknowledging possible other choices or empirically justifying their choice. Instead, we systematically identify a space of choices that an optimistic estimator can make along two parameters that also capture the choices made in prior work:

• $\bullet$

  Path length: The estimator can identify a set of paths to consider based on the path lengths, i.e., number of edges or hops, in $CEG_{O}$, which can be: (i) maximum-hop (max-hop); (ii) minimum-hop (min-hop); or (iii) any number of hops (all-hops).

• $\bullet$

  Estimate aggregator: Among the set of paths that are considered, each path gives an estimate. The estimator then has to aggregate these estimates to derive a final estimate, for which we identify three heuristics: (i) the largest estimated cardinality path (max-aggr); (ii) the lowest estimated cardinality path (min-aggr); or (iii) the average of the estimates among all paths (avg-aggr).

Recall that a Markov table stores the cardinalities of patterns up to some size $h$. Given a Markov table with $h\geq 2$, optimistic estimators can produce estimates for any acyclic query with size larger than $h$. But what about large cyclic queries, i.e., cyclic queries with size larger than $h$?

Faced with a large cyclic query $Q$ , the optimistic estimators we have described do not actually produce estimates for $Q$. Instead, they produce an estimate for a similar acyclic $Q^{\prime}$ that includes all of $Q$’s edges but is not closed. To illustrate this, consider estimating a 4-cycle query in Figure 5 using a Markov table with $h$=$3$. Recall that the estimates of prior optimistic estimators are captured by paths in the $CEG_{O}$ for 4-cycle query, shown in Figure 6(a). Consider as an example, the left most path, which corresponds to the formula: $|\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}|\times|\xrightarrow{B}\xrightarrow{C}\xrightarrow{D}|/|\xrightarrow{B}\xrightarrow{C}|$. Note that this formula is in fact estimating a 4-path $\xrightarrow{A}\xrightarrow{B}\xrightarrow{C}\xrightarrow{D}$ rather than the 4-cycle shown in Figure 5. This is true for each path in $CEG_{O}$.

More generally, when queries contain cycles of length $>h$, $CEG_{O}$ breaks cycles in queries into paths. Although optimistic estimators generally tend to underestimate acyclic queries due to the independence assumptions they make, the situation is different for cyclic queries. Since there are often significantly more paths than cycles, estimates over $CEG_{O}$ can lead to highly inaccurate overestimates. We note that this problem does not exist if a query contains a cycle $C$ of length $>h$ that contains smaller cycles in them, such as a clique of size $h+1$, because the early cycle closing rule from Section 4.2 will avoid formulas that estimate $C$ as a sub-query (i.e., no $(\emptyset,C)$ sub-path will exist in CEG).

We next describe an alternative modified CEG to address this problem. Consider a query $Q$ with a $k$-cycle $C$ where $k>h$. Note that in order to not break cycles into paths, we need CEG edges whose weights capture the cycle closing effect when extending a sub-query $S$ that contains $k$$-$$1$ edges of $C$ to a sub-query $S^{\prime}$ that contains $C$. We capture this in a new CEG called $CEG_{OCR}$, for Optimistic Cycle closing Rate, which modifies $CEG_{O}$ as follows. We keep the same vertices as in $CEG_{O}$ and the same set of edges, except when we detect two vertices $S$ and $S^{\prime}$ with the above property. Then, instead of using the weights from the original $CEG_{O}$ between $S$ and $S^{\prime}$, we use pre-computed cycle closing probabilities. Suppose the last edge that closes the cycle $C$ is $E_{i}$ and it is between the query edges $E_{i-1}$ and $E_{i+1}$. In the Markov table, we store the probabilities of two connected $E_{i-1}$ and $E_{i+1}$ edges to be connected by an additional $E_{i}$ edge to close a cycle. We denote this statistic as $P(E_{i-1}*E_{i+1}|E_{i})$. We can compute $P(E_{i-1}*E_{i+1}|E_{i})$ by computing all paths that start from $E_{i-1}$ and end with $E_{i+1}$ of varying lengths and then counting the number of $E_{i}$ edges that close such paths to cycles. On many datasets there may be a prohibitively many such paths, so we can sample $p$ of such paths. Suppose these $p$ paths lead to $c$ many cycles, then we can take the probability as $c/p$. In our implementation we perform sampling through random walks that start from $E_{i-1}$ and end at $E_{i+1}$ but other sampling strategies can also be employed. Figure 6(b) provides the $CEG_{OCR}$ for the 4-cycle query in Figure 5. We note that the Markov table for $CEG_{OCR}$ requires computing additional $P(E_{i-1}*E_{i+1}|E_{i})$ statistics that $CEG_{O}$ does not require. The number of entries is at most $O(L^{3})$ where $L$ is the number of edge labels in the dataset. For many datasets, e.g., all of the ones we used in our evaluations, $L$ is small and in the order of 10s or 100s, so even in the worst case these entries can be stored in several MBs. In contrast, storing large cycles with $h$$>$$3$ edges could potentially require $\Theta(L^{h})$ more entries.

MOLP was defined in reference (Joglekar and Ré, 2018) as a tighter bound than the AGM bound that uses additional degree statistics about input relations that AGM bound does not use. We first review the formal notion of a degree. Let $\mathcal{X}$ be a subset of the attributes $\mathcal{A}_{i}$ of some relation $\mathcal{R}_{i}$, and let $v$ be a possible value of $\mathcal{X}$. The degree of $v$ in $\mathcal{R}_{i}$ is the number of times $v$ occurs in $\mathcal{R}_{i}$, i.e. $deg(\mathcal{X}(v),\mathcal{R}_{i})=|\{t\in\mathcal{R}_{i}|\pi_{\mathcal{X}}(t)=v\}|$. For example, in Figure 2, $deg(s(3),E)=3$ because the outgoing $E$-degree of vertex 3 is 3. Similarly $deg(d(2),A)$ is 1 because the incoming $A$-degree of vertex 2 is 1. We also define $deg(\mathcal{X},\mathcal{R}_{i})$ to be the maximum degree in $\mathcal{R}_{i}$ of any value $v$ over $X$, i.e., $deg(X,\mathcal{R}_{i})=\max_{v}deg(\mathcal{X}(v),\mathcal{R}_{i})$. So, $deg(d,A)=3$ because vertex 13 has 3 incoming $A$ edges, which is the maximum A-in-degree in the dataset. The notion of degree can be generalized to $deg(X(v),Y,\mathcal{R}_{i})$, which refers to the “degree of a value $v$ over attributes $X$ in $\pi_{Y}\mathcal{R}_{i}$”, which counts the number of times $v$ occurs in $\pi_{Y}(\mathcal{R}_{i})$. Similarly, we let $deg(X,Y,\mathcal{R}_{i})=\max_{v}deg(X(v),Y,\mathcal{R}_{i})$. Suppose a system has stored $deg(X,Y,\mathcal{R}_{i})$ statistics for each possible $\mathcal{R}_{i}$ and $X\subseteq Y\subseteq\mathcal{A}_{i}$. MOLP is:

The base of the logarithm can be any constant and we take it as 2. Let $m_{A}$ be the optimal value of MOLP. Reference (Joglekar and Ré, 2018) has shown that $2^{m_{A}}$ is an upper bound on the size of $Q$. For example, in our running example, the optimal value of these inequalities is 96, which is an overestimate of the true cardinality of 78. It is not easy to directly see the solution of the MOLP on our running example. However, we will next show that we can represent the MOLP bound as the cost of minimum-weight $(\emptyset,Q)$ path in a CEG that we call $CEG_{M}$.

With these interpretations we can now construct $CEG_{M}$.

• $\bullet$

  Vertices: For each $X\subseteq\mathcal{A}$ there is a vertex. This represents the subquery $\Pi_{X}Q$.

• $\bullet$

  Extension Edges: Add an edge with weight $\log(deg(X,Y,$ $\mathcal{R}_{i}))$ between any $W_{1}=X\cup E$ and $W_{2}=Y\cup E$, for which there is $s_{Y\cup E}\leq s_{X\cup E}+\log(deg(X,Y,\mathcal{R}_{i}))$ inequality. Note that there can be multiple edges between $W_{1}$ and $W_{2}$ corresponding to inequalities from different relations.

• $\bullet$

  Projection Edges: $\forall X\subseteq Y$, add an edge with weight 0 from $Y$ to $X$. These edges correspond to projection inequalities and intuitively indicate that, in the worst-case instances, $\Pi_{Y}Q$ is always as large as $\Pi_{X}Q$.

Figure 7 shows the $CEG_{M}$ of our running example. For simplicity, we use actual degrees instead of their logarithms as edge weights and omit the projection edges in the figure. Below we use $(\emptyset,\mathcal{A})$ instead of $(\emptyset,Q)$, to represent the bottom-to-top paths in $CEG_{M}$ because $\Pi_{\mathcal{A}}Q=Q$.

With this connection, readers can verify that the MOLP bound in our running example is 96 by inspecting the paths in Figure 7. In this CEG, the minimum-weight $(\emptyset,\mathcal{A})$ path has a length of 96 (specifically $\log_{2}(96)$), corresponding to the leftmost path in Figure 7. We make three observations.

We review the CBS estimator very briefly and refer the reader to reference (Cai et al., 2019) for details. CBS estimator has two subroutines Bound Formula Generator (BFG) and Feasible Coverage Generator (FCG) (Algorithms 1 and 2 in reference (Cai et al., 2019)) that, given a query $Q$ and the degree statistics about $Q$, generate a set of bounding formulas. A coverage is a mapping ($R_{j}$, $A_{j})$ of a subset of the relations in the query to attributes such that each $A_{j}\in\mathcal{A}$ appears in the mapping. A bounding formula is a multiplication of the known degree statistics that can be used as an upper bound on the size of a query. In Appendix B, we show using our CEG framework that in fact the MOLP bound is at least as tight as the CBS estimator on general acyclic queries and is exactly equal to the CBS estimator over acyclic queries over binary relations, which are the queries which reference (Cai et al., 2019) used. Therefore BFG and FCG can be seen as a method for solving the MOLP linear program on acyclic queries over binary relations, albeit in a brute force manner by enumerating all paths in $CEG_{M}$. We do this by showing that each path in $CEG_{M}$ corresponds to a bounding formula and vice versa. These observations allow us to connect two worst-case upper bounds from literature using CEGs. Henceforth, we do not differentiate between MOLP and the CBS estimator. It is important to note that a similar connection between MOLP and CBS cannot be established for cyclic queries. This is because, although not explicitly acknowledged in reference (Cai et al., 2019), on cyclic queries, the covers that FCG generates may not be safe, i.e., the output of BFG may not be a pessimistic output. We provide a counter example in Appendix C. In contrast, MOLP generates a pessimistic estimate for arbitrary, so both acyclic or cyclic, queries.

For all of our experiments, we use a single machine with two Intel E5-2670 at 2.6GHz CPUs, each with 8 physical and 16 logical cores, and 512 GB of RAM. We represent our datasets as labeled graphs and queries as edge-labeled subgraph queries but our datasets and queries can equivalently be represented as relational tables, one for each edge label, and SQL queries. We focused on edge-labeled queries for simplicity. Estimating queries with vertex labels can be done in a straightforward manner both for optimistic and pessimistic estimators e.g., by extending Markov table entries to have vertex labels as was done in reference (Mhedhbi and Salihoglu, 2019). We used a total of 6 real-world datasets, shown in Table 2, and 5 workloads on these datasets. Our dataset and workload combinations are as follows.

We then transformed the JOB workload (Leis et al., 2018) into equivalent subgraph queries on our transformed graph. We removed non-join predicates in the queries since we are focusing on join cardinality estimations, and we encoded equality predicates on types directly on edge labels. This resulted in 7 join query templates, including four 4-edge queries, two 5-edge queries, and one 6-edge query. All of these queries are acyclic. There were also 2- and 3-edge queries, which we ignored because as their estimation is trivial with Markov tables of size 3. We generated 100 query instances from each template by choosing one edge label uniformly at random for each edge, while ensuring that the output of the query was non-empty. The final workload contained a total of 369 specific query instances.

We begin by comparing the 9 optimistic estimators we described on the two optimistic CEGs we defined. In order to set up an experiment in which we could test all of the 9 possible optimistic estimators, we used a Markov table that contained up to 3-size joins (i.e., h=3). A Markov table with only 2-size joins can not test different estimators based on different path-length heuristics or any cyclic query. We aim to answer four specific questions: (1) Which of the 9 possible optimistic estimators we outlined in Section 4.2 leads to most accurate estimates on acyclic queries and cyclic queries that contain $\leq$$3$ edges on $CEG_{O}$? (2) Which of the 9 estimators lead to most accurate estimates for cyclic queries with cycles of size $>3$ on $CEG_{O}$ and $CEG_{OCR}$? (3) Which of these CEGs lead to most accurate estimations under their best performing estimators? (4) Given the accuracies of best performing estimators, how much room for improvement is there for designing more accurate techniques, e.g., heuristics, for making estimates on $CEG_{O}$ and $CEG_{OCR}$?

To compare the accuracies of different estimators, for each query $Q$ in our workloads we make an estimate using each estimator and compute its q-error. If the true cardinality of $Q$ is $c$ and the estimate is $e$, then the q-error is $\max\{\frac{c}{e},\frac{e}{c}\}\geq 1$. For each workload, this gives us a distribution of q-errors, which we compare as follows. First, we take the logs of the q-errors so they are now $\geq 0$. If a q-error was an underestimate, we put a negative sign to it. This allows us to order the estimates from the least accurate underestimation to the least accurate overestimation. We then generate a box plot where the box represents the 25th, median, and 75th percentile cut-off marks. We also compute the mean of this distribution, excluding the top 10% of the distribution (ignoring under/over estimations) and draw it with a red dashed line in box plots.

Our next set of experiments aim to answer: How much does the bound-sketch optimization improve the optimistic estimators’ accuracy? This question was answered for the CBS pessimistic estimator in reference (Cai et al., 2019). We reproduce the experiment for MOLP in our context as well. To answer this question, we tested the effects of bound sketch on the JOB workload on IMDb and Acyclic workload on Hetionet and Epinions. We excluded WatDiv and DBLP as the max-hop-max estimator’s estimates are already very close to perfect on these datasets and there is no room for significant improvement (recall Figure 9). Then we applied the bound sketch optimization to both max-hop-max (on $CEG_{O}$) and MOLP estimators and measured the q-errors of the estimators under partitioning budgets of 1 (no partitioning), 4, 16, 64, and 128.

Our results are shown in Figure 12. As demonstrated in reference (Cai et al., 2019), our results confirm that bound sketch improves the accuracy of MOLP. The mean accuracy of MOLP increases between 15% and 89% across all of our datasets when moving between 1 and 128 partitions. Similarly, we also observe significant gains on the max-hop-max estimator though the results are data dependent. On Hetionet and Epinions, partitioning improves the mean accuracy at similar rates: by 25% and 89%, respectively. In contrast, we do not observe significant gains on IMDb. We note that the estimations of 68% of queries in Hetionet and 93% in Epinions strictly improved, so bound sketch is highly robust for optimistic estimators. We did not observe significant improvements in IMDb dataset for max-hop-max, although 93% of their q-errors strictly improve, albeit by a small amount. We will compare optimistic and pessimistic estimators in more detail in the next sub-section but readers can already see by inspecting the scale in the y-axes of Figure 12 that as observed in reference (Park et al., 2020) the pessimistic estimators are highly inaccurate and in our context orders of magnitude less accurate than optimistic ones.

The optimistic and pessimistic estimators we consider in this paper are summary-based estimators. Our next set of experiments compares max-hop-max (on $CEG_{O}$) and MOLP against each other and against other baseline summary-based estimators. A recent work (Park et al., 2020) has compared MOLP against two other summary-based estimators, Characteristic Sets (CS) (Neumann and Moerkotte, 2011) and SumRDF (Stefanoni et al., 2018). We reproduce and extend this comparison in our setting with our suggested max-hop-max optimistic estimator. We first give an overview of CS and SumRDF.