Learning Hierarchical Relational Representations through
Relational Convolutions

To place our framework in the context of previous work, we briefly discuss related forms of relational learning below, pointing first to the review of relational learning inductive biases by [6].

Graph neural networks (GNNs) are a class of neural network architectures which operate on graphs and process “relational” data [26, 21, 31, 34, 20, 38, e.g.,]. A defining feature of GNN models is their use of a form of neural message-passing, wherein the hidden representation of a node is updated as a function of the hidden representations of its neighbors on a graph [12]. Typical examples of tasks that GNNs are applied to include node classification, graph classification, and link prediction [14].

In GNNs, the ‘relations’ are given to the model as input via edges in a graph. In contrast, our architecture, as well as the relational architectures described below, operate on collections of objects without any relations given as input. Instead, such relational architectures must infer the relevant relations from the objects themselves. Still, graph neural networks can be applied to these relational tasks by passing in the collection of objects along with a complete graph.

Several works have proposed architectures with the ability to model relations by incorporating an attention mechanism [33, 34, 29, 40, 22, e.g.,]. Attention mechanisms, such as self-attention in Transformers [33], model relations between objects implicitly as an intermediate step in an information-retrieval operation to update the representation of each object as a function of its context.

There also exists a growing literature on neural architectures that aim to explicitly model relational information between objects. An early example is the relation network proposed by [30], which produces an embedding representation for a set of objects based on aggregated pairwise relations. [32] proposes the PrediNet architecture, which aims to learn relational representations that are compatible with predicate logic.  [18] proposes CoRelNet, a simple architecture based on ‘similarity scores’ that aims to distill the relational inductive biases discovered in previous work into a minimal architecture. [3, 2] explore relational inductive biases in the context of Transformers, and propose a view of relational inductive biases as a type of selective “information bottleneck” which disentangles relational information from object-level features. [36] provides a cognitive science perspective on this idea, arguing that a relational information bottleneck may be a mechanism for abstraction in the human mind.

In this section, we formalize a relational convolution operation which processes pairwise relations between objects to produce representations of the relational patterns within groups of objects. Suppose that we have a sequence of objects $(x_{1},\ldots,x_{n})$ and a relation tensor $R$ describing the pairwise relations between them, obtained by an MD-IPR layer via $R[i,j]=r(x_{i},x_{j})$. The key idea is to learn a template of relations between a small set of objects, and to “convolve” the template with the relation tensor, matching it against the relational patterns in different groups of objects. This transforms the relation tensor into a sequence of vectors, each summarizing the relational pattern in some group of objects. Crucially, this can now be composed with another relational layer to compute higher-order relations—i.e., relations on relations.

Fix some filter size $s<n$, where $s$ is a hyperparameter of the relational convolution layer. One ‘filter’ of size $s$ is given by the graphlet filter $f_{1}\in\mathbb{R}^{s\times s\times d_{r}}$. This is a template for the pairwise relations between a group of $s$ objects. Since pairwise relations can be viewed as edges on a graph, we use the term “graphlet filter” to refer to a template of pairwise relations between a small set of objects. Let $g\subset[n]$ be a group of $s$ objects among $(x_{1},\ldots,x_{n})$. Then, denote the relation sub-tensor associated with this group by $R[g]:=[R[i,j]]_{i,j\in g}$. We define the ‘relational inner product’ between this relation subtensor and the filter $f_{1}$ by

This is simply the standard inner product in the corresponding Euclidean space $\mathbb{R}^{s^{2}d_{r}}$. This quantity represents how much the relational pattern in $g$ matches the template $f_{1}$.

In a relational convolution layer, we learn $n_{f}$ different filters. Denote the collection of filters by $\bm{f}=\left(f_{1},\ldots,f_{n_{f}}\right)\in\mathbb{R}^{s\times s\times d_{r}\times n_{f}}$, which we call a graphlet filter. We define the relational inner product of a relation subtensor $R[g]$ with the graphlet filters $\bm{f}$ as the $n_{f}$-dimensional vector consisting of the relational inner products with each individual filter,

This vector summarizes various aspects of the relational pattern within a group, captured by several different filters¹¹1We have overloaded the notation $\left\langle\cdot,\cdot\right\rangle_{\mathrm{rel}}$, but will use the convention that a collection of filters is denoted by a bold symbol (e.g., $\bm{f}$ vs $f_{i}$) to distinguish between the two forms of the relational inner product.. Each filter corresponds to one dimension. This is reminiscent of convolutional neural networks, where each filter gives us one channel in the output tensor.

For a given group $g\subset[n]$, the relational inner product with a graphlet filter, $\left\langle R[g],\bm{f}\right\rangle_{\mathrm{rel}}$, gives us a vector summarizing the relational patterns inside that group. Let ${\mathcal{G}}$ be a set of groupings of the $n$ objects, each of size $s$. The relational convolution between a relation tensor $R$ and a relational graphlet filter $\bm{f}$ is defined as the sequence of relational inner products with each group in ${\mathcal{G}}$

In this section, we assumed that ${\mathcal{G}}$ was given. If some prior information is known about what groupings are relevant, this can be encoded in ${\mathcal{G}}$. Otherwise, if $n$ is small, ${\mathcal{G}}$ can be all possible combinations of size $s$. However, when $n$ is large, considering all combinations will be intractable. In the next subsection, we consider the problem of differentiably learning the relevant groups.

In the above formulation, the groups are ‘discrete’. Having discrete groups can be desirable for interpretability if the relevant groupings are known a priori or if considering every possible grouping is computationally and statistically feasible. However, if the relevant groupings are not known, then considering all possible combinations results in a rapid growth of the number of objects at each layer.

In order to address these issues, we can explicitly model and learn the relevant groups. This allows us to control the number of objects in the output sequence of a relational convolution such that only relevant groups are considered. We propose modeling groups via an attention operation.

Consider the input $(x_{1},\ldots,x_{n}),\,x_{i}\in\mathbb{R}^{d}$. Let $n_{g}$ be the number of groups to be learned and $s$ be the size of the graphlet filter (and hence the size of each group). These are hyperparameters of the model that we control. For each group $g\in[n_{g}]$, we learn $s$ different queries, $\{q_{i}^{g}\}_{g\in[n_{g}],i\in[s]}$, that will be used to retrieve a group of size $s$ via attention. The $i$-th object in the $k$-th group is retrieved as follows,

where $\bar{x}_{i}^{g}$ is the $i$-th object retrieved in the $g$-th group, $q_{i}^{g}$ is the query for retrieving the $i$-th object in the $g$-th group, $\mathtt{key}(x_{j})$ is the key associated with the object $x_{j}$, and $\beta$ is a temperature scaling parameter.

The $\mathtt{key}$ for each object is computed as a function of its position, features, and/or context. For example, to group objects based on their position, the key can be a positional embedding, $\mathtt{key}(x_{i})=PE_{i}$. To group based on features, the $\mathtt{key}$ can be a linear projection of the object’s feature vector, $\mathtt{key}(x_{i})=W_{k}x_{i}$. To group based on both position and features, the $\mathtt{key}$ can be a sum or concatenation of the above. Finally, computing keys after a self-attention operation allows objects to be grouped based on the context in which they occur.

The relation subtensor $\bar{R}[g]\in\mathbb{R}^{s\times s\times d_{r}}$ for each group $g\in[n_{g}]$ is computed using a shared MD-IPR layer $r(\cdot,\cdot)$,

The relational convolution is computed as before via,

Overall, relational convolution with group attention can be summarized as follows: 1) learn $n_{g}$ groupings of objects, retrieving $s$ objects per group; 2) compute the relation tensor of each group using an MD-IPR module; 3) compute a relational convolution with a learned set of graphlet filters $\bm{f}$, producing a sequence of $n_{g}$ vectors each describing the relational pattern within a (learned) group of objects.

Computing input-dependent queries. In the simplest case, the query vectors are simply learned parameters of the model, representing a fixed criterion for selecting the $n_{g}$ groups. The queries can also be produced in an input-dependent manner. There are many ways to do this. For example, the input $(x_{1},\ldots,x_{n})$ can be processed with some sequence or set embedder (e.g., through a self-attention operation) producing a vector embedding that can be mapped to different queries $\{q_{i}^{g}\}_{i,g}$ using learned linear maps.

Entropy regularization. Intuitively, we would ideally like the group attention scores in Equation 6 to be close to discrete assignments. To encourage the model to learn more structured group assignments, we add an entropy regularization to the loss function, ${\mathcal{L}}_{\mathtt{entr}}=(n_{g}\cdot s)^{-1}\sum_{g,i}H(\alpha_{i,\cdot}^{g})$, where $H(\alpha_{i,\cdot}^{g})=-\sum_{j}\alpha_{ij}^{g}\log(\alpha_{ij}^{g})$ is the Shannon entropy. As a heuristic, this regularization can be scaled by a factor proportional to $\log(\mathtt{n\_classes})/\log(n)$ so that it doesn’t dominate the underlying task’s loss. Sparsity regularization in neural attention has been explored in several previous works, including through entropy regularization [25, 24, 4, e.g.,].

Symmetric relational inner products. So far, we considered ordered groups. That is, the relational pattern computed by the relational inner product $\left\langle R[g],\bm{f}\right\rangle_{\mathrm{rel}}$ for the group $(1,2,3)$ is different from the group $(2,3,1)$. In some scenarios, symmetry in the representation of the relational pattern is a useful inductive bias. To capture this, we define a symmetric variant of the relational inner product that is invariant to the ordering of the elements in the group. This can be done by pooling over all permutations in the group. In particular, we suggest max-pooling or average-pooling, although any set-aggregator would be valid. We define the permutation-invariant relational inner product as

where $g!$ denotes the set of permutations of the group $g$, and pooling is done independently across dimensions.

Computational efficiency. Equation 6 can be computed in parallel with ${\mathcal{O}}(n\cdot n_{g}\cdot s\cdot d)$ operations. When the hyperparameters of the model are fixed, this is linear in the sequence length $n$. Equation 7 can be computed in parallel via efficient matrix multiplication with ${\mathcal{O}}(n_{g}\cdot s^{2}\cdot d_{r}\cdot d_{\mathrm{proj}})$ operations. Finally, Equation 8 can be computed in parallel with ${\mathcal{O}}(n_{g}\cdot s^{2}\cdot d_{r}\cdot n_{f})$ operations. The latter two computations do not scale with the number of objects in the input, and are only a function of the hyperparameters of the model.

A relational convolution block (including a MD-IPR module) is a simple neural module that can be composed to build a deep architecture for learning iteratively more complex relational feature representations.

Following the notation in Figure 2, let $n_{\ell}$ denote the number of objects and $d_{\ell}$ the object dimension at layer $\ell$. A relational convolution block receives as input a sequence of objects of shape $n_{\ell}\times d_{\ell}$ and returns a sequence of objects of shape $n_{\ell+1}\times d_{\ell+1}$ representing the relational patterns among groupings of objects. The output dimension $d_{\ell+1}$ corresponds to the number of graphlet filters $n_{f}$, and is a hyperparameter. The sequence length $n_{\ell+1}$ corresponds to the number of groups, and is $\lvert{\mathcal{G}}\rvert$ in the case of given discrete groups (Section 3.1) or a hyperparameter $n_{g}$ in the case of learned groups via group attention (Section 3.2). Each composition of a relational convolution block computes relational features of one degree higher (i.e., relations between relations).

A common recipe for building modern deep learning architectures is by using residual connections [15] and normalization [5]. This can be achieved for relational convolutional networks by fixing the number of groups $n_{g}$ and number of filters $n_{f}$ hyperparameters to be the same across all layers, such that the input shape and output shape remain the same. Then, letting $H_{\ell}$ denote the hidden representation at layer $\ell$, the overall architecture becomes $H_{\ell+1}=\mathrm{Norm}(H_{\ell}+W_{\ell+1}\mathrm{RelConvBlock(H_{\ell})})$, where $W_{\ell+1}$ is a linear transformation that controls where information is written to in the residual stream. This ResNet-style architecture allows for the hidden representation to encode relational information at multiple layers of hierarchy, retaining the information at shallower layers. Additionally, we can insert MLP layers to process the relational representations before the next relational convolution layer. In this paper, we limit our exploration to relatively shallow networks of the form $H_{\ell+1}=\mathrm{RelConvBlock}(H_{\ell})$.

The relational games dataset was contributed as a benchmark for relational reasoning by [32]. It consists of a family of binary classification tasks for identifying abstract relational rules between a set of objects represented as images. The object images depict simple geometric shapes and consist of three different splits with different visual styles for evaluating out-of-distribution generalization, referred to as “pentominoes”, “hexominoes”, and “stripes”. The input is a sequence of objects arranged in a $3\times 3$ grid. Each task corresponds to some relationship between the objects, and the target is to classify whether the relationship holds among the objects in the input or not (see Figure 4).

In our experiments, we evaluate out-of-distribution generalization by training on the pentominoes objects and evaluating on the hexominoes and stripes objects. The input to the models is presented as a sequence of $9$ objects, with each object represented as a $12\times 12\times 3$ RGB image. All models share the common architecture $(x_{1},\ldots,x_{n})\to\texttt{CNN}\to\{\cdot\}\to\texttt{MLP}\to\hat{y}$, where $\{\cdot\}$ indicates the central module being tested. In all models, the objects are first processed independently by a CNN with a shared architecture. The processed objects are then passed to the central module of the model. The final prediction is produced by an MLP with a shared architecture. In this section, we focus our comparison on four models: RelConvNet (ours), CoRelNet [18], PrediNet [32], and a Transformer [33]²²2The GNN baselines failed to learn the relational games tasks in a way that generalizes, often severely overfitting. For clarity of presentation, we defer results on the GNN baselines to Appendix A. . The pentominoes split is used for training, and the hexominoes and stripes splits are used to test out-of-distribution generalization after training. We train for 50 epochs using the categorical cross-entropy loss and the Adam optimizer with learning rate $0.001$, $\beta_{1}=0.9,\beta_{2}=0.999,\epsilon=10^{-7}$, and batch size $512$. For each model and task, we run 5 trials with different random seeds. Appendix A describes further experimental details about the architectures and training setup.

Out-of-distribution generalization. Figure 5 reports model performance on the two hold-out object sets after training. On the hexominoes objects, which are similar-looking to the pentominoes objects used for training, RelConvNet and CoRelNet do nearly perfectly. PrediNet and the Transformer do well on the simpler tasks, but struggle with the more difficult ‘match pattern’ task. The ‘stripes’ objects are visually more distinct from the objects in the training split, making generalization more difficult. We observe an overall drop in performance for all models. The drop is particularly dramatic for CoRelNet³³3The experiments in [18] on the relational games benchmark use a technique called “context normalization” [35] as a preprocessing step. We choose not to use this technique since it is an added confounder. We discuss this choice in Appendix C.. The separation between RelConvNet and the other models is largest on the ‘match pattern’ task of the stripes split (the most difficult task and the most difficult generalization split). Here, RelConvNet maintains a mean accuracy of 87% while the other models drop below 65%. We attribute this to RelConvNet’s ability to naturally represent higher-order relations and model groupings of objects. The CNN baseline learns the easier ‘same’, ‘between’, and ‘occurs’ tasks nearly perfectly, but completely fails to learn the more difficult ‘xoccurs’ and ‘match pattern’ tasks. This hard boundary suggests that explicit relational architectural inductive biases are necessary for learning more difficult relational tasks.

Data efficiency. We observe that the relational inductive biases of RelConvNet, and relational models more generally, grant a significant advantage in sample-efficiency. Figure 6 shows the training accuracy over the first 2,000 batches for each model. RelConvNet, CoRelNet, and PrediNet are explicitly relational architectures, whereas the Transformer is not. The Transformer is able to process relational information through its attention mechanism, but this information is entangled with the features of individual objects (which, for these relational tasks, is extraneous information). The Transformer consistently requires the largest amount of data to learn the relational games tasks. PrediNet tends to be more sample-efficient. RelConvNet and CoRelNet are the most sample-efficient, with RelConvNet only slightly more sample-efficient on most tasks.

On the ‘match pattern’ task, which is the most difficult, RelConvNet is significantly more sample-efficient. We attribute this to the fact that RelConvNet is able to model higher-order relations through its relational convolution module. The ‘match pattern’ task can be thought of as a second-order relational task—it involves computing the relational pattern in each of two groups, and comparing the two relational patterns. The relational convolution module naturally models this kind of situation since it learns representations of the relational patterns within groups of objects.

Learning groups via group attention. Next, we analyze RelConvNet’s ability to learn useful groupings through group attention in an end-to-end manner. We train a $2$-layer relational convolutional network with $8$ learned groups and a graphlet size of $3$. We group based on position by using positional embeddings for $\mathtt{key}(x_{i})$. In Figure 7, we visualize the group attention scores $\alpha_{ij}^{g}$ (see Equation 6) learned from one of the training runs. For each group $g\in[n_{g}]$, the figure depicts a $3\times 3$ grid representing the objects attended to in that group. Since each group contains $3$ objects, we represent the value $(\alpha_{ij}^{g})_{i\in[3]}$ in the $3$-channel HSV color representation. We observe that 1) group attention learns to ignore the middle row, which contains no relevant information; and 2) the selection of objects in the top row and the bottom row is structured. In particular, group $2$ considers the relational pattern within the bottom row and group $8$ considers the relational pattern in the top row, which is exactly how a human would tackle this problem. We refer to Figure 11 for an exploration of the effect of entropy regularization on group attention. We find that entropy regularization is necessary for the model to learn and causes the group attention scores to converge to interpretable discrete assignments.

Set is a card game that forms a simple-to-describe but challenging relational task. The ‘objects’ are a set of cards with four attributes, each of which can take one of three possible values. ‘Color’ can be red, green, or purple; ‘number’ can be one, two, or three; ‘shape’ can be diamond, squiggle, or oval; and ‘fill’ can be solid, striped, or empty. A ‘set’ is a triplet of cards such that each attribute is either a) the same on all three cards, or b) different on all three cards.

In Set, the task is: given a hand of $n>3$ cards, find a ‘set’ among them. Figure 8(a) depicts a positive and negative example for $n=5$, with $*$ indicating the ‘set’ in the positive example. This task is deceptively challenging, and is representative of the type of relational reasoning that humans excel at but machine learning systems still struggle with. To solve the task, one must process the sensory information of individual cards to identify the values of each attribute, and then reason about the relational pattern in each triplet of cards. The construct of relational convolutions proposed in this paper is a step towards developing machine learning systems that can perform this kind of relational reasoning.

In this section, we evaluate RelConvNet on a task based on Set and compare it to several baselines. The task is: given a collection of $n=5$ images of Set cards, determine whether or not they contain a ‘set’. All models share the common architecture $(x_{1},\ldots,x_{n})\to\texttt{CNN}\to\{\cdot\}\to\texttt{MLP}\to\hat{y}$, where $\{\cdot\}$ indicates the central module being tested. The CNN embedder is pre-trained on the task of classifying the four attributes of the cards and an intermediate layer is used to generate embeddings. The output MLP architecture is shared across all models. Further architectural details can be found in Appendix A.

In Set, there exists $\binom{81}{3}=85\,320$ triplets of cards, of which $1\,080$ are a ‘set’. We partition the ‘sets’ into training (70%), validation (15%), and test (15%) sets. The training, validation, and test datasets are generated by sampling $n$-tuples of cards such that with probability $1/2$ the $n$-tuple does not contain a set, and with probability $1/2$ it contains a set among the corresponding partition of sets. Partitioning the data in this way allows us to measure the models’ ability to “learn the rule” and identify new unseen ‘sets’. We train for 100 epochs with the same loss, optimizer, and batch size as the experiments in the previous section. For each model, we run 10 trials with different random seeds.

When using the default optimizer hyperparameters as in the previous experiment without hyperparameter tuning, we find that RelConvNet is the only model able to meaningfully learn the task in a manner that generalizes to unseen ‘sets’. In particular, we observe that many baselines severely overfit to the training data, failing to learn the rule and generalize (see Section B.1). Although RelConvNet did not require hyperparameter tuning, we carry out an extensive hyperparameter sweep for all other baselines individually in order to validate our conclusions against the best-achievable performance for each baseline. We ran a total of 1600 experimental runs searching over combinations of architectural hyperparameters (number of layers) and optimization hyperparameters (weight decay, learning rate schedule) individually for each baseline, with the goal of finding a hyperparameter configuration that is representative of the best achievable performance for each model class on this task. The results of the hyperparameter sweep are summarized in Appendix B.

Figure 8(b) shows the hold-out test accuracy for each model. Figure 8(c) shows the training and validation accuracy over the course of training. Here, RelConvNet uses the Adam optimizer with the default Tensorflow hyperparameters (constant learning rate of $0.001$, $\beta_{1}=0.9,\beta_{2}=0.999$) while each baseline has its own individually-optimized hyperparameters, described in Appendix B.

We observe a sharp separation between RelConvNet and all other baselines. While RelConvNet is able to learn the task and generalize to new ‘sets’ with near-perfect accuracy (avg: 97.9%), no other model is able to reach a comparable generalization accuracy even after hyperparameter tuning. The next best is the GAT model (avg: 67.5%). Several models are able to fit the training data, reaching near-perfect training accuracy, but they are unable to “learn the rule” in a way that generalizes to the validation or test sets. This suggests that while these models are powerful function approximators, they lack the inductive biases to learn hierarchical relations.

In Figure 9 we analyze the geometry of the representations learned by the relational convolution layer. We consider all triplets of cards, compute the relation subtensor using the learned MD-IPR layer, and plot the relational inner product with the learned graphlet filter $\bm{f}\in\mathbb{R}^{s\times s\times d_{r}\times n_{f}}$. The result is a $n_{f}$-dimensional vector for each triplet of cards. We perform PCA to plot this in two dimensions, and color-code each triplet of cards according to whether or not it forms a ‘set’. We find that the relational convolution layer learns a representation of the relational pattern in groups of objects that separates ‘sets’ and ‘non-sets’. In particular, the two classes form clusters that are linearly separable even when projected down to two dimensions by PCA. This explains why RelConvNet is able to learn the task in a way that generalizes while the other models are not. In Appendix E we expand on this discussion, and further analyze the representations learned by the MD-IPR layer, showing that the learned relations map to the color, number, shape, and fill attributes.

It is perhaps surprising that models like GNNs and Transformers perform poorly on these relational tasks, given their apparent ability to process relations through neural message-passing and attention, respectively. We remark that GNNs operate in a different domain compared to relational models like RelConvNe, PrediNet, and CoRelNet. In GNNs, the relations are an input to the model, received in the form of a graph, and are used to dictate the flow of information in a neural message-passing operation. By contrast, in relational convolutional networks, the input is simply a set of objects without relations—the relations need to be inferred as part of the feature representation process. Thus, GNNs operate in domains where relational information is already present (e.g., analysis of social networks, biological networks, etc.), whereas our framework aims to solve tasks that rely on relations but those relations need to be inferred end-to-end. This offers a partial explanation for the inability of GNNs to learn this task—GNNs are good at processing network-style relations when they are given as input, but may not be able to infer and hierarchically process relations when they are not given. In the case of Transformers, relations are modeled implicitly to direct information retrieval in attention, but are not encoded explicitly in the final representations. By contrast, RelConvNet operates on collections of objects and possesses inductive biases for learning iteratively more complex relational representations, guided only by the supervisory signal of the downstream task.

Models like CoRelNet and PrediNet have relational inductive biases, but lack compositionality. On the other hand, deep models like Transformers and GNNs are compositional, but lack relational inductive biases. This experiment suggests that compositionality and relational inductive biases are both necessary ingredients to efficiently learn representations of higher-order relations. RelConvNet is a compositional architecture imbued with relational inductive biases and a demonstrated ability to tackle hierarchical relational tasks.

In this paper, we proposed a compositional architecture and framework for learning hierarchical relational representations via a novel relational convolution operation. The relational convolution operation we propose here is a ‘convolution’ in the sense that it considers a patch of the relation tensor, given by a subset of objects, and compares the relations within it to a template graphlet filter via an appropriately-defined inner product. This is analogous to convolutional neural networks, where an image filter is compared against different patches of the input image. Moreover, we propose an attention-based mechanism for modeling useful groupings of objects in order to maintain scalability. By alternating inner product relation layers and relational convolution layers, we obtain an architecture that naturally models hierarchical relations.

In our experiments, we observed that general-purpose sequence models like the Transformer struggle to learn tasks that involve relational reasoning in a data-efficient manner. The relational inductive biases of RelConvNet, CoRelNet, and PrediNet result in significantly improved performance on the relational games tasks. These models each implement different kinds of relational inductive biases, and are each designed with different motivations in mind. For example, PrediNet’s architecture is loosely inspired by the structure of predicate logic, but can be understood as ultimately producing representations of pairwise difference-relations, with pairs of objects selected by an attention operation. CoRelNet is a minimal relational architecture that consists of computing an $n\times n$ inner product similarity matrix followed by a softmax normalization. RelConvNet, our proposed architecture, provides further flexibility across several dimensions. Like CoRelNet, it models relations as inner products of feature maps, but it achieves greater representational capacity by learning multi-dimensional relations through multiple learned feature maps or filters. More importantly, the relational convolutions operation enables learning higher-order relations between groups of objects. This is in contrast to both PrediNet and CoRelNet, which are limited to pairwise relations. Our experiments show that the inductive biases of RelConvNet result in improved performance in relational reasoning tasks. In particular, the Set task, where RelConvNet was the only model able to generalize non-trivially, demonstrates the necessity for explicit inductive biases that support learning hierarchical relations.

The tasks considered here are solvable by modeling only second-order relations. In the case of the relational games benchmark of [32], we observe that the tasks are saturated by the relational convolutional networks architecture. While the “contains set” task demonstrates a sharp separation between relational convolutional networks and existing baselines, this task too only involves second-order relations. A more thorough evaluation of this architecture, and future architectures for modeling hierarchical relations, would require the development of new benchmark tasks and datasets that involve a larger number of objects and higher-order relations. This is a subtle and non-trivial task that we leave for future work.

The modules proposed in this paper assume object-centric representations as input. In particular, the tasks considered in our experiments have an explicit delineation between different objects. In more general settings, object information may need to be extracted from raw stimulus explicitly by the system (e.g., a natural image containing multiple objects in apriori unknown positions). Learning object-centric representations is an active area of research [28, 13, 22, 19], and is related but separate from learning relational representations. These methods produce a set of embedding vectors, each describing a different object in the scene, which can then be passed to the central processing module (e.g., a relational processing module such as RelConvNet). In future work, it will be important to explore how well RelConvNet integrates with methods for learning object-centric representations in an end-to-end system.

The experiments considered here are synthetic relational tasks designed for a controlled evaluation. In more realistic settings, we envision relational convolutional networks as modules embedded in a broader architecture. For example, a relational convolutional network can be embedded into an RL agent to enable performing tasks involving relational reasoning. Similarly, relational convolutions can perhaps be integrated into general-purpose sequence models, such as Transformers, to enable improved relational reasoning while retaining the generality of the architecture.