Object-Centric Relational Representations
for Image Generation

The code to reproduce the experiments and generate the PRO dataset is available online¹¹1https://github.com/LucaButera/graphose_ocrrig. It is all written in Python 3.9, leveraging Pytorch (Paszke et al., 2019) and Pytorch Lightning (Falcon & team, 2019) to define models and data, while Hydra (Yadan, 2019) is used to manage the experiment configuration. Weights & Biases (Biewald, 2020) is used to log and compare experiment results. We run all the experiments on an NVIDIA RTX A5000 GPU equipped with 24GBs of VRAM.

Each model is trained under the same settings and metrics are computed over three different runs each. In particular, we use the Adam (Kingma & Ba, 2015) optimizer, with cosine annealing (Loshchilov & Hutter, 2017) learning rate schedule with period $300$ for the PRO dataset and $150$ for the Humans one, starting learning rate $0.002$ and final learning rate $0.00002$. Models are trained up to $300$ (PRO) or $150$ (Humans) epochs each, with a batch size of $64$ and early stopping with patience $50$ on the FID. Notice that the learning rate for the parameters of the pre-trained mask generator used in GraPhOSE was reduced to $\frac{1}{2}$ (PRO) or $\frac{1}{100}$ (Humans) with respect to the learning rate of the other parameters, in order to avoid catastrophic foregetting at the beginning of training. This procedure could be enhanced by slowly raising the learning rate of the pre-trained parameters back to the value of other parameters, as training epochs go by, with the aim of favoring mask adaptation after the training has stabilized during the first epochs. The reduction was much stronger for Humans in order to counter balance the heavy bias the dataset had towards poses of people standing up with arms straight down. However, this reduction factor is indeed an hyper-parameter that can be effectively used to allow for more or less deviation from the surrogate masks, depending on the downstream task characteristics.

Referring to Equation (7), we explicitly define ${\bm{T}}_{ij}$ as

where vectors ${\bm{p}}$ denote node coordinates, $\arctan 2$ is the element-wise 2-argument arctangent and $a$ denotes a parameter that regulates the scaling ratio of the second dimension w.r.t. the first one. In our experiments $a$ is set to $10$.

Referring to Section 5.1, we set the Barbasi-Albert model’s number of edges for new nodes parameter to $1$ in $90\%$ of the cases and to $2$ in the remaining $10\%$, if the number of nodes is less than $10$. For graphs with more than $10$ nodes, it is always set to $1$. Erdos-Renyi model’s edge probability is set to $\exp\left(\frac{1}{n}\right)-0.95$, where $n$ is the number of nodes.

Our synthetic Pose-Representable Objects (PRO) dataset consists of square images (i.e., with same height and width) rendered from random graphs that represent certain object’s parts and their color. A graph representing an object has a fixed connectivity (i.e., adjacency matrix) and each node has color (from a pool of possible ones) and class attributes together with a coordinate in 2D space. Coordinates and colors, for each node in a sample, are generated randomly within reasonable boundaries (e.g. to stay within the image or to comply to a specific structure, like that of a regular polygon). Figure 10 shows some examples of each object type in our dataset; the graph that describes the object has the edges superimposed in white, while the nodes are colored accordingly to their attributes. Images from the PRO dataset can contain multiple objects, however we experimented with up to four objects for image, in order to keep their size reasonably big. Regarding the objects’ structural constraints, we highlight the following properties: Pie has an inner angle of up to $135^{\circ}$, as the two pie tips have the same class and it would not be possible to discriminate which part of the pie has to be filled otherwise. Pie nodes have always the same color. Scissors have the structural constraint that the angle between the blades and the handles must be the same and be contained between $30^{\circ}$ and $90^{\circ}$. Also blades and handles must respectively have the same color. For the Hand object we have constraints on the maximum relative angle between each phalanx, in order to avoid impossible orientations; moreover, fingers must have a consistent color. Hands can uniformly be either "right" or "left" hands. The Robotic Arm, similar to the Hand, has constraints on the relative angles between its parts, to similarly avoid collapsed configurations. Additionally, the number of segments in the arm can vary between $3$ and $7$ and their color must not change. Similarly, also the prongs must have a consistent coloring. Hollow Polygon and Filled Polygon must always have all nodes of the same color and the presence of a "center" node determines if the polygon is filled or not. We experiment with polygons up to $8$ vertices. The Lattice is the only structure with randomized connectivity, as nodes, uniformly distributed between $3$ and $9$, are sampled from a Poisson Disk distribution (Cook, 1986) and are connected if their distance is less than $20\%$ of the whole image width. The lattice is also the only object in which each node has a random color and the segment connecting two nodes is colored with a linear gradient going from one node’s color to the other’s.

Table 2 contains a summary of the classes for each object’s node. We would like to underline that the presence of structure in the assignment of colors and classes to object nodes is not just to make them visually consistent, but is fundamental in order to make it possible to reconstruct missing attributes by using relational information and acquired knowledge on how said structure behaves.

Regarding dataset size, in our experiments we generated $100000$ images. We employed $1000$ of these as validation set, and another $1000$ as test set. Images were sampled so that the dataset contains a uniform distribution of different objects. Moreover, also the number of objects that appear in an image at once, between $1$ and $4$, is uniformly distributed across the dataset. Train, validation and test splits, while random, preserved these uniform distributions.

The Humans dataset consists of roughly $300000$ images, coming from MPII Human Pose (Andriluka et al., 2014), Deep Fashion (Liu et al., 2016) and Market $1501$ (Zheng et al., 2015), which are all widely adopted benchmarks for pose estimation and other human body related tasks. Out of these, around $3000$ images, were used for validation and an analogous size for testing. Train, validation, and test sets were partitioned at random but the proportion of samples from the $3$ original benchmarks was maintained. Within the Humans task, the only object type is Person, while there are $14$ node classes (left and right shoulders, elbows, wrists, ankles, knees, hips plus base of the neck and top of the head). Note that, differently from the PRO dataset, these real-world images contain different backgrounds and lighting conditions. We do not address the explicit conditioning over these properties in our model, however these could be incorporated via a vector representation that acts as a global graph attribute.

In the following we give a more precise specification for the architecture we used in our experiments, while specifying that these represent only a possible way of implementing our approach and deeper or more complex architectures might be better suited for different tasks. Coherently with previous notation, Tables 3, 4, 5, 6 and 7 describe the architectures of the different components in our framework; while Tables 8 and 9, for clarity, describe recurring building blocks (i.e., used multiple times) inspired by Brock et al. (2019). In particular, we clarify the meaning of some layers: MultiEmbedding simply denotes multiple embedding layers (Bengio et al., 2000) with the same output size, used to obtain a continuous representation for different discrete node attributes. The first hyperparameter refers to the embedding size, while the second describes the way in which embeddings of different features are combined into a single embedding. In particular, cat refers to concatenation along the feature dimension. The usage of embedding layers can be exchanged for multiple linear layers, if the input features are already continuous. PoseConv is a layer that implements the function described in Equation (3); by respectively naming its three hyperparameters $i$, $h$ and $o$, we can further specify that, in this implementation, $g_{s}$ and $g_{l}$ are linear layers with input size $i+2$ (2 is the size of $p_{i}$) and output size $h$, while $g_{g}$ is a linear layer with input size $h$ and output size $o$. Analogously, PoseConv2D refers to a layer implementing Equation (5), where $g_{o}$ is a ConvBlock2D with $in=2\cdot i$, $out=o$ and $up=2$, followed by $f\left({\bm{\mathsfit{H}}}\right)=\sigma\left({\bm{\mathsfit{H}}}\right)\odot{\bm{\mathsfit{H}}}$. $g_{s}$ is a times two upsampling followed by a $1\times 1$ 2D convolution with $0$ padding, input size $i$ and output size $o$ and, finally, $g_{g}$ is also a $1\times 1$ 2D convolution with $0$ padding and both input and output size set to $o$. In this case $i$ is the first hyperparameter of PoseConv2D, while $o$ is the second. FNNPoseConv is the FNN equivalent of PoseConv, computing

where, naming the first hyperparameter $i$ and the second $o$, $g_{nr}$ and $g_{s}$ are both linear layers, the former with input size $i+2$ and output size $o$ and the latter with input size $i$ and output size $o$. Note that, when using the FNN Conditioner, the PoseConv layers of the discriminator are swapped for equivalent FNNPoseConv ones. Furthermore, note that, unless differently stated, all Conv2D operations use a kernel of size $3\times 3$ with stride and padding set to $1$. All layer weights are normalized through spectral normalization (Miyato et al., 2018), Upsample operations are implemented through bilinear interpolation, while Downsample ones use 2D average pooling. For completeness, we clarify that the Sum(H,W) operation in Table 7, refers to summing the tensor ${\bm{\mathsfit{O}}}_{2}\in\mathbb{R}^{B\times C\times H\times W}$ over the last two dimensions, thus obtaining the matrix ${\bm{O}}_{2}\in\mathbb{R}^{B\times C}$.

Tables 10 and 11 show, respectively, the architectures of the downstream generator and discriminator, used for the human dataset. In such case the GraPhOSE architecture remained the same, although the number of hidden features was increased by a factor of two and $\mu_{\theta}$ dropped the ConvBlock2D, as the target images had shape $64\times 64$, hence no further upsampling was required. The main differences are present at the discriminator and generator level, as for this more complex task we borrowed the building blocks of BigGAN (Brock et al., 2019). In particular, DConvBlock2D^∗ has a slightly different architecture in which the first Conv2D layer outputs $2\cdot out$ features and is followed by a ReLU and another Conv2D layer that outputs $out$ features. ConvBlock2D^∗, instead, takes as additional input the downsampled layout mask ${\bm{\mathsfit{L}}}^{*}$, which goes through a Conv2D layer that outputs $out$ features; these are concatenated to the output of the Conv2D on the standard input and go through a second Conv2D($2\cdot out$, $out$) layer, before the skip connection. This serves the purpose of conditioning the generative process at different scales, as, in this case, the downstream generator starts from a $1\times 4\times 4$ feature map, ${\bm{\mathsfit{Z}}}$, which is upsampled at each step, instead of directly using the layout mask and always keeping the same feature map size. For completeness, ${\bm{\mathsfit{Z}}}$ represents additional latent conditioning, required to account for the background, and is obtained by reshaping the output of a linear layer with output size $16$, which takes as input a vector of normally distributed noise of size $8$.

Table 12 summarizes the parameter counts for GraPhOSE and the baselines, differentiating between the smaller versions used for the PRO dataset and the larger ones used for Humans.

We chose Frechet Inception Distance (Heusel et al., 2017) and Inception Score (Salimans et al., 2016) as metrics for our generative models as both are commonly used to asses generation quality. However, only FID is effectively useful in assessing how much the generated samples resemble those coming from the real distribution, as this metric compares the similarity between the distribution of images coming from the generator and those coming from the dataset. Inception Score is limited to evaluation of some properties of the generated images distribution, hence it is less useful as a metric of comparison between models because it can only tell us if the model is capable of generating images that are sufficiently distinct and recognizable, even though they might actually be completely different from those of the real distribution. In other words, IS is only influenced by the capability of the generative model of producing samples that can be distinctively allocated to a class and that, at the same time, are diverse within each class. In our setting, the differences between GraPhose and each baseline are not expected to impact such properties, which are mostly tied to the downstream model, which is always the same for each task. This makes the resulting Inception Score quite uninformative as a way of comparing the various models; nonetheless, it highlights that learning the object’s relational masks does not indeed harm the appeal of the generated samples. For the PRO dataset we also reported the Structural Similarity Index Score (SSIM) (Wang et al., 2004). This metric is used as a measure of image reconstruction accuracy, hence it is suitable to evaluate generation quality only in situations where the input to the model uniquely and completely describes the image to be generated. That is the case for the PRO task, as there is a deterministic mapping between the graph and the resulting image. However, that is not the case for the Humans dataset, in which the input graph just describes the person’s pose but carries no information regarding the clothing or the background, hence the reason for not employing such metric in that scenario.

As discussed, our formulation allows for sub-conditioning the pose graph, i.e., having nodes without any semantic attribute assigned, in order to let the generative process decide how to treat them. Figure 11 shows some examples of this behavior, in which white nodes represent nodes without any semantic attribute. We can see that the generation is mostly consistent when the attributes can be recovered from neighbouring nodes. However, in cases in which the information needs to be reconstructed through multiple hops (e.g., the scissors handle) the model uses the nearest color instead. This might be the result of biases in the data. Mind that we use the term reconstruction loosely here, as the model is trained with a generative objective, hence we might expect different results if we train it on a strict reconstruction task. Figure 12, instead, shows what happens when we have a growing chain of missing attributes. We can see that semantics are coherently filled in, by GraPhOSE, up to a certain number of subsequent masked nodes. This is, in part, due to architectural constraints, as the number of hops information can traverse depends on the number of message passing steps. On the other hand, the usage of graph convolution operators that try to prevent over-smoothing might also play a role in this. However, notice we did not train our model to specifically address high chances of partial conditioning, i.e., missing attributes; different results can be expected if we purposefully train it on datasets with such characteristics. Even though, in such experiments, the color relationships between object components are rather simple, we think these results, tied with the architectural constraints involved, can be a promising basis for further, and more specifically aimed, experimentation.

Figure 13 shows additional examples of how the pre-learned masks consistently adapt to the downstream task during the end-to-end training, while figure 14 shows how directly training on the downstream task leads to masks that are not visually meaningful. This is particularly severe for the case of human poses, where the dataset is heavily biased towards a few particularly common ones.

Figure 15 shows further examples of how node-mask locality is preserved after downstream training, and adapted to the specifics of the objects in the task. We can notice that the Pie (e) is the only object for which this locality seems to be lost, however it is not surprising as its graph representation diameter is only $2$, with just $3$ nodes, hence it is harder to avoid over-smoothing. For graphs with larger diameter and/or number of nodes, we see that locality is retained.