Learning to generate visual content is a fundamental task in computer vision and graphics, with numerous applications from sim-to-real training of autonomous agents, to creating visuals for games and movies. While the quality of generating still images has leaped forward recently (Brock et al., 2019; Karras et al., 2020), generating videos remains in its infancy. Generating actions and interactions between objects is perhaps the most challenging aspect of conditional video generation. When constructing a complex scene with multiple agents (e.g. a driving scene with several cars or a basketball game with a set of players) it would be important to have control over multiple agents and their actions as well as to coordinate them w.r.t. each other (e.g. to generate a Pick ’n’ Roll move in basketball). Actions create long-range spatio-temporal dependencies between people and objects they interact with. For example, when a player passes a ball, the entire movement sequence of all the entities (thrower, ball, receiver) must be coordinated and carefully timed. This paper specifically addresses this challenge, the task of generating coordinated and timed actions, as an important step towards generating videos of complex scenes.

Current approaches for conditional video generation are not well suited to condition the generation on multiple coordinated (potentially simultaneous) timed actions. One line of work, Class Conditional Video Generation (Tulyakov et al., 2018; Nawhal et al., 2020a) relies on a single action (Tulyakov et al., 2018) and a potential object class (Nawhal et al., 2020a). However, this approach cannot deal with variable number of objects, multiple simultaneous actions and does not control the timing of the actions. Future Video Prediction (Watters et al., 2017; Ye et al., 2019), generates future frames based on an initial input frame, but one frame will not suffice for generation of multiple coordinated actions. In another line of work, Video-to-Video Translation (Wang et al., 2018a, 2019), the goal is to translate a sequence of semantic masks into an output video. However, segmentation maps contain only object class information, and thus do not explicitly capture the action information. As Wang et al. (2018a) noted for the case of generating car turns, semantic-maps do not have enough information to distinguish car turns from a normally driving car, which leads to mistakes in video-translation. Finally, Text-to-Video (Li et al., 2018; Gupta et al., 2018) work is promising because language can describe complex actions. However, in applications that require a precise description of a scene, natural language is not ideal due to ambiguities (MacDonald et al., 1994), user subjectivity (Wiebe et al., 2004) and difficulty to infer specific timings, if such information is missing. Hence, we address this problem with a more structured approach.

To generate videos based on actions, we propose to represent actions in an object-centric graph structure we call an “Action Graph” (AG) and define the new task of “Action Graph to Video” generation. An AG is a graph structure aimed at representing coordinated and timed actions. Its nodes represent objects and edges represent actions annotated with their start and end time (Figure 1). AGs are an intuitive representation for describing timed actions and would be a natural way to provide precise inputs to generative models. A key advantage of AGs is their ability to describe the dynamics of objects and actions precisely in a scene. In the “Action Graph to Video” task, the input is an initial frame of the video with the ground-truth layout consisting of the objects classes and bounding boxes, and an AG.

There are multiple unique challenges in generating a video from an AG that cannot be addressed using current generation methods. First, each action in the graph unfolds over time, so the model needs to “keep track” of the progress of actions rather than just condition on previous frames as commonly done. Second, AGs may contain multiple concurrent actions and the generation process needs to combine them in a realistic way. Third, one has to design a training loss that captures the spatio-temporal video structure to ensure that the semantics of the AG is accurately captured.

To address these challenges, our AG2Vid model uses three levels of abstraction. First, an action scheduling mechanism we call “Clocked Edges” tracks the progress of actions over time. Based on this, a graph neural network (Kipf & Welling, 2016) operates on the AG with the Clocked Edges and predicts a sequence of scene layouts. Finally, pixels are generated conditioned on the predicted layouts. We apply our AG2Vid model to CATER (Girdhar & Ramanan, 2020) and Something-Something V2 (Goyal et al., 2017) datasets and show that our approach results in videos that are more realistic and semantically compliant with the input AG than the baselines.

Last, we evaluate the AG2Vid model in a zero-shot setting where AGs involve novel compositions of the learned actions. The structured nature of our approach enables it to successfully generalize to new “composite” actions, as confirmed by the human raters. We also provide qualitative results of our model trained on Something-Something V2 in a challenging setting where AGs include multiple actions, while at training time each AG has a single action.

Video generation is challenging because videos often contain long range dependencies. Recent generation approaches (Vondrick et al., 2016; Denton & Fergus, 2018; Lee et al., 2018; Babaeizadeh et al., 2018; Villegas et al., 2019; Kumar et al., 2020) extended the framework of unconditional image generation to video, based on a latent representation. For example, MoCoGAN (Tulyakov et al., 2018) disentangles the latent space representations of motion and content to generate a sequence of frames using RNNs; TGAN (Saito et al., 2017) generates each frame in a video separately while also using a temporal generator to model dynamics across the frames. Here, we tackle a different problem by aiming to generate videos that comply with Action Graphs (AGs).

Class conditional action generation allows the generation of video based on a single action (Tulyakov et al., 2018; Nawhal et al., 2020b). A recent method, HOI-GAN (HG) (Nawhal et al., 2020b), was proposed for the generation task of a single action and object. Specifically, HG addresses the zero-shot setting, and the model is tested on action and object compositions which are first presented at test time. Our focus is on generation of multiple simultaneous actions over time, performed by multiple objects. Our approach directly addresses this challenge via the AG representation and the notion of clocked edges.

Other conditional video generation methods have attracted considerable interest recently, with focus on two main tasks: video prediction (Mathieu et al., 2015; Battaglia et al., 2016; Walker et al., 2016; Watters et al., 2017; Kipf et al., 2018; Ye et al., 2019) and video-to-video translation (Wang et al., 2019; Chan et al., 2019; Siarohin et al., 2019; Kim et al., 2019; Mallya et al., 2020). In video prediction, the goal is to generate future video frames conditioned on few initial frames. For example, it was proposed to train predictors with GANs (Goodfellow et al., 2014) to predict future pixels (Mathieu et al., 2015). However, directly predicting pixels is challenging (Walker et al., 2016). Instead of pixels, researchers explored object-centric graphs and performed prediction on these (Battaglia et al., 2016; Luc et al., 2018; Ye et al., 2019). While inspired by object-centric representations, our method is different from these works as our generation is goal-oriented and guided by an AG. The video-to-video translation task was proposed by Bansal et al. (2018); Wang et al. (2018a), where a natural video is generated from a different video based on frame-wise semantic segmentation annotations or keypoints. However, densely labeling pixels for each frame is expensive, and might not even be necessary. Motivated by this, researchers have sought to perform generation conditioned on more accessible signals including audio or text (Song et al., 2018; Fried et al., 2019; Ginosar et al., 2019). Here, we propose to synthesize videos conditioned on an AG, which is cheaper to obtain compared to semantic segmentation and is a more structured representation compared to natural audio or text.

Various methods have been proposed to generate videos based on input text (Marwah et al., 2017; Pan et al., 2017; Li et al., 2018). Most recent methods typically used very short captions which do not contain complex descriptions of actions. For example, Li et al. (2018) used video-caption pairs from YouTube, where typical captions are ”playing hockey” or ”flying a kite”. Gupta et al. (2018) proposed the Flinstones animated dataset and introduced the CRAFT model for text-to-video generation. While the CRAFT model relies on text-to-video retrieval, our approach works in an end-to-end manner and aims to accurately synthesize the given input actions.

Scene Graphs (SG) (Johnson et al., 2015, 2018) are a structured representation that models scenes, where objects are nodes and relations are edges. SGs have been widely used in various tasks including image retrieval (Johnson et al., 2015; Schuster et al., 2015), relationship modeling (Krishna et al., 2018; Schroeder et al., 2019; Raboh et al., 2020), SG prediction (Xu et al., 2017; Newell & Deng, 2017; Zellers et al., 2018; Herzig et al., 2018), and image captioning (Xu et al., 2019). SGs have also been applied to image generation (Johnson et al., 2018; Deng et al., 2018; Herzig et al., 2020), where the goal is to generate a natural image corresponding to the input SG. Recently, Ji et al. (2019) presented Action Genome, a video dataset where videos of actions from Charades (Sigurdsson et al., 2016) are also annotated by SGs. More generally, spatio-temporal graphs have been explored in the field of action recognition (Jain et al., 2016; Sun et al., 2018; Wang & Gupta, 2018; Yan et al., 2018; Girdhar et al., 2019; Herzig et al., 2019; Materzynska et al., 2020). For example, a space-time region graph was proposed by Wang & Gupta (2018) where object regions are taken as nodes and a GCN (Kipf & Welling, 2016) is applied to perform reasoning across objects for classifying actions. Recently, it was also shown (Ji et al., 2019; Yi et al., 2019; Girdhar & Ramanan, 2020) that a key obstacle in action recognition is the ability to capture the long-range dependencies and compositionality of actions. While inspired by these approaches, we focus on generating videos which is a very different challenge.

Our goal in this work is to build a model for synthesizing videos that contain a specified set of timed actions. A key component in this effort is developing a semantic representation to describe the actions performed by different objects in the scene. Towards this end, we propose to use a graph-structure we call Action Graph (AG). In an AG, nodes correspond to objects, and edges correspond to actions that the objects participate in. Objects and actions are labeled with semantic categories, while actions are also annotated with their start and end times. Formally, an AG is a tuple $(\mathcal{C},\mathcal{A},O,E)$ defined as follows:

A vocabulary of object categories $\mathbf{\mathcal{C}}$: Object categories can include attributes, e.g., “Blue Cylinder” or “Large Box”.

A vocabulary of actions $\mathbf{\mathcal{A}}$: Some actions, such as “Slide” may also have attributes (e.g., the destination coordinates).

Object nodes $\mathbf{O}$: A set $O\in\mathcal{C}^{n}$ of $n$ objects.

Action edges $\mathbf{E}$: Actions are represented as labeled directed edges between object nodes. Each edge is annotated with an action category and with the time period during which the action is performed. Formally, each edge is of the form $(i,a,j,t_{s},t_{e})$, where $i,j\in\{1,...,n\}$ are object instances, $a\in\mathcal{A}$ is an action and $t_{s},t_{e}\in\mathbb{N}$ are action start and end time. Thus, this edge implies that object $i$ (which has category $o_{i}$) performs an action $a$ over object $j$, and that this action takes place between times $t_{s}$ and $t_{e}$. We note that an AG edge can directly model actions over a single object and a pair of objects. For example, “Swap the positions of objects $i$ and $j$ between time 0 and 9” is an action over two objects corresponding to the edge $(i,swap,j,0,9)$. Some actions, such as “Rotate”, involve only one object and will therefore be specified as self-loops.

We now turn to the key challenge of this paper: transforming an AG into a video. Naturally, this transformation will be learned from data. The generation problem is defined as follows: we wish to build a generator that takes as input an AG and outputs a video. We will also allow conditioning on the first video frame and layout, so we can preserve the visual attributes of the given objects.²²2Using the first frame and layout can be avoided by using a SG2Image model (Johnson et al., 2018; Ashual & Wolf, 2019; Herzig et al., 2020) for generating the first frame.

There are multiple unique challenges in generating a video from an AG. First, each action in the graph unfolds over time, so the model needs to “keep track” of the progress of actions. Second, AGs may contain multiple concurrent actions and the generation process needs to combine them in a realistic way. Third, one has to design a training loss that captures the spatio-temporal video structure to ensure that the semantics of the AG is accurately captured.

We evaluate our AG2Vid model on two datasets: CATER (Girdhar & Ramanan, 2020) and Something-Something V2 (Goyal et al., 2017) (henceforth SmthV2). For each dataset, we learn an AG2Vid model with a given set of actions. We then evaluate the visual quality of the generated videos and measure how they semantically comply with the input actions. Finally, we estimate the generalization of the AG2Vid model to novel composed actions.

(2) Something-Something V2 contains real videos of humans performing basic actions. Here we use the eight most frequent actions: “Push Left”, “Push Right”, “Move Down”, “Move Up”, “Cover”, “Uncover”, “Put” and “Take”. Every videos has a single action and up to three different objects, coming from $122$ different object classses, including the hand which is performing the action. We use the bounding box annotations of the objects from Materzynska et al. (2020). We set the first frame where the objects are present as the action’s start time and the last one as the end time.

Layout Generation Ablation Results. We compare the choice of GCN to the RNN baseline, as an alternative to the implementation of the AG2Vid LGF. To evaluate, we use the mIOU (mean Intersection Over Union) between the predicted and GT bounding boxes. The results confirm the advantage of GCN over RNN with a mIOU of $93.09$ vs. $75.71$ over CATER and $51.32$ vs. $41.28$ over SmthV2. We include the full results in Sec. 4.2 in the Supplementary.
Loss Ablation Results. Table 2 reports ablations over the losses used in FGF, confirming that each loss, including the pixel actions discriminator loss, contributes to the overall visual quality on CATER and SmthV2.
Semantic and Visual Quality. We include human evaluation results in Table 1, which indicate that AG2Vid is more semantically accurate and has better visual quality than the baselines. Comparing to AG2Vid-GTL, it is slightly worse, as expected. We also provide qualitative AG2Vid generation examples in Figure 4, and comparison to baselines in Figure 5. For more results and a per-action analysis of the correctness of the generated actions see the Suppl.Section 4.5. Table 3 evaluates visual quality using several automatic metrics, with similar takeaways as in Table 1.
Action Timings. To evaluate to what extent the AG2Vid model can control the timing of actions, we generated pairs of AGs where in the first AG an action is coordinated to appear earlier than in the second AG, keeping everything else constant. We then use the AG2Vid model to create corresponding videos and ask MTurk annotators to choose the video where the action appears first. In 89.45% of the cases, the annotators confirm the intended result. For more information see Section 4.4 in the Supplementary.
Composing New Actions. Finally, we validate the ability of our AG2Vid model to generalize at test time to zero-shot compositions of actions. Here, we manually define four new “composite” actions. We are using learned atomic actions to generate new combinations that did not appear in the training data (either by having the same object perform multiple actions at the same time, or having multiple objects perform coordinated actions), see Figure 6. Specifically, in CATER, we created the action “Swap” based on “Pick-Place” and “Slide”, and action “Huddle” based on multiple “Contain” actions. In SmthV2 we composed the “Push-Left” and “Move-Down” to form the “Left-Down” action, and “Push-Right” with “Move-Up” to form the “Right-Up” action. For each generated video, raters were asked to choose the correct action class from the list of all possible actions. The average class recall for CATER and SmthV2 is $96.65$ and $87.5$ respectively. See Supplementary for more details.

To further demonstrate that AG2Vid can generalize to AGs with more objects and actions than seen in training, we include additional examples. In Figure 7, the AG contains two simultaneous actions that were never observed together during the training. In Figure 8, the AG contains three consecutive actions, a scenario also not seen during training. These examples demonstrate the ability of our AG2Vid model to generalize to new and more complex scenarios.

In this work, we aim to generate videos of multiple coordinated and timed actions. To represent the actions, we use an Action Graph, and propose the “Action Graph to Video” task. Based on the AG representation, our AG2Vid model can synthesize complex videos of new action compositions and control the timing of actions. The AG2Vid model uses three levels of abstraction: scheduling actions using the “Clocked Edges” mechanism, coarse layout prediction, and fine-grained pixels level prediction. Despite outperforming other methods, AG2Vid still fails in certain cases, e.g, when synthesize occluded or overlapping objects. We believe more fine-grained intermediate representations like segmentation masks can be incorporated to alleviate this difficulty. Furthermore, while it was shown that AG2Vid can synthesize actions that have rigid motion, synthesizing actions with non-rigid motion might require conditioning on random noise, or by explicitly defining action attributes like speed or style. We leave such extensions for future work.

We would like to thank Lior Bracha for her help running MTurk experiments, and to Haggai Maron and Yuval Atzmon for helpful feedback and discussions. This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC HOLI 819080). Trevor Darrell’s group was supported in part by DoD including DARPA’s XAI, LwLL, and/or SemaFor programs, as well as BAIR’s industrial alliance programs. Gal Chechik’s group was supported by the Israel Science Foundation (ISF 737/2018), and by an equipment grant to Gal Chechik and Bar-Ilan University from the Israel Science Foundation (ISF 2332/18). This work was completed in partial fulfillment for the Ph.D degree of Amir Bar.

As explained in the main paper, we used a Graph Convolution Network (GCN) (Kipf & Welling, 2016) to predict the layout $\ell_{t}$ at time step $t$. The GCN uses the structure of the action graph, and propagates information along this graph (in $K$ iterations) to obtain a set of layout coordinates and embedding per object.

Each object category $c\in\mathcal{C}$ is assigned a learned embedding $\bm{\phi}_{c}\in\mathbb{R}^{D}$ and each action $a\in\mathcal{R}$ is assigned a learned embedding $\bm{\psi}_{a}\in\mathbb{R}^{D}$. We next explain how to obtain the layouts $\ell_{t}$ using a GCN. Consider the action graph $A_{t}$ at time $t$ with the corresponding clocked edges $(i,a,j,t_{s},t_{e},r_{t})$. Denote the layout for node $i$ at time $t$ by $\ell_{t,i}$. The GCN iteratively calculates a representation for each object and each action in the graph. Let $\bm{z}^{k}_{t,i}\in\mathbb{R}^{D}$ be the representation of the $i^{th}$ object in the $k^{th}$ layer of the GCN at time $t$. Similarly, for each edge in $A_{t}$ given by $e=(i,a,j,t_{s},t_{e},r_{t})$ let $\bm{u}^{k}_{t,e}\in\mathbb{R}^{D}$ be the representation of this edge in the $k^{th}$ layer at time $t$. These representations are calculated as follows. At the GCN input, we set the representation for node $i$ to be: $\bm{z}^{0}_{t,i}=[\bm{\phi}_{o(i)},\ell_{t-1,i}]$. And, for each edge $e=(i,a,j,t_{s},t_{e},r_{t})$ set $\bm{u}^{0}_{t,e}=[\bm{\psi}_{a},r_{t},\ell_{t-1,i},\ell_{t-1,j}]$.

All representations at time $0$ are transformed to $D$ dimensional vectors using an MLP. Next, we use three functions (MLPs) $F_{s},F_{a},F_{o}$, each from $\mathbb{R}^{D}\times\mathbb{R}^{D}\times\mathbb{R}^{D}$ to $\mathbb{R}^{D}$. These receive the subject, action and object representations and return three new representations. The updated $i^{th}$ object representation is then:⁵⁵5Note that a node can appear both as a “subject” and an “object” thus two different sums in the denominator.

Similarly, the representation for edge $e$ is updated via: $\bm{u}^{k+1}_{t,e}=F_{a}(\bm{z}^{k}_{t,i},\bm{u}^{k}_{t,e},\bm{z}^{k}_{t,j})$.

Finally, we transform the GCN representations above at each time-step $t$ to a layout $\ell_{t}$ as follows. Let $K$ denote the number of GCN updates. The layout coordinates of $\ell_{t,i}$ are the output of an MLP applied to $\bm{z}^{K}_{t,i}$, which are simply the set of the predicted normalized bounding box coordinates. The object descriptor of the the $i^{th}$ object is $\bm{z}_{t,i}^{K}$ and we denote the final object descriptors matrix at time $t$ as $\bm{z}_{t}\in\mathbb{R}^{O\times D}$. The predicted bounding box coordinates are trained via the $L_{1}$ loss as described in Section 2 of the main paper.

We elaborate on the Perceptual loss and Feature Matching loss from Section 4.2.

For the Something Something dataset (Goyal et al., 2017), we use the eight most frequent actions. These include: “Putting [something] on a surface”, “Moving [something] up”, “Pushing [something] from left to right”, “Moving [something] down”, “Pushing [something] from right to left”, “Covering [something] with [something]”, “Uncovering [something]”, “Taking [one of many similar things on the table]”. See Figure 10 for qualitative examples. The box annotations of the objects from the videos are taken from (Materzynska et al., 2020). For the CATER dataset we include the four actions as provided by (Girdhar & Ramanan, 2020). These include: “Rotate”, “Cover”, “Pick Place” and “Slide”. See Figure 9 for qualitative examples. For “Pick Place” and “Slide” which are otherwise random, we include the action destination coordinates.