TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

Previous works have primarily focused on vision-based human action prediction. One line of research has focused on video-based action prediction, utilizing the recent past sequence of video frames to forecast future actions. Given the availability of large-scale action video datasets (e.g., Ego4D [2], Human3.6M [9], Home Action Genome [1]), many researchers have adopted machine-learning-based approaches [8, 7, 5, 4], often employing sequential models such as LSTM [6] or transformers [15, 14]. Some studies have integrated multimodal models to improve performance, combining video with other modalities (e.g., video-acoustic models [3], video-text models [11, 10]). Others have focused on object-based cues within the video to predict future actions [12, 13]. Another research direction involves predicting future motions from past motions, using only the human body’s pose data. For instance, [19] introduced a stochastic model for predicting diverse future actions with probability, while [17, 18] incorporated graph convolutional networks with attention mechanisms to improve prediction accuracy. Additionally, [16] proposed the use of a diffusion model to account for multi-person interactions.

In existing video-based action prediction research, a limitation lies in the spatial and temporal restrictions of scene context. Most studies focus on information within the limited field of view of ego-centric or third-person cameras, which only provides localized data. This narrow perspective hinders understanding of broader, more complex behaviors in real-world environments. Additionally, privacy concerns restrict the observation of individuals’ daily activities over extended periods in uncontrolled settings, limiting the ability to learn a diverse range of daily action patterns from existing datasets. To overcome these constraints, we propose leveraging large language models (LLMs), which encode vast everyday knowledge in natural language. By incorporating LLMs, we remove spatial and temporal limitations, enabling a more flexible and generalized approach to action prediction that captures broader patterns applicable to diverse real-world scenarios.

With the rise of large language models (LLMs), many studies have leveraged LLMs to analyze social human behaviors. Notably, [23, 22] made a significant impact on the field by simulating town-scale, multi-agent human daily life, incorporating social interactions between agents throughout the entire pipeline using LLMs. This work has inspired subsequent efforts to extend the scale and complexity of such simulations [24, 25, 26]. However, a notable gap exists between their work and ours, as their focus is primarily on macroscopic-scale social behavior simulations. Additionally, their approach centers on intention-conditioned action generation within these simulations, which contrasts with our goal of predicting human actions in real-world scenarios.

To the best of our knowledge, only a few studies have explored LLM-based human action prediction. [20] utilized an LLM to predict the next human activity by identifying the object in the environment that the person is most likely to interact with. [21] focused on LLM-based human action prediction to enable a robot to plan corresponding supportive actions. In their approach, they provided the LLM with object lists and action history to predict the next human action. However, while they account for scene context, they do not consider physical constraints such as human motion or the spatial structure of the environment.

Furthermore, vision-language models (VLMs)[34] have recently emerged as a prominent technology. These models possess the capability to recognize and understand both images and text, and they are utilized in a wide range of applications. Although they could potentially be applied to our task, pre-trained VLMs generally lack the ability to predict physical quantities under complex image constraints since they are primarily trained to establish correspondences between images and text. In contrast, our method can evaluate complex image constraints and predict the target object by employing a physically-aware prediction module that operates independently of the LLM module.

In daily life, individuals perform sequential actions while interacting with various objects. For instance, one might take a glass from a cupboard, fill it with a drink from the fridge, and then sit on the couch to drink. In this study, we focus on predicting the transition between these sequential actions, specifically forecasting the next action as a person moves toward the target object. We assume the scene is a room-scale home environment where we can observe various scene states except for the persons’ internal states (Fig.2). The input comprises both text-based and image-based scene contexts. The text-based context includes semantic details—such as the time of day, object lists, a person’s action history, and conversations—while the image-based context provides physical scene information through a 2D image that shows an object layout map and the person’s trajectory. In our task, we first predict the target object using both contexts and then forecast the future action based on this prediction.

Fig.1 illustrates an overview of our approach. We propose a multi-modal action prediction framework that incorporates both an LLM and human trajectories. The core idea is to integrate two different perspectives—physical and semantic factors—through an object-based action prediction framework. Our framework consists of two primary steps: target object prediction (III-C) and action prediction (III-D). In the target object prediction step, we first utilize a LLM to predict a person’s target object based on the input scene context, generating a probability distribution over potential objects in the room from a semantic perspective (III-C1). Subsequently, we incorporate the person’s past trajectory to infer their likely destination, applying physical constraints to refine the prediction of the next target object (III-C2).

In the action prediction phase, the individual’s subsequent action is predicted using a LLM, based on the identified target object.

Further details of the approach are provided in the following sections.

At this stage, we employ the LLM to predict the action corresponding to the target object identified in the previous step. The input to the LLM includes the scene context and the predicted target object, and the LLM outputs the most plausible action the person is likely to perform within the given scene context.

To evaluate our multimodal method, we required input data with a text prompt describing the scene context, trajectory, and scene map. As no existing dataset met these criteria, we constructed a new dataset for evaluation.

Fig.5 illustrates the pipeline we employed to generate the evalution dataset. We utilized the Habitat-Matterport 3D Semantics Dataset (HM3DSem) [27] to obtain scene map and object data in home environments. HM3DSem provides a diverse set of 3D models of home environments, with semantic labels. From HM3DSem, we extracted scene maps represented as binary grids, where a value of 1 indicates areas walkable areas (within 0.3 meters of the floor height), and 0 corresponds to all other regions. These maps have dimensions of 256 x 256 pixels, mapping to a physical space of 10m x 10m. Additionally, we identified the pixel regions corresponding to each object within the scene map.

Furthermore, we manually designed daily action scenarios, detailing factors such as day/time, persona, current location, target object, subsequent actions, conversations, and more. These scenarios were transformed into text prompts for input to the LLM. We generated a trajectory from the current location to the target object by leveraging a model trained on the LocoVR dataset, which incorporates the scene map to guide the path (visit our website[32] for the detail).

Finally, we obtained 67 data pairs from the combination of 10 action scenarios and 9 scene maps. The total number of pairs is lower than the combination count because cases where objects from the action scenarios were absent in the scene maps were excluded.

Since action prediction using room-scale scene context is a novel task, few existing studies could directly apply to our work. Therefore, we introduce the following three baseline methods as benchmarks:

LLM: In this method, the LLM predicts the target object and its corresponding human action based on a text-based scene context. The input comprises day/time information, action history, conversation, and an object candidate list. The LLM produces scores for each target object and selects the predicted action associated with the highest-scoring target object. This approach is equivalent to the LLM-based action prediction modules described in [20, 21], though our implementation differs because the original codes were unavailable.

VLM: We employed pre-trained VLM models to benchmark baseline performance on multimodal data. The VLM input consists of a text-based scene context and an image that displays the scene map and the trajectory. In the image, each object and trajectory is uniquely color-coded, and a color list of the objects (e.g., sink: cyan, table: gold, trajectory: red) is included in the text-based scene context to help the model accurately identify object locations and trajectories. The VLM then outputs scores for the target objects and selects the corresponding action for the most plausible object, similar to an LLM.

Trajectory: We evaluated a standalone trajectory-based method, which is a component of our approach. This serves as an ablation study to assess how our method improves performance on the target object prediction compared with standalone trajectory-based method.

For both LLM and VLM, we used two types of pre-trained models—ChatGPT-4o-mini and Llama3.2—to evaluate performance across different model architectures.

We provided three types of text-based scene context as input data to assess the model’s robustness under degraded conditions: (1) full scene context, (2) scene context without conversation, and (3) scene context without conversation and past action history. These variations simulate real-world scenarios where certain information may be unavailable. The evaluation examines how our method mitigates performance degradation, demonstrating its robustness to noise in practical settings. Additionally, we evaluate performance under three trajectory progress conditions: d¿1m, d¿2m, and d¿3m. The performance differences across these conditions reveal how the trajectory information contributes in each method.

We conducted evaluations on both target object prediction and action prediction tasks. For both tasks, we conducted three trials for each evaluation and used the average of these trials as the final result to ensure the reproducibility.

Target object prediction: This task serves as an essential intermediate module for action prediction. The objective is to identify the ground truth target object from a set of 30–40 objects in each scene. For evaluation, we used the top-5 accuracy metric, which considers a prediction correct if the ground truth object appears among the top five predicted objects.

Action prediction: In this task, we evaluate action prediction accuracy by computing the cosine similarity between the predicted actions and their corresponding ground truth values using the SBERT library.