Social Cue Detection and Analysis Using Transfer Entropy

It is broadly recognised that social cues play an important role in communication. However, for social cue detection, researchers usually predefine a fixed set of social cues composed of known postures and gestures such as nodding or head shaking (Urakami and Seaborn, 3 15)(Bousmalis et al., 2 01)(Bremers et al., 2023). NovA (Baur et al., 2013) is a well-developed system for nonverbal signal detection, which detects postures and gestures using event-based gesture analysis (Kistler et al., 7 01), and classifies them into a predefined set of cues. This system also provides a movement expressivity measurement. However, these measurements are not used for cue detection.

Social cues are widely used in robotics, especially for Human-Robot Interaction (HRI). Tomari et al. proposed a socially-aware navigation planner for wheelchair robots, tracking the head orientations of the participants to estimate personal space assuming humans are more protective of the space in front of them (Tomari et al., 2014). Hansen et al. developed an adaptive system for natural interaction between mobile robots and humans based on the person’s pose and position, which estimates the interaction intention of the user and then uses it as a basis for socially-aware navigation based on a person’s social space (Tranberg Hansen et al., 2009). Escobedo et al. used the commonly visited destinations of a wheelchair robot’s user to estimate the probable intended destination of the user and then accept the user’s face and voice commands for navigation (Escobedo et al., 2014). These works are dependent on the proxemics theory proposed by Hall, who defined general social zones of humans (Hall, 1966). However, the concept of social zones relies on averaged heuristics. Hall validated his study only for US citizens (Rios-Martinez et al., 2015), which means these metrics may not translate well to other cultural contexts. Many human-following robot designs use social cues, e.g., in (Mi et al., 2016) where body orientation is used to estimate the intended turning direction. In (Moustris and Tzafestas, 2016), the relative position between a human and a robot is used as a feature to anticipate the human. Hu et al. (Hu et al., 4 04) use human orientation as an input for anticipatory robot behaviours. In general, most robots are designed to act on heuristic or manually selected social cues, and it is unclear whether these social cues are reliable or repeatable. Consequently, most current research focuses on detecting these cues to predict intent or behavior.

Cue detection usually requires tuning and is not necessarily repeatable or cross-cultural. There are no examples of research where interaction cues are automatically sourced. We tackle these issues in this work. The perceptive social cue detection framework proposed here identifies social cues both spatially and in time from raw data, requiring no predefined cue sets and comparatively less manual specification. Since it is data-driven, the framework is agnostic to social norms or heuristics and can be applied to any context where motion data can be tracked.

The study of social cues is related to intent communication, which refers to behaviours that allow an observer to quickly and correctly infer the intention of the agent generating the behaviour (Busch et al., 2017)(Dragan et al., 2013). The research in (Lichtenthäler et al., 2012) shows that motions that communicate intent increase perceived safety during virtual human-robot path-crossing tasks. In (Busch et al., 2017), the authors consider robustness, efficiency and energy as universal costs in a reinforcement learning scheme, with results showing an increase in the ability of a human to interpret a robot’s intention. Research studying projected visual legibility cues (Hetherington et al., 2021), has shown that projected arrows are generally more interpretable than flashing lights in a navigation setting. Importantly, the ways people communicate and understand intent are different, and there is no universal method to measure the communication of intent. However, if communicating intent is interpreted as the transfer of information between agents, this concept can be captured by the proposed framework, which could be used as a standard for measurement and to guide the selection of robot motions or actions.

TE is a measure that allows the analysis of the information transfer and potential causal relationships between two simultaneous time series. In the economics literature, Baek et al. use TE to analyse the market influence of companies in the U.S. stock market (Baek et al., 2005). He and Shang (He and Shang, 2017) compare different TE methods for analysing the relationship between 9 stock indices from the U.S., Europe and China. TE has been used to analyse animal-animal or animal-robot interactions such as (Shaffer and Abaid, 2020)(Porfiri, 2018). It has also been adopted to analyse joint attention (Sumioka et al., 2007) and model pedestrian evacuation (Xie et al., 2022). Berger et al. apply TE in robotics to detect human-to-robot perturbations using low-cost sensors (Berger et al., 2014). The above studies succeed in quantizing the information transfer or the relationship between their targets using TE, but tend to focus on specific features or aspects of interest. Our work seeks to provide a more general framework for detecting and analysing perceptible social cues, by looking for changes in information transfer over time and in space.

Mutual information (MI) is an information measure that captures the shared information between random variables. Klyubin et al. proposed the concept of empowerment for intrinsic motivation for reinforcement learning, which measures the maximum MI (channel capacity) between the agent’s actuation and their sensors (Klyubin et al., 2005). This is expanded by Mohamed and Rezende with a lower complexity maximisation approach to MI (Mohamed and Rezende, 2015). Jaques et al. use MI as a social influence reward for multi-agent deep reinforcement learning in Sequential Social Dilemmas (SSDs) (Leibo et al., 2017) to encourage collaborations between agents (Jaques et al., 2019). While MI measures the shared information, TE measures the time-asymmetric information transfer. Unlike MI, the asymmetric property of TE allows us to analyse the directionality of information flow, which is beneficial for analysing the exchange of social cues. By applying TE, we aim to quantize social cue transfer to better understand and use social cues for robots. In addition, most TE-related research reduces multi-dimensional features down to a single dimension to compute TE, while in practice, most features are multi-dimensional. We tackle this issue by using neural networks to build the probability distributions needed to estimate TE.

Transfer entropy is closely related to the concept of Wiener-Granger causality²²2These are equivalent when linear Gaussian models are used. (Granger, 1969). TE, $T_{Y\rightarrow X}$, can be defined as the conditional mutual information between two variables $X_{t}$ and $Y_{t}$ (Bossomaier et al., 2016), which is formulated as follows.

Here, $I$ denotes the mutual information and $H$ Shannon’s entropy (Shannon, 1948). This equation defines the information transfer from $Y$ to $X$, where $t$ is the time, and $k$, $l$ are the history length of $X_{t}$ and $Y_{t}$. In some literature, $X$ is termed the target and $Y$ the source. TE can be intuitively interpreted as the reduction of uncertainty in a state $X$ predicted solely based on its own history when an additional information source $Y$ is introduced (Bossomaier et al., 2016). The key idea of the proposed framework is to use TE to measure the information transfer of perceptible social cues.

1. Preparation. We start by identifying a target $X$ and source $Y$. In social cue analysis scenarios, the source can be a feature that transmits cues of interest (e.g., a pedestrian’s head orientation), and the target features that would be influenced by the cues (e.g., another pedestrian’s pose). Next, time series data is collected for both the target and source. Prior to modelling, we use Takens’ delay embedding (Takens, 1981) to create higher-dimensional embeddings for the time-history series of features,

where $d$ is the history length and $\delta$ is the unit time step. The history length for the target and source do not necessarily need to be the same, one can also denote them separately such as $k,l$ in Eq. (1). In general, the history needs to be long enough to capture the temporal relationship between cue cause and effect. In our experiments, we match this with estimates of human response times.

2. Model the Target. We next establish a baseline model, conditioned on only the history features of the target. The framework allows the application of different probabilistic prediction models. For instance, we can use a simple linear vector autoregressive model (VAR) to model the target $\mathbf{X}_{t}$.

where $c$ is the constant intercept of the model, $\alpha$ is the time-invariant matrix that matches the dimension of $\mathbf{X}_{t}^{(d)}$ and $\epsilon$ an error term. We can also model the target using other modelling techniques such as a multilayer perceptron (MLP), a Gaussian process or potentially more complex neural network architectures. Regardless of the underlying model, we intentionally form the target output distribution as a conditional multivariate Gaussian with mean $f(\mathbf{X}_{t}^{(d)})$ and covariance $f^{\mathbf{\sigma}}(\mathbf{X}_{t}^{(d)})$,

This simplifies later TE calculations. After obtaining the base model conditioned only on the target history, the source is included to build a second model of the target. For example, assuming a vector autoregressive model, we obtain

Here, we illustrate the process using a vector autoregressive model, but models that capture non-linear effects could also be used. We use neural networks to model behaviours for the experiments below.

3. Compute and Analyse Transfer Entropy. Modelling targets using conditional multivariate Gaussian random variables allows Shannon’s differential entropy to be calculated as follows,

Here, $D$ denotes the dimension of the target variable. Using (6) to calculate the entropy for each model, one can calculate the TE from the source to target as

Finally, we can analyse the information transfer from the source to the target using the measured TE.

The signals received from agents are usually in continuous form. Researchers often discretise these continuous signals in order to statistically compute TE using histograms or other frequentist approaches (Berger et al., 2014)(Orange and Abaid, 2 01). Kernel density estimation has also been recommended in (Schreiber, 2000) and frequently used in the information analysis toolbox (Lizier, 2014), a popular tool for modelling TE in economics. However, these methods require a large amount of data to build sufficiently dense probability distributions, especially if the feature dimension is high (Orange and Abaid, 2 01). Therefore, they are not helpful for local time series analysis with a limited number of samples, which means these methods are not suitable for computing local TE in order to analyse information transfer spatially or temporally. Even if we have sufficient data, building high-dimensional density distributions is not computationally efficient, which means these approaches are unsuitable for online algorithms. To tackle this issue, we use neural networks in our framework to estimate distributions, and then calculate entropy directly from the estimated conditional probabilities. This allows us to estimate TE locally and continuously, which is also computationally more efficient and achieves spatial and temporal information transfer analysis. It should be noted that since our data is continuous and entropy is computed using Shannon’s differential entropy, the transfer entropy can be negative indicating the absence of information being communicated.

A graphical illustration of the intuition underlying perceptible cue detection is given in Fig. 2. We use the TE to detect perceptible social cues.³³3We provide an example at https://github.com/jhy9968/TEscd.git Positive TE means the information transfers from the source to the target. However, there can be continuous information transfer during social interactions. We are most interested in the set of special events that trigger observer responses. Therefore, to meet the criteria of a perceptible social cue in our framework, we require TE measures to meet the requirements that: 1) the TE has to be larger than zero indicating general information transfer; and 2) the TE has to be larger than a threshold to be identified as a perceptible cue. To determine the threshold, we use double exponential smoothing (DES) (Gardner Jr., 1985) to find the moving mean and standard deviation of the TE. Let us denote $\mu_{t}$, $\sigma_{t}$ and $T_{t}$ as the moving mean, moving standard deviation of TE and current TE at time $t$. The DES works as follows:

Here, $b_{t}$ is the smoothed trend at time $t$, $\alpha$ is a smoothing factor and $\beta$ a trend smoothing factor. Then the perceptible cue threshold is computed as

where $\gamma$ is a tunable parameter. We use $\gamma=3$ in the work. The thresholding process filters out TE trends, which represent the accumulated information during continuous information exchanges. TE regions that meets the requirement $TE_{t}>0$ and $TE_{t}>threshold_{t}$ indicate perceptible social cues. In order to further remove the effect of the trend, we apply a 1st order high-pass filter to the TE before cue detection. Since human reaction time to visual cues approximately ranges from 0.18-0.9s without distractions (Fugger et al., 1 01)(Hermens et al., 1 01), the critical frequency of the high-pass filter is set to 1Hz in this work. In the current framework, the selection of $\alpha$ and $\beta$ is related to the time constant of DES, which depends on the expected cue period. The time constants for smoothing $\tau_{\alpha}$ and trend smoothing $\tau_{\beta}$ are calculated as follows:

where $\Delta T$ is the sampling time interval of the data. A basic principle is that $\tau_{\alpha}$ should be roughly the length of one cue in the setting, and $\tau_{\beta}$ should be generally shorter than $\tau_{\alpha}$.

To appraise the ability of TE to study social interactions, we first study a group-joining activity. The CongreG8 dataset (Yang et al., 2021) contains 380 full-body motion trials of free-standing conversational groups of three humans and a newcomer who approaches the groups with the intent of joining them. Four participants play a game called Who’s the Spy. The game contains three group members and one adjudicator (the newcomer). Three group members gather at the centre of the arena, two are given cards with the same item, the third is given a card with a different item. Group members take turns describing the item on their card. Meanwhile, the adjudicator walks around the arena listening to the group members’ discussion. Once the adjudicator has identified a spy, they join the group and point out the spy. The interactions are grouped into two categories. When the adjudicator approaches the group and the group members accommodate the adjudicator (e.g., a group member moves to make space), the trial is labelled as Welcome. Alternatively, if the group members stand still and ignore the adjudicator, the sample is labelled as Ignorance. Due to the high level of freedom within this experiment, participants’ behaviours are largely different between each trial resulting in large variance in the sampled data.

We apply the general workflow (3.2) to the Welcome and Ignorance scenarios separately to compute the TE for each trial for two directions (from the adjudicator to the group members, and the reverse) and study the peak values of TE. We set the position of adjudicator or group members as target features depending on the direction of transfer being investigated. We consider the group members as a whole, so a single target/source is defined for all the group members. A re-sampling rate of 10Hz and history length of 10 with a unit time step of 0.1s for the embedding are applied. Since this is a relatively more complex dataset, a Gaussian Emission Variational Autoencoder (VAE) (Kingma and Welling, 2022) with a two-hidden-layer encoder, a two-hidden-layer decoder and a latent space with 8 nodes is used to model the distributions over targets.

In this experiment, the adjudicator leads the group-joining event. Depending on the scenario, the responses from the group members are different. For the Welcome scenarios, group members should respond more to the perceptible social cues from the adjudicator when compared to the Ignorance scenarios. Therefore, we expect to see a difference in peak TE from the adjudicator to the group members because we assume that the peak TE occurs during the joining process when the majority of information transfer from the adjudicator to the group happens. Therefore, we hypothesize that: The peak values of TE from the adjudicator to the group members should have large differences for the Welcome and Ignorance scenarios, when compared with peak TE in the other direction (group members to adjudicator).

To test this hypothesis, we calculate the peak TE values of each trial for both scenarios and both information transfer directions. Then we conduct a two-sided t-test between the two scenarios for both directions. This test (Table 1) shows a significant difference between Ignorance and Welcome settings in the adjudicator to group member direction, but not from group members to the adjudicator, supporting our hypothesis. This also supports our proposal that TE can correctly quantify information transfer in social settings.

We test the framework’s ability to identify when perceptible social cues occur in a human-human handover scenario. We use a dataset (Carfì et al., 2019) comprising over 1000 recordings collected from 18 right-handed volunteers performing human-human handovers. 6-axis inertial data was acquired from two smartwatches on participants’ wrists. During their single-blind experiments, a volunteer (receiver) and an experimenter (giver) form a pair. The two participants start from the diagonal corners of a square experimental area and walk towards the centre of the square to perform handovers. Several strategies are applied during the handovers:

• •

  Normal: experimenter gives the object in a normal fashion.

• •

  Quick: experimenter moves their arm faster.

• •

  Delay: once the volunteer initiates the transferring gesture, the experimenter keeps their arm still ($\approx 2$s) before reaching towards the volunteer to give the object.

• •

  Holding: experimenter holds the object in place ($\approx 2$s) after both persons have touched it, i.e., they do not release the object once the volunteer has grasped it.

• •

  Wrong pose: as the volunteer initiates the transfer gesture, the experimenter unexpectedly moves their arm towards the volunteer’s left shoulder, an unnatural pose for right-handed volunteers.

The original sampling rate of the dataset is approximately $7Hz$, which we interpolated to a frame rate of $115Hz$ to increase the number of window samples to analyse. We used wrist angular velocity data from experiments where the same object, a ball, is used. In this experiment, the magnitude of the triaxial angular velocity is the only feature analysed for perceptible social cues. We use a history length of 4 with a unit time step of 0.14s for the Taken’s delay embeddings. These numbers were selected empirically but could be chosen based on model predictive quality. The average human reaction time to detect visual stimuli is approximately 0.18-0.20s (Thompson et al., 1992), so 0.4s is a reasonable window to capture human reactions. For cue detection, we set the $\alpha=0.005$ and $\beta=0.01$, which gives us $\tau_{\alpha}=1.7s$ and $\tau_{\beta}=0.9s$. We expect that perceptible cues occur when the handover starts and ends with the giver initiating the handover and the receiver reaching in response to their cues (Ortenzi et al., 1 12).

To demonstrate the ability of the framework, we detect the perceptible cue regions for all the trials using the proposed framework and generate plots that visualise the frequency of occurrence of perceptible cues at each frame for different strategies and information transfer directions. The results are shown in Fig. 3. We trim the data so the frame consistently starts when the distance between the two participants is 1m. In a regular handover scenario, two main social cues should occur at the start and end of a handover respectively (Ortenzi et al., 1 12). We also expect the cues from a giver to be followed by the cues from the receiver. This is visible in the normal strategy result, with two high cue count regions in both directions. The first and second regions correspond to the start and end of the handover respectively. We can see that high cue count regions in the receiver-to-giver (R2G) direction follow the giver-to-receiver (G2R) direction. Using the normal strategy as a baseline for comparison, we can also analyse the other scenarios. In the quick scenario, due to the faster movements, the cues occur and end faster when compared to the normal strategy; i.e, the spacing between cue counts is comparatively shorter. The fast movement of the giver could also confuse the receiver, resulting in a wider spread of the receiver’s reaction cue. In the delay scenario, the participant was not given instructions on how long to delay during the experiment, thus the time of the starting cue of the handover is widely distributed. We observe a large delay before the first high cue count region appears, and the cues are more widely distributed. For the holding strategy, the time gap between the starting and ending cue should be longer compared to normal. This larger gap is observed between the first and second high cue count regions in both directions. For the wrong pose scenario, the wrong pose of the giver could confuse the receiver, resulting in weaker responses from the receiver. Therefore, the plots appear to be similar to the normal scenario. However, we observe the first high cue count region in the G2R direction has a lower cue occurrence frequency compared to normal. This analysis demonstrates that the proposed framework can identify when perceptible social cues occur. Additionally, interesting behaviour patterns can be extracted from the TE analysis, which can be useful for downstream anthropomorphic and personalized robot interaction design.

We next apply the proposed framework to spatially analyse perceptible social cue transfer in a socially-aware navigation scenario requiring substantial non-verbal communication.

To maximise the exchange of perceptible social cues, we designed a leader-predictor front-following task simulating a person-following robot scenario (Fig. 4c) in a simulated intersection assembled using retractable barriers in a Vicon motion capture arena (Fig. 4b). The arena has a diameter of approximately 5.0m, and the junction size is a 1.6m$\times$1.6m square to permit comfortable side-by-side walking (Doss, 2023). The scenario involves two human participants: the predictor A and the leader B. At the beginning of each trial, B is asked to secretly select one of the destinations shown in Fig. 4a. A is asked to actively stay in front of B and attempt to reach the unknown destination in advance of B. The starting position and formation are illustrated in Fig. 4a. This design avoids potential bias where the initial pose of participants might influence their preference after entering the intersection. When each trial begins, B walks naturally towards the selected destination, treating A as an unknown pedestrian (i.e., keeping a comfortable distance, no direct communication, but still following general social norms such as adjusting speed to avoid collision). After entering the intersection (the black circle in Fig. 4a), A can move freely, provided they do not interfere with B’s progress. Direct communication channels such as speaking or hand signals are forbidden, forcing participants to communicate via more subtle motion cues. This ‘front-following’ task involves high levels of observation, prediction and collaboration, providing an interesting setting to analyse the use of social cues. Seven pairs (14 individuals aged 19 to 31, 3 females, 11 males) were recruited from the Monash University Clayton campus. Participants provided consent at the start of the experiment ⁴⁴4Approved by the Monash University ethics committee. Project ID: 33090.

The Vicon system provides a default frame rate of around $200$ Hz. Vicon markers on a cap and a belt are used to track A and B’s head and hip pose, with the centre position and the head and hip orientation recorded for each participant.⁵⁵5The data is publicly available at https://doi.org/10.26180/24719034.v1 Head and hip velocity are post-calculated using the positional information. Orientation and velocity vectors are projected onto the xy plane and then normalised. To apply the proposed framework, we take position, velocity, head and hip orientation as potential sources of cues and set each participant’s position as the target. A variational autoencoder model is fit to the target using embeddings with a history length of 4 and a unit time step of 0.1s. For cue detection, we set the $\alpha=0.01$ and $\beta=0.05$, which gives us time constants $\tau_{\alpha}=0.5s$ and $\tau_{\beta}=0.1s$. Compared to the handover task, the cues in this experiment can be quicker, subtler and noisier. Our hypothesis is that perceptible cues appear in locations where a decision needs to be made or communicated about travel direction.

Similar to the handover experiment, we detect perceptible cue regions for all the trials using the proposed framework and generate plots that visualise the perceptible cue frequency at each spatial location in a 2-dimensional grid space representing the experiment arena. The results are shown in Fig. 5. We observe that for the leader to predictor (B2A) direction, most high-frequency cue regions appear in the region leading to the intersection. For the predictor to leader (A2B) direction, high-frequency cue regions often appear inside or slightly before the intersection. This outcome is consistent with the leader’s role in the front-following task, who first transfers cues to the predictor (B2A) prior to entering the intersection. This communication is usually followed by cues confirming the travel direction sent by the predictor (A2B) before leaving the intersection. This pattern of cue communication in physical space is clearly shown in Fig. 5. Although in theory, we would expect the perceptible cue regions to be similar for tuning left and right scenarios, the results show that they have noticeable differences. Most predictors (A) preferred to stay in the top left corner of the intersection while waiting and observing the leader (B) during the experiment ⁶⁶6This experiment is done in Australia where the norm is to keep left on sidewalks.. This matches with post-experiment survey results that showed 42.9% of the predictors usually prefer to walk on the left side of the road or a corridor (28.6% right and 28.6% no preference). Therefore, physical space occupancy could be an influential factor for information transfer. This highlights the potential of the framework to capture information about cultural navigation biases.

The analysis above shows the proposed framework can identify where perceptible social cues occur.