Echo: A Large Language Model with Temporal Episodic Memory

As illustrated in Figure 3, the design of character cards encompasses seven attributes: ”Name,” ”Occupation,” ”Age,” ”Gender,” ”Hobbies,” ”Personality,” and ”Social Relationships.” Specifically, we randomly generated attribute values for all attributes except for ”Social Relationships.” Subsequently, we utilized the LLM to generate the ”Social Relationships” attribute values based on the other six attributes.

The plots generated by LLMs differ significantly from actual real-life scenarios. Therefore, we manually created an event library, from which 20 events are sampled to form a plot. The library contains three types of events: common events, real events, and hallucinatory events. Common events are designed to enable the model to generate data based on semantic memory questions and answers while enriching the context. They include routine occurrences in daily life, such as greetings, inquiries about common knowledge, and discussions about career-related issues. Real events are events that have actually occurred and are related to episodic memory. They are used to prompt the human role to ask the Echo assistant if it remembers a related event. Hallucinatory events are fabricated events that have never occurred. They are used to prompt the human agent to ask the AI assistant about non-existent events and simultaneously remind the AI assistant not to be misled. Notably, since all event prompts are removed during the training of the Echo model, hallucinatory events help reduce the LLM’s tendency to generate false information and enhance the model’s understanding and reasoning abilities regarding episodic memory.

In the design of environments, we initially considered only temporal information. We first established a series of time-stamped nodes arranged in chronological order (e.g., Monday, September 4, 2006, 21:42:56, Monday, September 4, 2006, 21:55:38). These time-stamped nodes are then automatically added to the conversation history between the human role and the Echo assistant, indicating the time at which each round of dialogue takes place.

As illustrated in Figure 4, we designed distinct prompt templates for both the human role and the Echo assistant. The highlighted sections in the figure are replaced with information from Section 3.1. Specifically:

• •

  Human Role Prompt: This includes the character card and all plot details, enabling the LLM to assume various human roles and engage in dialogues with the AI assistant according to different plots.

• •

  AI Assistant Prompt: This incorporates both hallucinatory plots and common plots. This setup helps the LLM acting as the AI assistant to reduce episodic memory hallucinations and proactively seek relevant information in a human-like manner.

Based on these prompt templates, we generate initial prompts for the human and the Echo assistant, denoted as $P_{u}$ and $P_{a}$, respectively.

Algorithm 1 provides the pseudocode for the data generation process. We initialize and maintain two separate history records, $H_{u}$ and $H_{a}$, for the human role and the AI assistant using initial prompts $P_{u}$ and $P_{a}$, respectively. In lines 4-12 of Algorithm 1, we alternately control the two agents representing the human and the assistant to engage in dialogue. Temporal information is incorporated during the conversation in lines 6-8. We check if farewell phrases such as ”goodbye” or ”talk to you later” appear in the response. If any of these phrases are detected, or if the number of conversation rounds exceeds 60, the stopping criterion is considered to be met, and the current data generation process is terminated. Finally, we remove the initial prompt $P_{a}$ from $H_{a}$ to obtain the final dataset, denoted as $Data$, which constitutes one piece of data in our EM-Train dataset.

We evaluate a series of LLMs on EM-Test, including the current state-of-the-art open-source and closed-source models. Particularly, we select LLAMA3-8b (Dubey et al., 2024), ChatGLM3-6B (GLM et al., 2024) for open-source models, and for closed-source models, we employ GPT-3.5-turbo (OpenAI, 2023), GPT-4 (OpenAI, 2023), ChatGLM3-trubo (GLM et al., 2024).

In MADGF, the LLM serving as the agent is Qwen2-72B-Instruct (Yang et al., 2024), which is a high-performance open-source LLM. We use chatglm3-6B (GLM et al., 2024) as the base model for Echo, and implement it with full fine-tuning.

In the quantitative analysis, we first collect the responses of LLMs at the test points, then ask human annotators to score these responses on a scale of 1 to 10. Additionally, we use the widely adopted Sentence Transformer model, all-MiniLM-L6-v2¹¹1https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, to encode the LLMs’ responses and the reference standard outputs provided by the test set, obtaining $\mathcal{E}_{\text{LLM}}$ and $\mathcal{E}_{\text{Standard}}$. We then calculate the cosine similarity $\mathcal{S}$ using the Equation 1.

We consider using the Pearson correlation coefficient, denoted by $\mathcal{R}$, to measure the correlation between the human score and the similarity metric. It is calculated using Equation 2.

where:

• •

  $x_{i}$ and $y_{i}$ are individual sample points indexed with $i$,

• •

  $\bar{x}$ and $\bar{y}$ are the mean values of $x$ and $y$ respectively,

• •

  $n$ is the number of sample points.

If the $\mathcal{R}$ value between two datasets is greater than 0.8, it is considered to be highly positively correlated (Cohen, 2013).

We present the results of LLMs on human scores and similarity metrics in Table 2 and Table 3, respectively. Our experimental analysis is provided from the following aspects: performance of several LLMs, human score and similarity metrics, comparisons across easy and hard levels, comparisons across different time spans, and consistency between human score and similarity metric.

Performance of Several LLMs. It can be observed that our Echo model achieved the best performance in both Human Score and Similarity Metric. Specifically, it scored 6.7 and 5.9 in the easy and hard levels of Human Score, respectively, and 84.0 and 74.5 in the Similarity Metric. Over different time spans, Echo nearly obtained the best or second-best scores across all metrics, except for the easy level ”One day” and ”Several Decades” in Human Score. These results indicate that the Echo model excels in EM capability. In contrast, among all models, the open-source ChatGLM3-6B performed the worst overall. As the base model of Echo, this indirectly demonstrates the effectiveness of the EM-Train data generated using the MADGF framework in enhancing a model’s EM capability. Moreover, GPT-4 also showed excellent comprehensive performance, achieving second-best scores in the easy level of Human Score and the hard level of Similarity Metric. In the hard level of Human Score, GPT-4 (5.4) narrowly trailed behind LLAMA3-8B (5.5), which had the second-best performance. In the easy level of Similarity Metric, GPT-4 (72.3) slightly lagged behind LLAMA3-8B (74.8), which also had the second-best performance.

Comparisons across Easy and Hard Levels. Most LLMs perform well on easy-level problems, but their performance drops at the hard level. Specifically, we found that all LLMs performed better overall at the Easy Level compared to the Hard Level. For instance, in Table 2, GPT-3.5-turbo scored 5.2 at the Easy Level, but only 4.0 at the Hard Level. Additionally, we observed that the average score for all models at the Hard Level in Table 2 was 4.5, which is 0.5 lower than the Easy Level (5.0). Similar results were seen in Table 3.

Comparisons across Different Time Spans. Models exhibit different performances across various time spans. Due to the inconsistency in model performance between the human score and similarity metric, we first consider the common performance under both metrics, then focus on the analysis based primarily on the human score. On the easy time span, we found that models perform better on ”several decades” questions. This is because the mean values for ”several decades” are the highest for all models according to both metrics (6.2 for human score and 82.8 for similarity metric). Meanwhile, we observed that models perform poorly on certain time spans (few days, few months, few years), which may be attributed to the difficulty models have in understanding these temporal concepts.

Consistency Between Human Scores and Similarity Metrics. We consider calculating the Pearson correlation coefficient $\mathcal{R}$ for two metrics to observe their correlation. The overall results of the Human Score (i.e., 2.7, 5.6, …, 6.7) and the Similarity Metrics (i.e., 57.0, 70.2, …, 84.0) are used to compute similarity at the Easy Level, and similarly for the Hard Level. Using Equation 2, we obtained Pearson correlation coefficients $\mathcal{R}$ of 0.935 for the Easy Level and 0.842 for the Hard Level, both greater than 0.8. Therefore, the results of the two metrics can be considered highly positively correlated. Given that the Human Score requires expensive manual evaluation, Similarity Metrics can be considered as a cost-effective alternative for subsequent model evaluations. Additionally, we found inconsistencies in the performance evaluation of models by Human Score and Similarity Metrics across different time spans. We calculated the Pearson correlation coefficient $\mathcal{R}$ between the mean values of the two metrics over different time spans, resulting in coefficients of 0.429 (moderate correlation) and 0.003 (very weak correlation) for the Easy Level and Hard Level, respectively. Our hypothesis is that the EM-Test may have insufficient data points for each time span, leading to inadequate statistical samples. This insufficiency results in inconsistent outcomes between Human Scores and Similarity Metrics.

We tested the EM capability of the models without considering time information, as shown in Table 4. We found that the models perform similarly whether or not time information is considered. Our Echo model and GPT-4 still performed well, achieving the first and second highest scores, respectively; while ChatGLM3-6B continued to perform the worst. These results indicate that the dataset EM-Train, obtained using our MADGF framework, can effectively improve the EM capability of models even when time information is not considered. In addition, we provide extended experiments regarding the model’s temporal awareness and reasoning capability in the appendix for further analysis.

We conducted experiments on the Echo model using real-life scenarios that require complex Episodic Memory abilities, as shown in Figure 7. Some dialogues unrelated to episodic memory have been omitted using vertical ellipses. These dialogues are intended to increase the challenge of improving the model’s Episodic Memory ability in long texts. Additionally, some content details unrelated to episodic memory skills have also been omitted. To test the model’s performance over longer time spans, all time information was manually provided.

The test dialogue on the left side of Figure 7 demonstrates that the model can accurately recall recent conversation content and timing, indicating its ability to understand time and associate it with events, which is a sign of Episodic Memory capability. In the test questions on the right side, the model shows even stronger Episodic Memory ability by recalling what human characters ate on a specific day from lengthy historical records, and understanding and judging whether conversations took place during a certain period.

We tested Echo using questions that do not require considering time information for responses, as shown in Figure 8. From the test dialogue, it is clear that Echo can accurately recall the human character’s favorite band and food, and provide relevant information even after multiple rounds of dialogue. Additionally, in the final round of test questions, Echo did not confuse any content that we had not actually told it, avoiding the hallucination issue. This problem often occurs when conversing with other LLMs.