DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory

One of the most common vision-language navigation (VLN) policies is training end-to-end models from scratch using offline datasets[5][6][7][8][9], but face fundamental limitations in generalization due to the prohibitive costs of collecting large-scale navigation data. These methods struggle to adapt to novel object configurations or unseen environments.An alternative line of work fine-tunes pre-trained vision-language models with robot-specific data [10][11][12][13], aiming to preserve semantic understanding while adapting to navigation tasks. However, this risks catastrophic forgetting of the original model’s capabilities, revealing the fragility of such approaches. The cost of fine-tuning is also non-negligible, requiring extensive data collection and annotation efforts.

Recent efforts have shifted toward zero-shot prompting of VLMs without fine-tuning, but these methods impose restrictive action spaces such as fixed angular increments and cardinal directions [7][14][15] or directions (”north,” ”south”, ”east,” ”west”) [2] that limit motion flexibility. This fixed-distance and fixed-direction action space either oversimplifies the situation or relies on extremely small distances or angles, causing the robot to make repeated small steps, which leads to an unacceptable time cost. In any case, such action spaces remain merely an idealized abstraction.

Visual prompting methods have emerged to enhance spatial reasoning in VLMs. Building on annotation techniques like Set-of-Mark [16], navigation-specific approaches overlay markers on input images to ground language instructions in visual features [17]. PIVOT [18] and VLMnav [19] represent this category, generating action proposals through arrow prompts. While these methods improve navigation performance, they still rely on fixed action spaces and lack consideration for hard constraints (walkable areas) and soft constraints (such as warning signs or task requirements).

Effective navigation in mobile robotics requires robust environmental memory systems that integrate sensory perception with spatial modeling. Traditional approaches build upon Simultaneous Localization and Mapping (SLAM) [20], which constructs geometric maps through sensor fusion and localization. While SLAM enables basic obstacle avoidance and path planning, its purely geometric representations lack semantic context, limiting applicability in human-centric environments and cannot be directly used for VLM.

This semantic gap has spurred development of augmented memory systems that layer object-level understanding onto spatial maps. Current solutions incorporate semantic information through two primary modalities: direct human interaction via dialogue systems [21], or automated annotation using machine learning models. The latter encompasses hierarchical 3D scene graphs encoding room connectivity[22], vision-language models generating textual scene descriptions [23], and region classifiers labeling navigable spaces in RGB-D streams [24]. Despite these advances, implementation barriers persist—particularly dependence on specialized sensors and compute-intensive hardware—which restrict real-world deployment scalability, especially unable to annotate the environment for a robot in real time. Existing methods thus face a fundamental tradeoff: geometric precision and semantic richness remain inversely correlated with practical feasibility across diverse robotic platforms.

Recent works like SayPlan [21] and Text2Map [25] demonstrate graph-based mapping approaches that encode spatial relationships through hierarchical structures. While we similarly adopt graph representations for location encoding, these methods rely on specialized prior knowledge—SayPlan requires pre-constructed 3D scene graphs (3DSG) of large environments, while Text2Map depends on human-provided navigation instructions for graph initialization. In contrast, our framework integrates memory construction as an inherent component of exploration, enabling VLMs to dynamically build graph-structured memory without prior environmental knowledge. This dual approach ensures robustness through self-supervised map generation while eliminating dependence on human annotations.

We introduce DyNaVLM, a navigation system that processes a goal $g$ (expressed in either natural language or an image), an RGB-D observation $I_{t}$ at every step, and the robot’s pose $\xi_{t}$ to determine an appropriate action $a_{t}\in A_{t}$. The agent maintains a memory function $\mathcal{M}_{t}=\Phi(\{I_{k},\xi_{k}\}_{k=0}^{t})$ that extracts spatio features from historical observations. The action space is parameterized in polar coordinates $a_{t}=(r_{t},\theta_{t})$, comprising a yaw-axis rotation and a forward displacement in the robot’s local frame. In summary, at every step $t$, our policy $\pi$ computes an action $a_{t}=\pi(s_{t},\mathcal{M}_{t})$, where $s_{t}=<I_{t},\xi_{t},g>$.

The Memory manager serves as the central middleware coordinating the interaction between the VLM and our graph-structured memory system. As illustrated in Fig. 2, this component maintains a dynamic knowledge graph (Memory Bank) that captures spatial relationships and semantic object information within the environment.

The graph structure of memory $G=(V,E)$ implements a hybrid representation where vertices $v_{i}\in V$ encode object instances with attributes {name, location, visual_features}, while edges $e_{ij}\in E$ model spatial adjacency and navigation accessibility. Upon receiving real-time perceptual input from RGB-D sensors, the manager performs two critical functions: (1) node embedding updates using VLM-generated object descriptors, (2) edge addition, where the VLM determines spatial relationships between nodes. The VLM can update the memory through functions

• •

  add_node(name, attributes): Create or update a node in the memory.

• •

  add_edge(start, target, relation): Add a directed edge between two nodes.

The manager exposes a structured query interface to the LLM controller through function templates including:

• •

  spatial_query(semantic_filter): Retrieves neighborhood subgraphs based on semantic constraints.

• •

  path_inference(start, target): Generates potential navigation trajectories.

This architecture facilitates efficient context propagation between the VLM’s visual grounding capabilities and the LLM’s symbolic reasoning, which is particularly crucial for handling ambiguous object references (e.g., ”find the charging station near seating areas”). Additionally, this structured memory can be easily converted back into natural language to describe objects and spatial relationships within the memory, serving as input for the VLM. Moreover, it enables seamless memory transfer and preservation.

Enabling a VLM to directly output navigation coordinates from an image is inherently challenging, as spatial reasoning is not explicitly encoded in standard language model training objectives. To bridge this gap without introducing task-specific training, we propose a candidate-based discretization strategy that simplifies the continuous search space into a finite set of uniformly sampled points. These candidate points are generated without prior environmental assumptions, ensuring minimal human bias and maximizing adaptability to novel scenes. The action proposal process involves two stages:

The Action Selector synthesizes geometric candidates, perceptual context, and retrieved memory to produce the final navigation action.

We adopt the same setup as VLMnav [19]. The agent has a cylindrical body with a radius of 0.17m and a height of 1.5m. It is equipped with an egocentric RGB-D sensor with a resolution of $(1080,1920)$ and a horizontal field-of-view of $131^{\circ}$. We use Gemini 2.0 Flash-Lite as the vision-language model.

Following  [2, 19, 27], we evaluate performance using:

• •

  Success Rate (SR): The fraction of episodes completed successfully.

  $$SR=\frac{N_{\text{success}}}{N_{\text{total}}}$$

  (6)

  where $N_{\text{success}}$ is the number of successfully completed episodes, and $N_{\text{total}}$ is the total number of episodes.

• •

  Success weighted by Path Length (SPL): A path efficiency measure penalizing unnecessary movement.

  $$SPL=\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{l_{i}}{\max(p_{i},l_{i})}$$

  (7)

  where $S_{i}$ is a binary success indicator for episode $i$, $l_{i}$ is the shortest path length from start to goal in episode $i$, and $p_{i}$ is the actual path length taken by the agent.

The proposed method was deployed on a Unitree Go2 quadrupedal robot. A RealSense D435 camera is used to acquire aligned RGB-D images, which is mounted at a downward angle of 10 degrees and a height of 85cm (related to the ground). A Unitree L1 LiDAR located beneath the robot’s head is used to provide local odometry for controlling the robot’s movement towards the target point. The experiments are deployed on a large flat floor containing open-plan offices of different sizes, meeting rooms, lecture hall and laboratory space. The overview of the scenarios are shown in Fig. 5.

In each task, the agent is prompted to find 1 or 2 nearest objects. We employed FastSAM [30] for the segmentation of the ground in the RGB image. For the determined target point, we project its position into the robot’s base frame using the aligned depth image. Coordinate transformation between the robot and the target position is updated from the local odometry to command the robot’s movement using the default motion controller of Go2. To reserve space for the movement, the L1 LiDAR is also used to detect surrounding obstacles. When an obstacle is detected within a cylindrical area whose diameter exceeds the length of robot by 15cm, the robot will first move a certain distance in the opposite direction of the obstacle, and then proceed towards the target.

The performance in real-world tasks are evaluated using the Success Rate (SR) metric mentioned in Sec. IV-B and the average cumulative distance (ACD) traversed across all succeeded tasks.

The success of real-world tasks is defined as the distance between the target object and the robot being less than 5m, with the target appearing within the last frame of camera. For the first target, the robot is allowed a maximum travel distance of 200m, which is approximately twice the distance required to traverse the entire experimental scenario. For the second target (if given), the robot is permitted an additional maximum travel distance of 100m.

We conducted five tasks in total, with three tasks requiring the agent to locate single targets: a water dispenser, a printer, and a trash can. In the remaining two tasks the agent were sequentially required to find two targets, specifically first the printer and then the trash can, and first the water dispenser and then the printer. To enhance the diversity of the tasks, we conducted three separate experiments for each task, initiating from distinct locations.

We compared the proposed method with the VLMnav baseline [19], ensuring fairness by starting from the same location and direction for the same experiment. The experimental results are shown in Table III. The superscript “1” denotes the target as the first instructional objective, while the superscript “2” signifies the target as the second instructional objective. The result indicates that our method notably outperforms the baseline in terms of both success rate and travel distance, with an particularly pronounced improvement in tasks requiring the identification of the second target.

Fig. 6 illustrates key frames from one of the tasks, where the agent starts from the laboratory space and gradually searches for the water dispenser. In the 4th frame, the proposed method chooses to walk towards the door because, through the memory manager, the agent determines that the current site is a laboratory space and therefore suggests leaving the site to find the water dispenser. Conversely, the baseline method demonstrates a propensity to navigate towards the faucet, which leads it into a loop and ultimately results in a failure to locate the target successfully.