Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation System

The well-being of PVI has been a major focus for researchers, leading to the continuous development of assistive technologies. Traditional aids such as white canes, guide dogs, and tactile paving have long been used to assist PVI in their life. Currently, updated versions of traditional assistive tools that integrate technology are emerging one after another, such as PVI assistance robots (Bhat and Zhao, 2022) and smart white canes (Khan et al., 2018; Ju et al., 2009). In addition, with the advent of the internet, mobile technology and wearable device, more advanced solutions have emerged, such as remote volunteer services (Yuan et al., 2017; Xie et al., 2023), where sighted volunteers provide real-time assistance, and specialized assisting systems (Boldu et al., 2020) designed to enhance mobility and understanding for PVI. In recent years, the integration of CV and AI has significantly advanced assistive technologies. An increasing number of smart assistive tools are being developed, and in addition, researchers are paying more attention to the collaboration between PVI and these tools and systems, as well as the user experience (Xie et al., 2024a). In this paper, we mainly focus on navigation tasks for PVI.

Researchers have developed various prototypes to support PVI in navigating both outdoors and indoors (Real and Araujo, 2019). These aids typically include two crucial features for independent mobility: obstacle avoidance and wayfinding (Rafian and Legge, 2017). Obstacle avoidance ensures that PVI can safely navigate their environment without encountering obstacles, often using traditional methods such as guide dogs and white canes. Wayfinding, in contrast, helps PVI to identify and follow a path to a specific location, requiring an understanding of their surroundings through digital or cognitive maps (Tversky, 1993), and accurate localization to track their movement within these maps.

The advent of smartphone-based applications like Google Maps (goo, 2021a), BlindSquare (BlindSquare, 2020), and others has significantly enhanced outdoor navigation using Global Positioning System (GPS) and mapping services such as the Google Maps Platform (goo, 2021b) and OpenStreet Map (ope, 2021). However, the accuracy of GPS can falter by up to ±5 meters (GPS.gov, [n. d.]), which poses challenges, especially in “last-few-meters” navigation (Saha et al., 2019). Indoor environments exacerbate these challenges due to poor GPS reception and the absence of detailed indoor mapping (Rodrigo et al., 2009; Li and Lee, 2010).

To address these challenges, researchers have suggested integrating GPS with other smartphone built-in sensors like Bluetooth (Sato et al., 2017)and Near-field communication (NFC) (Ganz et al., 2014), and creating rich indoor maps to capture environmental semantics (Elmannai and Elleithy, 2017). Despite the potential of these technologies, they require significant initial investment and ongoing maintenance  (Fallah et al., 2012; Bai et al., 2014; Pérez et al., 2017) to be effective and also depend on users carrying additional devices such as IR tag readers (Legge et al., 2013).

In recent advancements, the application of CV technologies has emerged as a cost-effective approach for enhancing indoor navigation (Budrionis et al., 2020; Yu et al., 2024). Using smartphones, CV-based systems can interpret visual cues like object recognition (Zientara et al., 2017), color codes, and significant landmarks or signage (Saha et al., 2019; Fusco and Coughlan, 2020). These systems can also process various tags like barcodes, RFID, or vanishing points for better navigation support (McDaniel et al., 2008; Tekin and Coughlan, 2010; Elloumi et al., 2013). However, the reliability of solely using CV for precise navigation for PVI remains insufficient (Saha et al., 2019). Our prototype integrates CV, particularly utilizing LiDAR, with a LLM to jointly provide assistance for PVI during their travels.

NaviGPT’s architecture is designed to seamlessly integrate various components, creating a robust and responsive navigation system for visually impaired users. The system consists of 7 primary modules: (1) User Interface (UI) Layer: A simplified, accessible interface optimized for voice and text inputs. (2) Navigation Engine: Built on the Apple Maps API for accurate location and routing services. (3) LiDAR Module: Utilize the LiDAR sensor of the device (iPhone 12 Pro and advanced models) to detect the distance information between the camera and the object. (4) Vibration Module: Invoke the device’s vibration module and use different vibration frequencies to inform the user about the proximity between the device and the object. The closer the object, the higher the vibration frequency; the farther the object, the lower the vibration frequency. (5) Image Capture Module: Utilizes the device’s camera to capture environmental images. (6) LLM Integration: Incorporates OpenAI’s GPT-4 for image description, information processing, and content generation. (7) Data Integration Layer: Combines inputs from various sources to provide comprehensive navigation assistance.

The UI of NaviGPT is designed with simplicity and intuitiveness in mind, catering specifically to the needs of visually impaired users. The interface consists of four main components strategically positioned for easy access (as shown in Figure 3): Map Interface: Located at the top of the screen, this simplified map view provides a visual reference for sighted assistants or users with partial vision. Voice Interaction Button: Positioned in the bottom right of the screen, this prominent microphone button allows users to easily activate voice commands and receive audio feedback. Camera Interaction Button: Placed at the bottom left of the screen, this button enables users to quickly capture images of their surroundings for AI analysis. LiDAR Detection: The LiDAR detection feature activates along with the always-on camera, with the detection area being the yellow square at the center. It can measure the distance between the camera and the object selected within the yellow square.

This minimalist design approach ensures that visually impaired users can interact with the system efficiently through touch and voice, reducing cognitive load and enhancing usability.

The core of NaviGPT’s intelligent navigation system lies in its unique integration of OpenAI’s GPT-4 with navigation (from Apple Maps) and image data (from user’s input):

Map Data Processing: Apple Maps API provides real-time location data, routing information, and points of interest (destination).

Context Generation: The system combines map data with photos taken by the camera and submits them to GPT-4 via an API. The map provides rich information for navigation tasks, such as the current location of BVI, the navigation route, and the destination, while the photos provide information about the current environment. This information offers the LLM extensive contextual details, enabling it to provide feedback based on the user’s purpose and current situation. Intelligent Response Generation: GPT-4 processes the query and generates natural language responses, providing navigation instructions, environmental descriptions, and safety alerts. Since each time of user input is dynamic, GPT-4 does not provide highly formatted feedback like that of traditional navigation systems (e.g., “In 200 feet, turn left”). However, we have used prompt engineering method to enhance the formatting of the LLM’s responses. This helps the LLM focus more effectively on handling tasks within the navigation context, allowing it to deliver targeted and efficient feedback.

Response Refinement: The system filters and refines the LLM’s output to ensure relevance and accuracy before presenting it to the user. As we mentioned before, we use preset prompt engineering methods to control the output. Unlike some LLM-powered applications such as Be My AI or ChatGPT, NaviGPT does not provide detailed descriptions of the images provided by BVI. This avoids lengthy responses, making navigation, an inherently dynamic task that requires some degree of real-time feedback, more concise and efficient.

NaviGPT’s workflow (see in Fig. 1) is designed to provide a seamless and intuitive experience:

Initialization: Upon launch, the application will request the necessary permissions (see in Fig. 2), including GPS location (for map navigation), camera (for photo interactions), photo library (for storing travel photos), microphone (for voice interactions), and speech recognition (for converting speech to text). It is important to note that these permissions are required to fulfill the app’s interaction and functionality needs, and the developers will not access this data in any way.

Destination Input: Users can input their destination via voice command or text. The speech is converted to text and processed by Apple Maps to extract the address. Apple Maps API is queried to validate the address and create a walking route. This interaction is similar to the process when the user individually uses Apple Maps. After this step, the user will continuously receive general navigation feedback from Apple Maps based on their current location. This feedback will include turn-by-turn directions, real-time updates on their route progress, and alerts for upcoming turns or changes in the path. The navigation will adjust dynamically as the user moves, ensuring they stay on the correct route, with the system providing appropriate guidance for reaching the destination.

Real-time LiDAR Detection and Dynamic Vibration Frequency’s Feedback: The LiDAR will remain active after NaviGPT starts, continuously and in real-time detecting the distance between the objects in the camera’s field of view (within the central yellow square) and the mobile device. The device will provide ongoing vibration feedback based on the proximity of the object. A dynamic vibration frequency curve is set according to the distance, with a usable threshold. Specifically, when an object is detected at 10 meters or more, the device will vibrate at the slowest frequency (once every 3 seconds). When an object is detected at 30 cm or less, the device will vibrate at the fastest frequency (5 times per second). Between 10 meters and 30 cm, the vibration frequency decreases as the distance increases, and increases as the distance decreases.

Environmental Data Capture: The user captures their surroundings using the camera button. In this step, when the user wants to explore their surroundings or notices an approaching obstacle through LiDAR detection and vibration feedback, they can quickly take a picture using the camera button in the UI. The photo will be automatically saved to their device’s photo album, and further processing will be carried out via an API call to GPT-4 for enhanced insights or contextual understanding.

Context Integration: The system aggregates: a) The captured image. b) Current location (from Apple Maps). c) Destination and next navigation step (from route planning of Apple Maps). This integrated data is combined and then transmitted to GPT-4 all at once.

LLM Processing and Response Generation: GPT-4 processes the multimodal prompt (examples are shown in Fig. 3(b), and 3(c)). It generates a natural language response that includes: (a) Description of the captured image. (b) Current location description. (c) Safety assessment of the immediate environment. (d) Next navigation instruction. (e) Any relevant warnings or additional information. If the captured photo has a poor angle or does not detect content related to navigation (such as in Fig. 3(a)), NaviGPT will still provide a response to the user’s input but will additionally prompt the user to retake a photo that is more suitable for navigation purposes. Notably, in this scenario, the GPT feedback audio will take priority over the general navigation prompts. Once the GPT feedback has finished playing, the regular navigation instructions will resume. Importantly, the navigation functionality will not be interrupted during the photo-taking process or while receiving the GPT feedback.

Continuous Monitoring and Updates: The system continuously updates the user’s location. It prompts for new image captures at key decision points or at regular intervals. The process repeats from step 3 to provide ongoing navigation assistance.

NaviGPT, developed based on Apple Maps, goes beyond the standard features of mainstream navigation systems like Google Maps and Apple Maps. While these platforms offer extensive functions, their complexity often overwhelms PVI. Designed primarily for sighted users, these tools require navigating intricate menus and visual prompts, which can hinder PVI users.

NaviGPT, however, is tailored specifically to PVI needs, prioritizing simplicity and accessibility. Instead of relying on visual elements, detailed maps, and text-based instructions, it combines LiDAR, vibration feedback, and contextual responses from a LLM, reducing the dependency on vision-based interactions. This allows PVI users to focus on navigation without processing complex visual data.

A key advantage of NaviGPT is its simplified UI, offering fast, spoken instructions with minimal distractions. Unlike existing apps that require multiple actions to configure settings or navigate cluttered screens, NaviGPT delivers clear and timely feedback. By providing focused guidance rather than excessive detail, it ensures a smoother, more efficient navigation experience for PVI users, making it easier to access essential information and move safely in dynamic environments.

By comparing NaviGPT with other LLM-integrated applications like Be My AI and Seeing AI, NaviGPT’s primary advantage lies in its seamless integration of a navigation system into the LLM interaction flow. It also utilizes LiDAR and vibration feedback to simulate a white cane, providing real-time feedback to PVI. This integration allows users to access key features, AI-based image identification, navigation, and road safety confirmation—without switching between apps, enhancing the overall experience.

Unlike Be My AI, which focuses on detailed image descriptions, NaviGPT is designed for daily travel and navigation, making it more efficient in recognizing and responding to the changing environment. PVI require quick access to safety information, such as detecting obstacles and signage, which NaviGPT provides in real-time. While Be My AI offers more detailed feedback, often exceeding 60 words, it typically requires longer wait and reading out time (over 5 seconds in tests). This level of detail may be unnecessary in dynamic walking scenarios, where rapid feedback is more valuable.

Table 1 compares feedback from Be My AI and NaviGPT. While Be My AI provides more detailed information, NaviGPT’s quicker, focused feedback ensures safe and efficient navigation through LiDAR and vibration alerts.