VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language

VTS operations primarily involve responding to vessel inquiries and monitoring ships that may pose navigational risks. While basic information retrieval, such as checking vessel data via identifiers, is straightforward, the task of monitoring and identifying vessels that may pose navigational risks is significantly more complex. Operators must reason over large volumes of dynamic traffic data and detect subtle early-stage indicators of potential hazards.

To address this, we reformulate the task as a Text-to-SQL problem, a well-established paradigm in the fields of artificial intelligence, enabling the translation of natural language queries from operators into executable SQL queries over structured vessel traffic database.

Unlike existing CPA-based (Closest Point of Approach) alert mechanisms, which highlight vessels only when collision risk is imminent, our focus is on earlier identification of vessels of interest—such as those approaching congested areas or experiencing prolonged signal loss—allowing for more proactive supervision.

Besides, VTS operations must also enforce localized navigational rules that restrict certain vessel types within specific maritime zones—particularly in high-traffic or constrained areas where factors like size, draft, or navigation behaviors affect traffic. For instance, very large crude carriers (VLCCs) and deep-draft vessels may be barred from entering certain channels during specific time windows. While not always posing immediate collision risks, violations of such rules can disrupt traffic order or compromise safety. Identifying these vessels is thus a key part of proactive supervision and falls within our broader definition of vessels of interest.

To better reflect practical VTS scenarios, we refine the problem into a knowledge-augmented variant of the Text-to-SQL task, tailored to the operational query styles used in VTS.

First, maritime supervision involves rich domain knowledge that extends beyond traffic data schema. This includes specialized maritime terminology, localized navigational rules and procedures specific to particular sea areas, as well as time-sensitive documents such as Notices to Mariners and real-time traffic advisories. These materials are essential for correctly interpreting operator queries and ensuring accurate rule-based supervision. Given their dynamic and context-dependent nature, such knowledge must be incorporated through knowledge-augmented mechanisms that can provide relevant textual guidance during query understanding and reasoning.

Second, operator-issued queries in VTS scenarios are typically short, fragmented, and task-oriented, deviating from standard natural language patterns. These utterances often omit syntactic elements, contain domain-specific shorthand, or follow operational communication protocols rather than grammatically complete sentences. As a result, models must demonstrate robustness to varied, informal, and context-dependent language expressions in order to correctly interpret intent and generate accurate SQL representations.

To support development and evaluation, we construct a domain-specific multimodal dataset comprising: (1) a relational database, implemented by MySQL8.0 of structured vessel traffic data, as shown in Table I; (2) a retrieval corpus of related maritime textual knowledge; and (3) A query–SQL pair test set containing diverse linguistic expressions and their corresponding executable SQL statements. These components reflect practical VTS operations and support realistic benchmarking of knowledge-augmented Text-to-SQL models. Dataset is available at VTS-SQL.

The query set is designed based on input from professional VTS operators to reflect real-world operational scenarios, covering multiple query types, as demonstrated in Table II. And, in addition to the operational style, each query is further provided in two additional linguistic variants: a concise command-style format and a more formal natural-language expression, as demonstrated in Table III.

As the complexity and diversity in maritime terminology, we introduce an interactive Named Entity Recognition-based (NER-based) Relational Reasoning module aimed at clarifying and confirming relational semantics between the natural-language query entities and available database entities through iterative interactions.

The proposed module, shown in Figure 2, employs a hierarchical NER method tailored to maritime contexts, identifying and annotating entities in multiple semantic granularities. Broad categories include geographical regions, named types, and navigational facilities. Such hierarchical annotation provides structured representations for entities, significantly mitigating ambiguity compared to traditional flat NER schemes.

Recognized entities are subsequently utilized to build corresponding verification queries directed toward underlying spatial-temporal vessel traffic database. The query returns are then synthesized into structured, informative prompts, enabling LLMs to leverage CoT reasoning in determining and clarifying potential links between user-query entities and actual entities existing in database. For example, upon identification of entities such as strait, Singapore Port and Pilot Station verification queries are dynamically generated and executed to check the presence, relevance, and relational context of these entities within the database.

The maritime domain involves extensive expert knowledge with complex semantic implications, including navigation planning, vessel type categorizations, geographic topology constraints, and port operation regulations. To effectively integrate such domain expertise into the Text-to-SQL process, we propose an agent-based framework to systematically operationalize maritime-specific knowledge. As illustrated in Figure 3, our framework dynamically introduces domain knowledge by intelligently decomposing a complex natural language query into a structured series of domain-specific subtasks, thus ensuring semantic clarity and operational effectiveness in the generated SQL queries.

Text-to-SQL tasks differ significantly from traditional database queries in the maritime spatiotemporal context. These differences mainly stem from the complexity of geospatial semantics, the ambiguity of temporal constraints, and the specialized terminology in the maritime field. To this end, we propose a Semantic Algebra Intermediate Representation (SAIR) method to bridge the semantic[22] gap between natural language queries and structured SQL.

As shown in Figure 4, the SAIR module consists of three key submodules, which respectively handle named entity understanding and domain knowledge injection in orange module, structured semantic mapping in green module, and geographic function instructions in blue module. This design reflects the key improvement in our approach, which consists of decomposing the natural language parsing process into multiple substeps with clear structure and controllable semantics through explicit semantic modeling.

Considering that the initially generated SQL queries may contain ambiguity, inaccuracy or grammatical errors (especially in complex spatiotemporal contexts), we propose the Query Rethink (RT) Mechanism, a novel validation and correction step[23] inspired by the thought chain paradigm. Specifically, given a SQL query draft, the model carefully checks its semantic coherence and logical consistency through an iterative reasoning process. By actively reconsidering the correctness of the generated SQL with domain-specific semantics and relational structures, the RT module identifies and dynamically corrects problematic query elements, ultimately generating refined and validated SQL statements.

The experiments were conducted on a high-performance computing system equipped with an Intel(R) Gold 6336Y CPU and NVIDIA RTX 4090 GPU with CUDA 12.1 support. The operating system was Ubuntu 22.04.

Inspired by the Exact Match (EM) and EXecution Accuracy (EX) in SQL evaluation [24], we aim to develop a customized evaluation metric specifically for SQL queries in the VTS domain. Existing metrics rely heavily on the exact match or execution correctness of result tuples, and fail to adequately address specific challenges and characteristics observed in vessel of interest identification, such as frequent redundant selections or missing relevant entries. Additionally, to balance precision and recall in safety-critical scenarios, we introduce a novel evaluation metric with penalty coefficients.

Equation 1 reflects the base score. Where $G^{T}$ represents the true result set, $G^{P}$ represents the predicted result set, and $|\cdot|$ represents the count of unique sample.

To further penalize predictions that introduce unnecessary over-selected tuples, we introduce a penalty factor $P_{f}$ as defined in Equation 2.

Proposed Match Score $M_{s}$ in Equation 3 incorporates both matching and penalization conditions within an integrated metric.

Designed $M_{s}$ effectively addresses distinct characteristics of this knowledge-augmented Text-to-SQL task by balancing recall on avoiding missing correct tuples and precision on discouraging unnecessary duplication.

As shown in Table IV, we conduct the comparison experiment based on the queries of operational style in our dataset. For fairness, all compared models are given the same base knowledge and rules uniformly through prompt-based inputs. Details of prompt is available with code.

Results show that our VTS-LLM achieves an overall accuracy of 77.80%, significantly outperforming SOTA baselines. And also it demonstrates superior performance across various prompt formats, particularly accounting for 74.30% in Alpaca-style prompts, highlighting its strong generalisation and adaptability.

Besides, in Table V, our model maintains robust advantages over open-source and specialized structured-data LLMs, and surpassing two dedicated Text-to-SQL models, DAIL-SQL with GPT-4o and SQLCoder with LLama as backbone.

To further analyze the effectiveness of each proposed component, we conducted ablation experiments as shown in Table VI. We separately drop three key modules in VTS-LLM: NER-based Relational Reaoning (NER), Semantic Algebra Intermediate Representation (SAIR), and Query Rethink mechanism (RT).

The ablation study demonstrates the reduction of each core components in evaluation performance: the VTS-LLM agent (#1) integrating all components achieves optimal performance (77.80%); Without NER Reasoning modules (#2), the performance degrades, with accuracy dropping to 74.25%; Without SAIR modules (#3) the performance degrades, with accuracy dropping to 74.06%; Without RT modules (#4) the performance degrades, with accuracy dropping to 70.22%.

Furthermore, among all compared models, we select Claude, the best-performing model under the operational linguistic style, and conduct a sensitivity analysis across command-style and formal nature language style. Results in Table VII reveal substantial variations in model performance across linguistic styles. A consistent pattern is observed across all models that performance degrades as query expressions become more concise and less formally structured.

In terms of model comparison, VTS-LLM consistently outperforms other baselines across various linguistic styles, demonstrating stronger robustness to language style variation. Particularly in the Command-Style setting, which is most representative of real-world VTS scenarios, VTS-LLM achieves a score of 72.40%, significantly outperforming Claude-3.5 (47.53%) and Claude-3.7 (49.12%), both of which experience substantial performance drops in this challenging linguistic style.

1) VTS-LLM achieves SOTA performance in this knowledge-augmented Text-to-SQL task, outperforming general-purpose models by 12–35% and exhibiting superiority over dedicated SQL models as well. While these models demonstrate strong performance on general datasets like BIRD and Spider, they remain inadequate for handling domain-specific queries and knowledge-augmented data contexts. The strength of VTS-LLM stems from a set of domain-specific adaptation strategies, including a novel semantic algebra for resolving column ambiguities, entity-relation reasoning for complex query understanding, agent-based domain knowledge fusion, and query rethink mechanisms for enhanced generation accuracy.

2) A notable performance disparity in the sensitive analysis indicate that different linguistic styles in our dataset pose systematically varying levels of difficulty for Text-to-SQL models, highlighting the need for robust linguistic adaptability. Moreover, our VTS-LLM demonstrates stronger robustness to linguistic style variations, making it better suited to the unstructured and often fragmented nature of queries that are representative of real-world scenarios. The lower scores of Claude models in this category suggest that they may struggle with the lack of formal structure in such prompts, highlighting the importance of specialized optimization for industrial use cases where clarity and brevity are prioritized over grammatical formality.