CleanS2S: Single-file Framework for Proactive Speech-to-Speech Interaction

Traditional text-based chatbots, while effective in structured scenarios, inherently lack the dynamic expressiveness of human conversation. Speech interaction introduces critical advantages: (1) naturalness, mirroring real-world dialogue through prosody, pacing, and spontaneous turn-taking; (2) accessibility, enabling hands-free operation for users; and (3) efficiency, bypassing laborious typing for rapid idea exchange. These attributes position speech-to-speech chatbots as pivotal tools for applications ranging from assistive technologies to immersive human-computer interaction.

As illustrated in Figure 1, CleanS2S implements a cascaded pipeline comprising three core components. The ASR model processes raw audio input into transcribed text, which is then contextualized by the LLM to generate coherent text responses. Finally, the TTS module converts the LLM’s textual output into natural-sounding speech. To ensure compatibility across diverse open-source implementations, the system employs standardized interface specifications. All ASR/TTS components are required to process 16kHz audio streams and utilize UTF-8 encoded text. WebSocket strictly follow predefined encoding/decoding protocols.

Besides, a hallmark of human-like conversation is the ability to seamlessly interrupt and redirect dialogue. To replicate this, CleanS2S implements a full-duplex and state-aware interruption system that monitors both voice and text inputs. During audio processing, Voice Activity Detection (VAD) continuously scans for speech input, while text-based interruptions can also be triggered. The system maintains a finite state machine to track interaction phases—listening, processing, and speaking—alongside priority queues for managing concurrent tasks. When an interruption occurs, ongoing TTS playback or LLM inference threads are immediately halted, audio buffers are purged, and the pipeline resets to the listening state. To minimize latency, non-blocking I/O and lightweight threading ensure low transition latency, even during computationally intensive LLM operations.

The term ”Clean” in CleanS2S reflects our design philosophy of simplicity and transparency, achieved through a unified single-file implementation encompassing the complete S2S pipeline. This minimalist design eliminates conventional multi-module complexity while maintaining full functionality through atomic configurations that enable adjustments with minimal code modifications. The system ensures complete operational transparency via inline logging hooks that capture comprehensive execution traces, including audio buffers, dialogue histories, and performance metrics, thereby facilitating straightforward debugging through visible data transformations. The implementation supports extensibility through subclassing, allowing custom model integration without core modifications, while its lean structure reduces both prototyping barriers and deployment overhead. This combination of features makes CleanS2S particularly valuable for rapid iterations, such as LLM studies or real-time TTS. Additionally, the framework’s modularity permits independent component upgrades or benchmarking without pipeline destabilization. Besides, CleanS2S also provides a complementary frontend implements deliberate UI/UX strategies to emulate natural conversational dynamics.

The current mainstream AI often adopts an ”obliging” response strategy (et al., 2021), in order to assist user inquiries. When encountering policy-violating or offensive content, agents follow predetermined prompts or protocols based on safety policies. In contrast, human responses vary with context severity, showing nuanced reactions absent in existing systems. Our goal is to enrich AI’s behavioral repertoire by introducing more human-like interaction diversity. Merely expanding system permissions is insufficient. A critical innovation is to endow AI with anthropomorphic self-awareness through cognitive architecture design (Dolgikh, 2024). While achieving subjective consciousness remains infeasible, we implement an artificial memory system that retains critical operational data, including interaction histories and user profiles—with particular emphasis on dialogue dynamics and patterns to establish temporal awareness.

Current interaction paradigms have two mechanical patterns: 1) Passive waiting for complete user input; 2) Automatic immediate response. We propose to introduce strategic flexibility by enabling AI to either interrupt ongoing user input or selectively ignore received messages. It should be emphasized that this kind of interruption by AI is distinct from the previous situation where users interrupted AI. Concretely, we formalize five human response patterns: 1) Input interruption; 2) Explicit refusal; 3) Deflective response; 4) Non-response; 5) Standard response. This transition from binary response generation to continuous interaction modeling enhances conversational agency while preserving coherence. By incorporating human-like variance in response timing and strategy selection, our framework can significantly reduce the formulaic nature characterizing current chatbots.

The memory module serves as the central data hub within our system, facilitating information exchange between users and agents (Figure 2). This component integrates three critical information dimensions: temporal signals, historical interactions, and factual anchors, thereby generating rich contexts for downstream modules. Existing works such as A-MEM (Xu et al., 2025) and MemGPT (Packer et al., 2024) illustrate the balance between real-time processing and long-term knowledge storage. Our framework maintains compatibility with such established methods, requiring only that implementations process user dialogue as input and generate structured conversational contexts as output. Building upon this, CleanS2S enhances temporal sensitivity while preserving their core advantages: (1) analyzing inputs against dialogue history to extract salient information, (2) summarizing content with consideration for temporal and character-based factors, and (3) delivering structured outputs to next decision-making components.

In CleanS2S, the module that receives data from the memory and makes decisions on the direction of the dialogue is the Subjective Action Judgement. Based on its guidance, the system executes operations via three pathways and returns results to users. We categorize the above 5 behaviors into three groups: (1) Model-dependent processing (rejection, perfunctory responses, routine replies), (2) Model-free processing (blocking), and (3) Special-case handling (interrupt user input). For model-dependent cases, the system combines behavioral guidance with input and history, then processes this through LLMs to generate appropriate responses. Model-free processing triggers access control, imposing permanent or temporary chat restrictions. For interruptions, the system monitors input and assesses content in real-time. When enough information justifies interruption before completion, it executes two processes: (1) Immediately output preset templates to the user to end the interaction, and (2) Continuation responses using a mechanism that combines interruption context with behavioral guidance.

The Subjective Action Judgement module serves as the decision-making component of the interactive system, designed to accurately assess input information. This module primarily executes two types of judgements: (1) swiftly determining whether to interrupt user input, which may stem from the increasing irrelevance of user-provided information or conflicts between user input and the system’s stance; and (2) deciding whether to implement rejection strategies toward user input, such as blocking mechanisms or providing perfunctory responses. We propose Action Judgement SFT that powers this module through a fine-tuned LLM, leveraging its general-purpose capabilities to flexibly address diverse scenarios. To obtain training data, we collect one-on-one interviews and talk show videos, extracting dialogues between participants as foundational text. To equip the model with judgement capabilities through Action Judgement SFT, we annotated the data based on the actual duration of pauses in real conversations, enabling the model to learn temporal interval assessments during responses. Besides, we introduce truncated speech segments at arbitrary positions during dialogues as negative samples, thereby suppressing the model’s tendency to make interruption decisions indiscriminately across all scenarios. The training method we proposed is Action Judgement SFT. By employing the above framework, the agent can respond more adeptly to complex scenarios in user inputs (Figure 3 and Table 2), including boring topics, unreasonable demands, affront, same opinion and conflicting opinion, where the agent may, for instance, tactfully interrupt user input to navigate the conversation constructively.