The most interesting part of a voice agent is not that it can talk. It is what happens when conversation, external tools, and application state stop moving in lockstep.
A user asks a question. A search starts. The agent begins answering. The user interrupts. Which work still belongs to the conversation—and which result must no longer influence it?
I built LexLatam Voice Agent to explore those boundaries in a small, inspectable application. It combines Spanish voice interaction through OpenAI Realtime, browser audio over WebRTC, and legal research through LexLatam’s Model Context Protocol (MCP) interface. The public TypeScript application handles conversation and tool execution; LexLatam’s proprietary retrieval implementation remains outside the repository.
This was a bounded, AI-assisted engineering spike around an existing research platform—not an attempt to build a new search engine or launch a production voice service. (GitHub)
Watch the demo · Explore the source
The recording presents a Spanish-language research conversation through a browser prototype. It does not demonstrate a deployed telephone or SIP service.
Start with a real capability, not a mock business
LexLatam already had the domain capability I needed: searching Panamanian legal sources. Its MCP interface exposes search_panama_law, returning potentially relevant documents with excerpts, references, publication metadata, and official links where available. (LexLatam.ai)
That made the experiment more useful than a voice agent calling a fictional order database. The voice application would consume a real capability without importing retrieval code, reading LexLatam’s database, or knowing how its corpus was processed.
The boundary is specific: the client knows the tool contract, not the search implementation. It is not a universal adapter that can consume any MCP server without changes.
Nor does MCP make returned information private by itself. The evidence sent across the interface is available to the consuming application and, when included in context, the voice provider. Keeping implementation private and controlling disclosed data are separate concerns.
Separate audio, session control, and legal search
The application has four participants:
WebRTC carries audio between the browser and OpenAI. The Node backend establishes the session and attaches a separate sideband WebSocket for tool handling and session control. OpenAI documents this two-connection pattern for Realtime applications. (OpenAI Developers)
This choice avoids turning Node into an audio relay while keeping tool credentials and execution in server code. It also creates another connection whose startup, failure, and cleanup must be handled.
I used native realtime speech rather than assembling separate speech-recognition, text-generation, and speech-synthesis services. That kept the experiment focused on the application boundaries instead of audio-pipeline plumbing. The tradeoff is less independent control over each speech stage.
In the browser, microphone acquisition, connection setup, playback, and cleanup are explicit. Even a small detail matters: a microphone permission prompt can finish after the user has already pressed Stop. The browser session code checks for that case and releases the newly acquired tracks instead of reviving a closed session.
The model requests a tool; the backend executes it
Model function calling and MCP are different interfaces here.
OpenAI produces a function request. The Node application validates that request and translates it into an MCP operation. On the research side, MCP provides tool discovery and invocation through tools/list and tools/call. (Model Context Protocol)
The client discovers the LexLatam tool and checks that its schema matches the contract this application expects. It does not automatically expose every discovered capability to the model.
Only search_panama_law is allowed. Its arguments must contain one nonempty query string, bounded to 2,000 characters. Returned evidence is checked for expected fields and size limits, and at most three sources are passed onward.
The same code distinguishes transport success from tool success: an HTTP response can succeed while the MCP result reports a tool error. That error must not become apparent research evidence.
For private testing, the backend sends a configured bearer credential to LexLatam. The token stays server-side; there is no browser login, credential-management UI, or OAuth implementation in this spike. Failed authentication does not trigger an anonymous retry. The verified private-access path is local, not a claim of production deployment. (GitHub)
Interruption is also a stale-result problem
Native interruption handles the audio side of barge-in. It does not settle what to do with a search that is already running.
Consider this sequence:
Question A starts a search.
The user begins question B before search A finishes.
Search A returns evidence for a question that is no longer current.
The application uses a small turn-counter guard. When new speech starts, the counter advances. A tool operation remembers the counter at the start of its handling and checks it before returning evidence.
When that work is superseded, the backend returns a discarded-result message for the tool call rather than its sources. It does not request a new spoken response from that stale completion. A session-local set of handled call IDs also suppresses duplicate execution.
These are narrow protections, not a distributed exactly-once guarantee or a proof against every possible event ordering.
Stopping work and refusing obsolete results are different responsibilities. Even where a local operation can be aborted, that does not prove a remote service stopped processing it.
Turn-taking needed live feedback
The initial silence-based detector was too eager during natural speaking pauses. Local testing led to a change from a 500-millisecond silence window to semantic voice activity detection with low eagerness. (GitHub)
Semantic VAD considers whether speech sounds complete, rather than relying only on silence. Lower eagerness gives the speaker more room, at the cost of potentially waiting longer before responding. (OpenAI Developers)
That is a conversation-design tradeoff, not simply a latency setting to minimize. The current session configuration keeps native interruption enabled without introducing a custom turn detector.
Make the behavior visible—and name the measurements correctly
The UI exposes conversation transcripts, session state, the current research operation, returned sources, and tool duration. A one-second status poll was enough for the demo; a second application event-streaming system was unnecessary.
The UI event handling also distinguishes response generation from audio-buffer activity. Generation finishing does not mean the listener has heard all the audio. Playback-related events inform the speaking state, but they are not a precise measurement of sound reaching a listener’s ear.
One local acceptance run reported 4.2 seconds for search_panama_law, returning three sources. That was the rounded backend tool duration, including argument validation, the MCP request, and result validation.
It is not end-to-end voice latency, an average, or a percentile. The delay a listener experiences also includes turn detection, the model’s tool decision, answer generation, and audio delivery. UI polling adds a separate display delay. (GitHub)
The lesson is simple: a visible number should explain the system, not make it appear faster than the measurement establishes.
Evidence provenance is not a correctness guarantee
The application constrains tool execution and displays source cards derived from validated MCP results. Its session instructions ask the model to use retrieved evidence, acknowledge gaps, and avoid inventing sources or legal applicability.
Those controls provide different kinds of assurance:
| Application boundary | What still needs evaluation |
|---|---|
| Code checks which tool may execute and validates its arguments. | Whether the model asked a useful research question. |
| Displayed source cards come from validated tool results. | Whether the spoken answer faithfully represents those sources. |
| A stale-result guard withholds superseded search evidence. | Whether the complete conversation behaves well across broader timing conditions. |
The distinction is reflected in the session instructions and focused execution tests.
A source can be correctly retrieved but incomplete for the question. An excerpt can omit a qualification or begin next to a different article heading. Neither a valid schema nor a plausible citation proves that the model interpreted the law correctly. The spoken answer—including any article number—still needs comparison with the source.
That is why I describe this as evidence-backed voice research, not legally verified advice or hallucination-free answering.
Test the boundaries; verify the experience live
The tests isolate predictable behavior: rejecting unsupported tools, suppressing duplicate execution, and withholding a delayed search result after a newer turn begins. The delayed-result test controls exactly when the search resolves, so the race can be exercised without a live model.
Local acceptance testing then covered what those tests cannot establish: Spanish conversation, audible replies, cited legal searches, deliberate interruption, and stopping and restarting a session. These results are documented in the completed project plan. (GitHub)
This is evidence that the bounded demo works—not a retrieval benchmark, load test, or comprehensive evaluation of interruption handling.
The remaining limitations are intentional. State is in memory. MCP availability is currently required at session startup, even for a greeting. Private remote access and a public multi-user deployment are not verified. There is no telephony integration, account system, persistent conversation archive, or provider failover. (GitHub)
The on-screen evidence panel is also not a durable audit record. Reconstructing a conversation months later would require retention and versioning work that this spike deliberately does not implement.
The engineering takeaway
The useful result was not another voice interface. It was an inspectable connection between a realtime conversation and an existing domain capability, with clear ownership of tools, credentials, evidence, and obsolete work.
The model can propose a request. Application code decides whether it can execute, which returned fields are accepted, and whether the result still belongs to the current turn. The quality of the resulting answer remains something to evaluate—not something a protocol boundary guarantees.
That is the pattern I wanted this experiment to make concrete: put explicit, testable controls around probabilistic behavior, and keep the system small enough that those controls are easy to inspect.
Leave a Reply