Every AI voice agent project hits the same fork in the road within the first week: WebRTC or WebSocket?
Pick wrong and you spend months fighting latency, reliability, or compliance problems that no amount of prompt engineering will fix. Pick right and the transport layer disappears - it just works while you focus on the conversation design that actually matters to users.
This isn't a theoretical comparison. OpenAI's Realtime API (GA as of late 2025) exposes both transports. AWS Bedrock AgentCore added WebRTC support in March 2026. Pipecat, LiveKit Agents, and every serious voice framework now force this choice up front. Here's how to make it well.
The core tradeoff in one sentence
WebRTC gives you the lowest possible latency by connecting the client directly to the media endpoint - but your server loses visibility into the audio stream.
WebSocket routes all audio through your server, giving you full control over compliance logging, tool orchestration, and business logic - at the cost of an extra network hop (80–150 ms).
That's the entire decision in its simplest form. Everything else is implementation detail.
Where each transport actually lives
| Transport | Connection path | Server-side control | Latency overhead |
|---|---|---|---|
| WebRTC direct | Browser/app ↔ OpenAI | Limited (ephemeral keys) | Lowest |
| WebSocket via server | Your server ↔ OpenAI | Full mediation | +80–150 ms |
| LiveKit/SFU + WebRTC | Browser ↔ SFU ↔ agent ↔ OpenAI | Full (agent in your VPC) | +50–80 ms |
| SIP (telephony) | Phone ↔ OpenAI | Limited routing | +50–100 ms |
The LiveKit/SFU pattern deserves special attention. It gives you WebRTC's user-perceived latency plus the server-side control of WebSocket. Your agent runs in your infrastructure, mediates tool calls, redacts PII, writes the audit log, and forwards audio to the user through the SFU. For most production deployments above demo scale, this is the architecture that wins.
The latency math that actually matters
800 ms voice-to-voice is the threshold where conversation still feels natural. Above 1.2 seconds, users start talking over the agent. Here's how the budget breaks down:
| Stage | Typical range | What controls it |
|---|---|---|
| Mic capture & encode | 20–40 ms | Native echo cancellation, 24kHz PCM |
| Network: client → agent | 30–80 ms | SFU proximity, WebRTC vs TCP |
| Agent → OpenAI | 10–60 ms | Same-region deployment |
| VAD & turn detection | 200–400 ms | Threshold tuning or push-to-talk |
| Model time-to-first-byte | ~500 ms | Cached prompts, tight instructions |
| Audio → client & render | 50–100 ms | Stream chunks, don't wait for full response |
| Total | ~800 ms | Achievable with discipline |
The critical insight: WebRTC saves you 80–150 ms on the network hop. That's meaningful when your total budget is 800 ms - it's the difference between "snappy" and "noticeable lag." But if your model TTFB is already 500 ms and your VAD adds 300 ms, saving 100 ms on transport won't fix the user experience. Optimize the whole pipeline, not just the wire.
When WebRTC is the right call
Choose WebRTC direct when:
- You're building a browser or mobile voice interface where latency is the primary UX differentiator
- The use case is an internal tool, prototype, or product where you don't need server-side mediation
- You can safely distribute ephemeral API keys from your backend
- Compliance logging can happen client-side or isn't required
The WebRTC advantage is real but narrow. The protocol uses UDP, so a dropped packet causes a tiny audio glitch rather than the stream stall you get with TCP-based WebSocket. The browser handles audio encoding (Opus), echo cancellation, noise suppression, and ICE negotiation natively. Your frontend code gets simpler.
The hidden cost: WebRTC's built-in audio processing (AEC, AGC, noise suppression) can interfere with server-side voice activity detection models like Silero. The audio that reaches your STT layer has already been processed in ways that weren't designed for machine consumption. This is a known production failure mode - the jitter buffer smooths audio in ways that confuse downstream VAD.
When WebSocket is the right call
Choose WebSocket via your server when:
- You need compliance logging, PII redaction, or audit trails on every conversation
- Your agent orchestrates multiple tools mid-conversation (CRM lookups, calendar booking, payment processing)
- Business logic must gate what the model says before it reaches the user
- You're building for healthcare, finance, or any regulated industry
- You need to inject context, swap models, or apply guardrails in real-time
The WebSocket advantage is control. Your server sees every audio frame, every tool call, every model response before it reaches the user. You can log it, redact it, block it, or augment it. For most production enterprise deployments, this control is non-negotiable.
Practitioner consensus backs this up. Developer communities report that OpenAI's Realtime API over WebRTC shows connection drops under load and mid-session model update failures. WebSocket connections are more predictable and easier to monitor in production.
The hybrid pattern most teams actually ship
In practice, most production voice agents don't use either transport in isolation. They use a LiveKit/SFU hybrid:
- Browser connects via WebRTC to a LiveKit (or similar) SFU - lowest latency for the user
- Your agent server connects via WebSocket to OpenAI's Realtime API - full control over the model interaction
- The SFU bridges them - forwarding audio between the WebRTC client session and your WebSocket-connected agent
This gives you sub-second perceived latency on the user side while maintaining server-side control over tools, compliance, and business logic. The SFU adds roughly 30–50 ms - well within the 800 ms budget.
Frameworks like Pipecat make this easier by supporting WebSocket and LiveKit interchangeably. You develop over WebSocket locally and deploy with LiveKit for browser clients without rewriting agent logic.
Production pitfalls to plan for
Long-session latency drift
Median turn latency climbs from ~800 ms early in a session to over 2 seconds after 20+ turns. The fix: rotate sessions every 8–12 turns by reseeding context into a fresh session, or use server-side conversation-item deletion to keep active context bounded.
Tool-call latency compounds
Every tool call adds its own round-trip. A CRM lookup that takes 800 ms creates a pause the user hears. Solutions: front-load common lookups (greet users by name without a tool call), parallelize batch tool calls, and consider speculative lookups for the next likely query.
Cold start on containerized agents
If your agent runs in a container (AgentCore, ECS, Cloud Run), the first connection after idle often fails. Session affinity matters - WebRTC's multi-step signaling handshake requires all messages to hit the same container instance. Plan for retry logic with 10-second timeouts.
Echo cancellation conflicts
Without client-side echo cancellation, the agent hears its own voice and barges in on itself. With WebRTC this is handled natively. With WebSocket, you need to implement it - and if you're using server-side VAD, the echo cancellation processing may interfere with voice detection accuracy.
The HIPAA constraint that changes everything
As of May 2026, OpenAI's Realtime API audio modality is not HIPAA-eligible under any BAA - not OpenAI's, not Azure's. The text-based Azure OpenAI endpoints are covered. The audio path is not.
If you're building a healthcare voice agent, your transport choice is already made: you need a chained pipeline (HIPAA-eligible STT → text LLM → HIPAA-eligible TTS) routed through WebSocket on your server, where you control every byte. WebRTC direct to OpenAI is off the table until the BAA coverage expands.
Decision flowchart
- Is the conversation regulated (healthcare, finance)? → WebSocket through your server. Non-negotiable.
- Do you need server-side tool orchestration or PII redaction? → WebSocket or SFU hybrid.
- Is this a browser/mobile product where latency is the key differentiator? → SFU hybrid (WebRTC to user, WebSocket to model).
- Is this an internal tool or prototype with <100 concurrent users? → WebRTC direct is fine.
- Are you below 10k minutes/month and validating PMF? → Consider Vapi or Retell (they abstract the transport choice for you) and migrate to self-managed when you cross the volume threshold.
What this means for your voice agent project
The transport layer is an architecture decision, not a library choice. It determines where your audio flows, what you can observe, what you can control, and how much latency your users experience. Getting it wrong isn't a bug you patch - it's a rearchitecture.
At Apptitude, we help teams make these decisions before code is written - during the discovery phase where architecture choices are cheap to change. If you're scoping a voice agent and the WebRTC-vs-WebSocket question is blocking your team, that's exactly the kind of decision our AI strategy engagements are built to resolve.
Sources: OpenAI Voice Agents documentation, OpenAI Realtime API Production Guide (Fora Soft, May 2026), AWS AgentCore WebRTC implementation (DEV Community, March 2026), LiveKit: Why WebRTC beats WebSockets for Voice AI.