Streaming LLM Responses in Production: SSE vs WebSocket

The streaming endpoint works in development. Tokens arrive, the UI renders them, the demo lands. Then real traffic hits and the gaps show up. A user closes their tab and the LLM keeps billing for forty seconds of generation nobody will ever read. A corporate proxy strips the connection mid-stream and the frontend hangs forever waiting for tokens. A content-filter refusal returns mid-generation and the UI just stops with no message. A burst of token chunks arrives, the page jank-scrolls, and the perceived latency that streaming was supposed to fix gets worse than blocking.

Streaming LLM responses is not a feature. It is an architecture. Get it wrong and you ship a system that looks fast in the demo and fragile in production. Get it right and you ship something that feels instantaneous, fails gracefully, and stops costing money the moment the user stops paying attention.

This guide is the engineering counterpart to AI chat UX patterns for production. It covers the transport choice (SSE vs WebSockets), the streaming pipeline from provider to browser, the failure modes that only show up under load, and the patterns that distinguish a production streaming layer from a demo. It sits inside the broader conversation about why your AI experiments are failing — and specifically why impressive AI pilots become shelfware, where the streaming layer is one of the things “the system underneath the chat box” includes.

Why Streaming Is the Default Now

Streaming LLM responses changed from a nice-to-have to table stakes for one reason: perceived latency. The numbers are consistent across practitioner writeups (Redis, OSValdo Restrepo’s streaming guide, Zylos Research on token delivery). A 10-second total generation with 200ms time-to-first-token (TTFT) feels faster than a 4-second total generation with 3-second TTFT, even though the second is more than twice as fast in real terms. Users tolerate generation time when they see progress. They abandon when they see nothing.

The practical thresholds:

TTFT under ~300ms feels instant.
TTFT 300–700ms feels snappy.
TTFT 700ms–1.5s feels acceptable for complex tasks if there is a visible loading state.
TTFT over 1.5s feels broken without explicit “thinking” UX.

Streaming buys roughly a 10–20x improvement in perceived responsiveness without the model being any faster. That is why every serious production deployment streams. The question is no longer whether to stream but how — and how to handle the cases that streaming creates.

SSE vs WebSockets: Pick SSE Unless You Have a Specific Reason

The transport debate has converged. For LLM streaming from server to client, Server-Sent Events (SSE) is the default. WebSockets are the answer to a different question. The reasoning, supported by every recent practitioner guide on the topic:

Property	SSE	WebSocket
Direction	Server → client	Bidirectional
Transport	Plain HTTP	HTTP upgrade
Proxy / CDN compatibility	Works everywhere	Often breaks behind corporate proxies
Scaling	Stateless, horizontal	Requires sticky sessions or socket brokers
Native API	EventSource (browser)	WebSocket (browser)
Reconnection	Built-in	Manual
Cancellation	Close socket	Close frame
Used by	OpenAI, Anthropic, Google APIs	Specific agentic interactive flows

LLM token streaming is a one-way operation: the client sends one prompt, the server sends tokens back. SSE was literally designed for this in the HTML Living Standard. It is plain HTTP, so it passes through every proxy, CDN, and corporate firewall that supports the basic web. WebSocket requires an upgrade handshake that frequently breaks in those environments. SSE is stateless, so horizontal scaling needs no sticky sessions and no socket broker. WebSocket needs both.

When WebSockets are the right answer. Bidirectional agentic flows where the user can inject context mid-generation, multi-turn voice, or low-latency tool execution with bidirectional state push. If your agent is purely “user prompts, model responds, user interrupts,” SSE is enough. If your agent has a UI that pushes events to the model mid-generation, WebSockets earn their complexity.

For most production AI chat, the architecture is SSE for the primary stream, plus a separate REST surface for user actions like “approve this tool call.” That keeps the streaming layer simple and the action layer auditable.

The End-to-End Streaming Pipeline

A production streaming setup has more moving parts than the first implementation reveals. The full pipeline:

[Browser EventSource]
       ↑ SSE
[Edge / CDN — pass-through, no buffering]
       ↑ SSE
[Application server — stream proxy + business logic]
       ↑ SSE
[Gateway / router (LiteLLM, Portkey, custom)]
       ↑ SSE
[LLM provider (OpenAI, Anthropic, ...)]

Each hop has rules that, when violated, kill the stream.

Edge and CDN configuration

CDNs and reverse proxies aggressively buffer HTTP responses by default. That buffering destroys streaming: tokens accumulate at the edge and arrive in a single chunk after the model finishes generating. The fixes vary by provider but the same headers work nearly everywhere:

Disable response buffering for the streaming route (X-Accel-Buffering: no on Nginx, equivalent flags on Cloudflare, Fastly, Vercel edge, etc.).
Set Cache-Control: no-cache, no-transform to keep intermediaries from rewriting or holding the response.
Keep Content-Type: text/event-stream.
Send periodic keepalive comments (: keepalive\n\n every 15–30 seconds) to defeat idle timeouts.

If your TTFT is good in development and bad in production, this is almost always the cause.

Streaming gateway: pass bytes through

If your app sits behind a gateway like LiteLLM, Portkey, OpenRouter, or a homegrown router, the rule is simple: forward the SSE bytes unchanged. Re-serializing every event into a new envelope adds per-token latency, risks subtle bugs (especially around the [DONE] sentinel and incremental JSON in tool-call arguments), and forces the client to understand two formats.

The gateway is also where you add the production capabilities that the raw provider stream does not have: per-request token quotas, per-tenant cost attribution, audit logging, fallback routing, and KV/semantic cache lookup. For the deeper view of that layer, see LLM routing in production and LLM caching strategies.

Application server: keepalives and connection signaling

Three things have to be true at the application server:

Streaming is enabled. Frameworks default to buffered responses. Streaming requires explicit opt-in (returning a stream or generator).
Keepalives are sent. Network paths between client and server drop idle TCP connections silently. Without keepalives, a long-thinking model produces zero traffic for 30+ seconds and intermediaries close the socket. Send SSE comments every 15–20 seconds during planning or tool execution.
Client disconnect propagates upstream. When the browser closes the EventSource, the server has to detect it and cancel the upstream LLM call. Otherwise generation continues, tokens get billed, and nobody reads the output. This is the single biggest production cost leak in streaming setups.

Token Streaming, Partial Responses, and Tool Calls

The actual content of an SSE stream from an LLM is more than tokens. A modern provider stream interleaves:

Content tokens — the visible response, typically delivered as deltas.
Tool-call deltas — incremental JSON building up function arguments. These arrive in pieces and have to be reassembled before the call is dispatched.
Reasoning tokens — chain-of-thought from reasoning models, often in a separate channel.
Usage metadata — final token counts, typically in the last event before [DONE].
Error events — surfaced inside the stream when the provider catches a problem after generation begins.
Done sentinel — [DONE] or equivalent marking end of stream.

A production frontend has to parse all of these and route them to different rendering paths. Tokens go into the message bubble. Tool-call deltas accumulate into the visible tool card and dispatch when complete. Reasoning tokens go into the optional collapsible panel (see the chat UX guide for the default-hidden pattern). Errors render as in-stream error states. Usage metadata feeds your LLM cost attribution pipeline.

A pattern that works on the client:

event: token     → append to current message
event: tool      → start or update tool card
event: tool_done → dispatch tool, render result inline
event: reasoning → append to hidden reasoning buffer
event: usage     → record cost, update budget meter
event: error     → render in-stream error, preserve partial
event: done      → finalize message, enable actions

Every event is distinguishable. Every event has a side effect that is testable in isolation. The stream is no longer a string; it is a typed event log.

The Stream Is Not a String

The most common production bug in LLM streaming is treating the stream as a single token stream and concatenating naively. Real provider streams interleave tool-call JSON, reasoning, usage data, and errors. A frontend that does not parse event types will silently corrupt tool arguments, dump reasoning into the visible answer, or render event: error as plain text. Parse the events as events.

Mid-Stream Error Handling

This is the section practitioners get wrong most often. Once the server has sent HTTP 200 OK and started streaming, the HTTP status code is no longer useful. An error that occurs at token 50 cannot be communicated by changing the response code — the code is already 200. The error has to arrive as a stream event the client knows how to parse.

The failure modes worth handling:

Failure	Where it originates	How it surfaces	Recovery
Provider rate limit (pre-stream)	Provider	HTTP 429 before stream starts	Backoff and retry; consider model fallback
Provider 5xx (pre-stream)	Provider	HTTP 5xx before stream starts	Retry with backoff
Content filter refusal	Provider	`event: error` mid-stream or empty completion	Surface to user, suggest edit
Token limit hit	Provider	Stream ends with `finish_reason: length`	Offer continuation or summarization
Tool execution error	Your tool	`event: tool_error` you emit	Retry tool, escalate, or ask user
Network drop (mid-stream)	Network	EventSource error, no `done` event	Resume or restart
Gateway timeout	Edge / proxy	Connection closes mid-stream	Resume or restart
Client tab closed	Browser	Server detects socket close	Cancel upstream LLM, stop billing

Every one of these needs a distinct UI state and a distinct backend behavior. Generic “Something went wrong” is the equivalent of throwing the user a 500-line stack trace and asking them to debug.

The frontend pattern that survives this:

Distinguish “stream ended without done” (likely network drop, offer resume) from “received event: error” (semantic failure, render the message).
Preserve the partial response. Never throw away tokens already shown.
Surface a single recovery affordance: “Resume,” “Retry,” or “Edit and resend,” depending on the failure class.

For systems with mature error handling, the in-stream error event is also the place to attach a correlation ID that flows into your LLM tracing pipeline. A user reporting a failure can hand you the ID and you can find the exact trace.

Cancellation: The Cost Leak Nobody Talks About

LLM streams cost money for every token generated. The provider does not stop generating just because the user closed the tab. The provider stops when the connection signals that it should stop. If your stack does not propagate that signal end-to-end, you pay for output that vanishes into the void.

The propagation chain has to be intact:

Browser closes the EventSource (user navigated away, closed tab, hit stop).
Edge / CDN observes the client-side close and closes its upstream connection.
Application server’s stream handler observes the closed connection (most frameworks expose this as a context cancellation or an iterator that raises on next read).
Application server cancels the upstream call to the LLM provider.
Provider detects the closed socket — typically within a few hundred milliseconds — and stops generation.

A break anywhere in that chain means the model keeps generating and you keep paying. The most common breakpoints:

Edge buffering. If the CDN buffers, the close signal does not propagate until the buffer flushes.
Stream proxying. A gateway that re-serializes the stream often holds the upstream connection open even after the downstream closes, because the proxying code reads everything before forwarding.
Framework handler. Some web frameworks complete the stream handler in the background even after the client disconnects, especially under default async configurations.

The test for whether your cancellation works is simple: start a long generation, close the tab at 10% complete, then check your provider’s usage logs for that request. If usage shows the full token count, your chain is broken.

The architecture pattern that fixes this: a context that flows from the inbound request through the upstream call, with explicit cancellation propagation. In Node, this is an AbortController passed through every layer. In Go, it is a context.Context. In Python, it is an asyncio.CancelledError that propagates. The implementation detail varies; the principle does not.

Backpressure, Smoothing, and Burst Control

Some providers and some gateways deliver tokens in bursts: a long pause, then a flood of 20 tokens at once, then another pause. Rendered naively, this produces a stutter that defeats the entire purpose of streaming. Recent practitioner writeups, including Zylos’s research on token delivery architectures, converge on a smoothing pattern:

Buffer incoming tokens in a short queue on the client.
Render at a steady cadence (typically targeting 15–40 tokens per second visible, depending on language and target audience).
If the buffer fills (provider is faster than render), increase render cadence. If the buffer empties, fall back to whatever the provider delivers.

The TokenFlow scheduling research showed 80%+ improvements in P99 TTFT and steadier delivery just by smoothing. For most products the implementation is a 50-line client-side smoother, not a research project.

Resumable Streams (and When Not to Bother)

A subset of products — long generation, expensive prompts, mobile users on flaky networks — benefit from resumable streams. The pattern: the server assigns a stream ID, persists tokens as they generate, and exposes a resume endpoint that picks up from a given token offset.

For most products this is overkill. Streams are short, retries are cheap, and the engineering cost of resumable infrastructure outweighs the UX gain. For long-form generation, deep research agents, or coding assistants where a single generation can run a minute or more, resumability moves from luxury to necessary.

The decision is: how often does a stream fail, and how expensive is restarting from scratch? Above some threshold (practitioner rule of thumb: failures cost more than ~30 seconds of regeneration plus the customer goodwill), build resumability. Below it, retry from scratch and spend the engineering elsewhere.

Build a Streaming Layer That Survives Production

Token streaming is a system, not a feature. metacto's [Operational AI](/solutions/operational-ai) practice builds the gateway, edge, and frontend patterns that make streaming feel instant, fail gracefully, and stop billing the moment the user leaves.

What the Streaming Layer Has to Be

A production LLM streaming layer is not “we added SSE to the endpoint.” It is an end-to-end pipeline where buffering is disabled at every hop, events are typed and parsed, errors are stream events with recovery paths, cancellation propagates from the browser to the provider in under a second, and tokens render at a perceived cadence that feels instant.

Most of the work happens after the demo lands. The transport choice is the easy decision; the discipline to wire the cancellation path end-to-end and parse the event stream correctly is the hard one. This is one layer of the system underneath the chat box — the gap between an impressive demo and production AI that the prompt is not the product names directly, and the gap why impressive AI pilots become shelfware catalogs in detail. Build it like it matters, because it does. The companion interface patterns sit one layer up in AI chat UX, and the case for when the AI should compose more than text is in generative UI in production.

Frequently Asked Questions

SSE or WebSockets for streaming LLM responses?

Server-Sent Events for almost every case. LLM token streaming is unidirectional, SSE was designed for exactly that pattern, it runs over plain HTTP so it passes through CDNs and corporate proxies cleanly, and it scales horizontally with stateless servers. WebSockets are the right answer only when you need bidirectional flows during generation — for example, user actions or context pushes that interrupt the model mid-stream. For the vast majority of production AI chat, SSE plus a separate REST surface for user actions is simpler, cheaper, and more reliable.

How do I handle errors mid-stream when status codes are already sent?

Once you have sent HTTP 200 OK and started streaming, errors cannot be signaled with a status code. They have to arrive as typed stream events the client parses, like an event: error with a structured payload. The frontend distinguishes those from a closed connection without a done event, renders them differently, preserves any partial response, and offers an appropriate recovery action. Treating the stream as a string and concatenating naively is the single most common bug in production LLM streaming.

What is time-to-first-token (TTFT) and why does it matter?

TTFT is the elapsed time from the request leaving the client to the first content token arriving. It dominates perceived latency. A model with 200ms TTFT and 10-second total generation feels faster than a model with 3-second TTFT and 4-second total. The practical thresholds: under 300ms feels instant, 300-700ms feels snappy, above 1.5s requires explicit thinking UX to stay tolerable. Optimizing TTFT is usually more valuable to UX than optimizing total generation time.

How do I make sure canceled streams stop billing?

Cancellation has to propagate from the browser all the way to the LLM provider. The chain is browser closes EventSource, edge or CDN observes the close, application server detects context cancellation, the upstream provider call is canceled, and the provider stops generation within a few hundred milliseconds. Edge buffering, stream-proxying gateways that re-serialize events, and frameworks that complete handlers in background after client disconnect are the three most common breakpoints. Test by closing a tab mid-generation and checking provider usage logs for that request — if usage shows the full token count, your propagation is broken.

Why does streaming work in development but not in production?

Almost always edge or proxy buffering. CDNs and reverse proxies buffer HTTP responses by default, which destroys streaming because tokens accumulate at the edge and arrive in one chunk after generation finishes. The fix is disabling response buffering on the streaming route, setting Cache-Control no-cache and no-transform, keeping the text/event-stream content type, and sending periodic keepalive comments to defeat idle timeouts. The provider-specific flag names vary but the pattern is universal.

Do I need resumable streams?

Usually no. For most products, streams are short, retries are cheap, and the engineering cost of resumable infrastructure outweighs the UX gain. Resumability pays off for long-form generation, deep research agents, coding assistants, or mobile users on flaky networks where a single generation can run a minute or more and restarting from scratch is expensive in both cost and customer goodwill. Above that threshold, build resumability. Below it, retry from scratch and spend the engineering budget on caching, routing, or cost attribution instead.

Sources and further reading

Streaming LLM Responses: Implementation Patterns for Production