Building ChatGPT-Style Streaming in React: FastAPI + Next.js Production Guide

Updated 2026-09-16. This guide was rewritten from scratch. In February it used a request-scoped generator, retired model IDs and an nginx header that does nothing. Versions tested here: FastAPI 0.141.1, Starlette 1.6.0, uvicorn 0.53.0, Redis 8.10.1, Next.js 16.3.5, eventsource-parser 4.1.0.

My FastAPI Server-Sent Events (SSE) endpoint generated 2,210 tokens for a React client that was already gone. I had read the stream for four seconds and then cut the client's network without closing the socket, which is what a phone losing signal or a laptop lid closing looks like to a server. At the cut, the server had generated 190 tokens. Sixty seconds later it had generated all 2,400, and the provider stream closed only when the answer ended on its own. It was the textbook endpoint behind a Next.js app: the model call runs inside the response generator, so a disconnect cancels it.

The same endpoint behaved perfectly when I killed the client process instead. Generation stopped at 158 tokens, the instant the connection closed. Every local test of cancel-on-disconnect looks like the second run. Most production drops look like the first.

Who decides when a streamed LLM generation stops

A ChatGPT-style chat has to answer one question that most codebases never state: how long does a generation live once nobody is reading it? There are three answers in circulation.

Connection-bound is the DEV post's own term, where it is contrasted with "task-bound" work such as agent runs. Completion-bound is my label for the resumable default, because none of the three sources that recommend it gives it a name.

My position: both published defaults are wrong for long generations, and connection-bound, the policy teams pick to save money, is the one that saves least. It cannot see silent drops, so it keeps paying for them. It also destroys every answer interrupted by a reload, so the user asks again and pays for the prefix twice. Completion-bound fixes reloads and bills every abandoned tab to the last token. In the simulation later in this article, presence-bound was cheapest in every mix I tried and was the only policy that lost no answers, while connection-bound lost answers in all of them.

Presence-Bound Generation ties a run's lifetime to a lease that the client renews. A reader that comes back inside the grace window resumes from the event log. A reader that never comes back stops costing money a bounded number of seconds after it left. I call the tokens a run generates after its last reader is gone for good the Orphan Tail, and the whole design exists to keep that number small without losing answers.

The idea of stopping inference when nobody is present is not new. Ably's AI Transport documents a presence set that agents check before ending a run or short-circuiting the LLM call, with a fixed presence timeout of "around 15 seconds". What that leaves open is the part this article is about. First, the grace window is a policy you size from user behaviour, not a transport constant. Second, the lease can live in Redis next to a plain Server-Sent Events (SSE) stream, with no vendor session. Third, the choice between the three policies has measurable consequences, which I measured.

Why cancelling the LLM call on client disconnect does not save tokens

Here is the connection-bound endpoint. It is the shape nearly every streaming tutorial teaches, now with FastAPI's native SSE support, which shipped in 0.135.0.

from collections.abc import AsyncIteratorfrom fastapi import FastAPIfrom fastapi.sse import EventSourceResponse, ServerSentEventfrom pydantic import BaseModelfrom provider import stream_tokensapp = FastAPI()class RunRequest(BaseModel):    prompt: str# Wrong for long answers: the generation lives exactly as long as this response.@app.post("/chat", response_class=EventSourceResponse)async def chat_inline(body: RunRequest) -> AsyncIterator[ServerSentEvent]:    async for delta in stream_tokens(body.prompt):        yield ServerSentEvent(data={"delta": delta})

Starlette supplies the cancellation. uvicorn's h11 implementation advertises Asynchronous Server Gateway Interface (ASGI) HTTP spec version 2.3, so Starlette's StreamingResponse runs a listener for http.disconnect next to your generator and cancels both when the message arrives. Your generator receives CancelledError at whatever it is awaiting. When a disconnect message arrives, this works well.

A silent drop does not send that message. When a phone leaves Wi-Fi, no FIN or RST packet reaches the server. Writes on the server keep succeeding because they land in the kernel's send buffer, and the kernel keeps retransmitting unacknowledged data. On Linux, the default tcp_retries2 of 15 "yields a hypothetical timeout of 924.6 seconds and is a lower bound for the effective timeout", according to the kernel documentation. A streamed answer of a few minutes finishes long before the Transmission Control Protocol (TCP) stack gives up. That bound describes whichever hop is actually flaky. Put nginx or a Content Delivery Network (CDN) in front and your origin's connection is to the proxy, so the proxy's client-side timeouts decide how fast the loss reaches your application, and the 15-second ping later in this article is designed to stop those timeouts firing.

I measured both cases in Docker on a Linux kernel with tcp_retries2=15, with a deterministic fake provider generating 40 tokens per second.

FastAPI's 15-second keep-alive ping does not help, because a ping is one more write into the same buffer. Connection-bound only catches clean closes: a closed tab, a reload, a killed process. That is the wrong half. A reload is the one case where the user wanted the answer, and connection-bound cancels it, so the user asks again and pays for the prefix a second time.

Resumable streams fix reloads, then bill every closed tab

Resumable LLM streams separate the generation from the connection. A producer writes every event to a log, readers replay the log from an offset, and a disconnect ends only the reader. Upstash described the pattern in April 2025, and Vercel's resumable-stream package states the lifetime rule plainly: "The producer will always complete the stream, even if the reader of the original stream goes away."

Vercel's AI SDK documentation is just as explicit about what Stop means afterwards. In a resumable setup, "closing a tab, refreshing the page, or calling stop() only closes the current HTTP connection and should not cancel the underlying generation", and its troubleshooting page adds: "Do not call the stop endpoint from route cleanup code, page unload handlers, or component unmount cleanup." Stop becomes a separate endpoint.

That is completion-bound. Written against the server later in this article, its entire lifetime rule fits in four lines:

async def lifetime_verdict(run_id: str) -> str | None:  # completion-bound    if await r.exists(key(run_id, "stop")):        return "stopped"    return None  # a reader that never comes back is not a reason to stop

Against the same fake provider, a user who leaves at two seconds and never returns gets nothing, and the run still produces the remaining 339 of 400 tokens. Reloads, on the other hand, are perfect: the reader comes back, replays from Last-Event-ID, and receives all 400 tokens in order with no gaps and no duplicates.

For short answers, completion-bound is the right choice and you can stop reading here. For a reasoning model writing 8,000 tokens at 76 tokens per second, every abandoned tab is a 105-second generation that nobody will ever read. Some teams already add a hard cap, such as the five-minute run timeout in Frontend Architecture for GenAI. A cap bounds the worst case. It does not know whether anyone is still there.

Presence-Bound Generation: a lease the client renews

A presence lease is a Redis key with a time to live (TTL). Creating the run sets it. Every client that is following the run renews it with a small POST every five seconds. A supervisor next to the generation checks the key twice a second, and when the key is gone it cancels the provider stream and writes a terminal event with reason: "abandoned".

Three details decide whether this works.

The client renews the lease, not the SSE handler. It is tempting to renew from the server side, inside the loop that streams events to the reader. That handler is exactly as blind to silent drops as the connection-bound generator, so it would keep renewing for a client that vanished minutes ago. In the Docker run above, the handler for the dropped client kept writing for the full 60 seconds, and a lease renewed from that loop would have stayed alive for every one of the 2,210 orphan tokens.

The usable tolerance is the grace window minus the heartbeat interval. The lease was last renewed somewhere inside the heartbeat interval before the client left. With an 8-second grace and a 2-second heartbeat, a reader that came back after 3 seconds found its run already cancelled. I hit this in the lab twice before I wrote the rule down.

Replay makes the grace window free for returning users. Tokens generated while a user is away are not waste if the user comes back, because the log hands them over on reconnect. Only runs whose readers never return add to the Orphan Tail.

Here is the whole loop in the lab, with an 8-second grace, the client renewing every 2 seconds, and a reader that stops at about 2 seconds.

After its reader disconnects, a presence-bound run takes one of the paths in this sequence.

sequenceDiagram
    participant C as Client tab
    participant A as FastAPI
    participant R as Redis
    participant P as Provider
    C->>A: POST /runs
    A->>R: SET presence EX grace
    A->>P: open stream
    loop every delta
        P-->>A: delta
        A->>R: XADD run log
    end
    C->>A: GET /runs/id/events
    A-->>C: replayed and live events
    loop every 5 seconds while following
        C->>A: POST /runs/id/presence
        A->>R: SET presence EX grace XX
    end
    rect rgb(152, 216, 200)
        Note over C,R: Reload: back inside the grace window
        C->>A: GET events with Last-Event-ID
        A-->>C: missed events, then live
    end
    rect rgb(255, 217, 61)
        Note over C,P: Tab closed: no renewals arrive
        A->>R: EXISTS presence returns 0
        A->>P: cancel stream
        A->>R: XADD done, reason abandoned
    end

FastAPI SSE implementation with Redis Streams and a presence lease

I ran every test in this article against the server below. It keeps the run's events in a Redis Stream, so any worker can serve any reader, and it holds three small keys per run: the presence lease, a stop flag and an owner heartbeat.

import asyncioimport jsonimport loggingimport osimport reimport uuidfrom collections.abc import AsyncIteratorfrom contextlib import aclosing, asynccontextmanagerfrom typing import Annotatedimport redis.asyncio as redisfrom fastapi import Depends, FastAPI, Header, HTTPExceptionfrom fastapi.sse import EventSourceResponse, ServerSentEventfrom pydantic import BaseModelfrom provider import stream_tokensREDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379")GRACE_S = int(os.environ.get("GRACE_S", "45"))  # presence lease lengthLOG_TTL_S = 3600  # how long a finished run stays replayableCHECK_EVERY_S = 0.5  # how often the supervisor reads the leaseOWNER_TTL_S = 5  # a run whose supervisor stops renewing this is presumed lostlog = logging.getLogger("runs")r = redis.from_url(REDIS_URL, decode_responses=True)# Strong references, or the event loop may drop a running taskruns: dict[str, asyncio.Task] = {}@asynccontextmanagerasync def lifespan(app: FastAPI):    yield    for task in runs.values():        task.cancel()    await asyncio.gather(*runs.values(), return_exceptions=True)    await r.aclose()app = FastAPI(lifespan=lifespan)def key(run_id: str, part: str) -> str:    return f"run:{run_id}:{part}"async def append(run_id: str, event: str, **data) -> None:    await r.xadd(key(run_id, "log"), {"event": event, "data": json.dumps(data)})async def produce(run_id: str, prompt: str, stats: dict) -> None:    # aclosing() makes cancellation reach the provider stream instead of leaving it to GC    async with aclosing(stream_tokens(prompt)) as deltas:        async for delta in deltas:            stats["deltas"] += 1            await append(run_id, "text", delta=delta)async def lifetime_verdict(run_id: str) -> str | None:    try:        _, _, stopped, present = await (            r.pipeline(transaction=False)            # The owner key proves this worker is still alive            .set(key(run_id, "owner"), 1, ex=OWNER_TTL_S)            .expire(key(run_id, "log"), LOG_TTL_S)  # a crashed run's log still expires            .exists(key(run_id, "stop"))            .exists(key(run_id, "presence"))            .execute()        )    except redis.RedisError:        # An unreachable Redis is not evidence that the reader left. Never cancel on it.        log.warning("lease_check_failed run=%s", run_id)        return None    if stopped:        return "stopped"    if not present:        return "abandoned"    return Noneasync def run_generation(run_id: str, prompt: str) -> None:    stats = {"deltas": 0}    producer = asyncio.create_task(produce(run_id, prompt, stats))    reason = "complete"    try:        while not producer.done():            await asyncio.wait({producer}, timeout=CHECK_EVERY_S)            if not producer.done() and (verdict := await lifetime_verdict(run_id)):                reason = verdict                producer.cancel()                await asyncio.wait({producer})        if not producer.cancelled() and (exc := producer.exception()):            await terminal(run_id, "error", message=type(exc).__name__, **stats)        else:            await terminal(run_id, "done", reason=reason, **stats)    except asyncio.CancelledError:  # worker shutdown        producer.cancel()        await terminal(run_id, "done", reason="shutdown", **stats)        raise    finally:        runs.pop(run_id, None)async def terminal(run_id: str, event: str, **data) -> None:    # Best effort: if Redis is the thing that failed, readers fall back to the owner key    try:        await append(run_id, event, **data)        await r.expire(key(run_id, "log"), LOG_TTL_S)    except redis.RedisError:        log.error("terminal_write_failed run=%s event=%s", run_id, event)class RunRequest(BaseModel):    prompt: str# Auth is omitted to keep the lifetime logic visible. In production, bind run_id to the# authenticated user and check that binding on every endpoint below.@app.post("/runs", status_code=202)async def start_run(    body: RunRequest,    idempotency_key: Annotated[str | None, Header()] = None,) -> dict:    run_id = uuid.uuid4().hex    if idempotency_key:        claim = f"idem:{idempotency_key}"        # A retried POST must not start a second generation. First writer wins.        if not await r.set(claim, run_id, ex=LOG_TTL_S, nx=True):            return {"run_id": await r.get(claim)}    await r.set(key(run_id, "owner"), 1, ex=OWNER_TTL_S)    # The lease starts now, so the gap before the first GET does not count as absence    await r.set(key(run_id, "presence"), 1, ex=GRACE_S)    await append(run_id, "start")    runs[run_id] = asyncio.create_task(run_generation(run_id, body.prompt))    return {"run_id": run_id}@app.post("/runs/{run_id}/presence", status_code=204)async def renew_presence(run_id: str) -> None:    lease = key(run_id, "presence")    # xx=True renews only a live lease; a lapsed lease means the run was already cancelled    remaining_ms, renewed = await (        r.pipeline(transaction=False)        .pttl(lease)        .set(lease, 1, ex=GRACE_S, xx=True)        .execute()    )    if not renewed:        # The client came back after the grace window closed        log.info("presence_late run=%s", run_id)        raise HTTPException(410, "run is no longer live")    # Time since the previous renewal. Normal beats sit near HEARTBEAT_MS;    # reconnects are the tail of this distribution.    log.info("presence_gap run=%s ms=%d", run_id, GRACE_S * 1000 - remaining_ms)@app.post("/runs/{run_id}/stop", status_code=204)async def stop_run(run_id: str) -> None:    await r.set(key(run_id, "stop"), 1, ex=LOG_TTL_S)async def existing_run(run_id: str) -> str:    if not await r.exists(key(run_id, "log")):        raise HTTPException(404, "unknown or expired run")    return run_idSTREAM_ID = re.compile(r"^\d+-\d+$")async def resume_cursor(last_event_id: Annotated[str | None, Header()] = None) -> str:    if last_event_id is None:        return "0-0"    if not STREAM_ID.match(last_event_id):        raise HTTPException(400, "malformed Last-Event-ID")    return last_event_idTERMINAL = {"done", "error"}@app.get("/runs/{run_id}/events", response_class=EventSourceResponse)async def run_events(    run_id: Annotated[str, Depends(existing_run)],    cursor: Annotated[str, Depends(resume_cursor)],) -> AsyncIterator[ServerSentEvent]:    while True:        batch = await r.xread({key(run_id, "log"): cursor}, block=5_000, count=200)        if not batch and not await r.exists(key(run_id, "owner")):            # Nothing new and no live supervisor: the worker died            # before writing a terminal event            yield ServerSentEvent(data={"message": "producer lost"}, event="error")            return        for _stream, entries in batch:            for entry_id, fields in entries:                cursor = entry_id                yield ServerSentEvent(                    raw_data=fields["data"], event=fields["event"], id=entry_id                )                if fields["event"] in TERMINAL:                    return

aclosing() is not decoration. Cancelling a task that is iterating an async generator does not close the generator underneath; it is closed when garbage collection finalizes it. With a real provider that generator owns an open HTTP stream, so without aclosing() a cancelled run can keep draining tokens. In the lab, the fake provider records when its finally block runs, and every stopped or abandoned run closed it.

Event IDs are Redis Stream IDs such as 1789494043550-0. They increase monotonically, so they double as the SSE id field, and XREAD with a stream ID returns only entries after it. This is the same replay contract that LangGraph checkpointers provide for graph state, applied to a chat stream. That makes Last-Event-ID resume a single command, with nothing to translate. resume_cursor rejects anything that is not a stream ID with a 400 before the stream opens.

Use raw_data for payloads you have already encoded. FastAPI's ServerSentEvent.data always JSON-encodes its value, so data="hello" goes out on the wire as data: "hello", quotes included. FastAPI's native response also sends a : ping comment every 15 seconds and sets Cache-Control: no-cache and X-Accel-Buffering: no for you. It does not set no-transform, which matters in the next section.

One failure is invisible to the lease, and the owner key covers it. If a worker dies without running its shutdown hooks, after an out-of-memory kill for example, no terminal event is ever written, and a reader would otherwise wait forever, since the keep-alive pings keep its connection busy. I force-killed a worker mid-run while a reader was attached to a second worker. Seven seconds later, that reader received event: error with producer lost.

Idempotency-Key closes the loop the rest of the article opens. A retried POST is the same waste in a worse form, because it starts a second full generation rather than extending one. The header claims the key with SET NX, and the loser of the race gets the winner's run_id. I fired five concurrent requests with one key and got one run ID and one generation.

One last note on wiring: lifespan replaces the deprecated @app.on_event hooks, and the two styles do not mix.

Plugging Claude Sonnet 5 into the FastAPI SSE endpoint

stream_tokens is the only provider-specific function. This adapter uses the Anthropic Python SDK; its stream manager closes the HTTP response in __aexit__, so the cancellation reaches the network.

# provider.pyfrom collections.abc import AsyncIteratorfrom anthropic import AsyncAnthropicclient = AsyncAnthropic()  # reads ANTHROPIC_API_KEY from the environmentasync def stream_tokens(prompt: str) -> AsyncIterator[str]:    async with client.messages.stream(        model="claude-sonnet-5",        max_tokens=8_000,        messages=[{"role": "user", "content": prompt}],    ) as stream:        async for text in stream.text_stream:            yield text

I have not run this adapter against a live Anthropic call. Every measurement in this article uses the fake provider and no credentials, so confirm in your own environment that cancellation closes the network stream the way it does in the lab.

Notice that the done event counts deltas, not tokens. That is deliberate. Providers report usage at the end of a stream: Anthropic's cumulative counts arrive in message_delta, and OpenAI's Chat Completions usage chunk is, per openai/openai-openapi#539, "never received if the stream is aborted mid-generation". A cancelled run is precisely the run whose real usage you will not receive, so estimate its Orphan Tail from what you logged, and reconcile against the provider's usage dashboard.

Next.js rewrites() buffers SSE in every browser: use a route handler

Your browser code should talk to the Next.js origin, not to FastAPI directly, which leaves two ways to forward /api/runs. I tested both under next start on Next.js 16.3.5, timing when each body byte arrived for a 100-token stream at 10 tokens per second.

Two results change how you should test this. Every browser sends Accept-Encoding: gzip. Plain curl does not. So the rewrite looks fine in a terminal and delivers the whole answer at once in Chrome. And FastAPI's native SSE sends no-cache without no-transform, which is exactly the combination that gets compressed.

That 30-second cut matches the rewrite proxy's source, where proxyTimeout defaults to 30,000 milliseconds and is configurable as experimental.proxyTimeout. In my tests it acted as an idle timeout: a continuous 50-second stream went through untouched. It truncates with a 200 status and no error event, so a client without a terminal-event check shows a half answer as finished.

Use a route handler. It streams without compression, it lets you choose which headers cross the boundary, and it gives you one place to attach authentication later.

// app/api/runs/[[...path]]/route.ts// streams that are idle for 30s.const BACKEND_URL = process.env.BACKEND_URL ?? "http://127.0.0.1:8804";const FORWARDED = ["content-type", "last-event-id", "idempotency-key"];export const dynamic = "force-dynamic";async function proxy(req: Request, ctx: { params: Promise<{ path?: string[] }> }) {  const { path = [] } = await ctx.params;  const suffix = path.map((p) => `/${encodeURIComponent(p)}`).join("");  const headers = new Headers();  for (const name of FORWARDED) {    const value = req.headers.get(name);    if (value) headers.set(name, value);  }  const upstream = await fetch(`${BACKEND_URL}/runs${suffix}`, {    method: req.method,    headers,    body: req.method === "POST" ? await req.text() : undefined,    // Safe to propagate: this only ends a reader. The generation's    // lifetime is the presence lease.    signal: req.signal,    cache: "no-store",  });  return new Response(upstream.body, {    status: upstream.status,    headers: {      "content-type": upstream.headers.get("content-type") ?? "application/json",      "cache-control": "no-cache, no-transform",    },  });}export { proxy as GET, proxy as POST };

It is an optional catch-all route, [[...path]], because POST /api/runs has no path segments and a required catch-all returns 404 for it. Look at the signal: req.signal line too. Under connection-bound, propagating the browser's abort into the upstream fetch would cancel generation. Here it only closes a reader, which is what makes the proxy boring.

React client: resume with Last-Event-ID, stop without cancelling on unmount

On the client there are four jobs: it parses the stream, renews the lease, reconnects from the last event ID when the connection stalls, and treats Stop as an explicit request. @microsoft/fetch-event-source used to be the default for the parsing part. Its last release, 2.0.1, was in April 2021, and by default it closes the connection when the document is hidden, which under a connection-bound backend cancels the generation every time a user switches tabs. This client uses eventsource-parser and owns its reconnect logic. AG-UI's protocol leaves the same reconnect gap open, which is the snapshot tax its state deltas pay without an event log behind them.

// lib/follow-run.tsimport { createParser } from "eventsource-parser";export type RunEvent =  | { event: "start"; id: string }  | { event: "text"; id: string; delta: string }  | {      event: "done";      id: string;      reason: "complete" | "stopped" | "abandoned" | "shutdown";      deltas: number;    }  | { event: "error"; id: string; message: string };export type TerminalEvent = Extract<RunEvent, { event: "done" | "error" }>;const STALL_MS = 20_000; // longer than FastAPI's 15s keep-alive pingconst HEARTBEAT_MS = 5_000; // well under the server's GRACE_Sconst GIVE_UP_AFTER_MS = 60_000; // stop reconnecting and let the UI show a failureexport class RunGoneError extends Error {}export async function startRun(prompt: string, base = "/api"): Promise<string> {  const res = await fetch(`${base}/runs`, {    method: "POST",    // If this POST is ever retried, the key stops it starting a second generation    headers: {      "content-type": "application/json",      "idempotency-key": crypto.randomUUID(),    },    body: JSON.stringify({ prompt }),  });  if (!res.ok) throw new Error(`start failed: ${res.status}`);  return (await res.json()).run_id;}export async function stopRun(runId: string, base = "/api"): Promise<void> {  await fetch(`${base}/runs/${runId}/stop`, { method: "POST" });}// Follows a run until it reaches a terminal event. Aborting `signal` detaches this reader// only; the run keeps going until its presence lease lapses or someone calls stopRun().export async function followRun(  runId: string,  onEvent: (e: RunEvent) => void,  signal: AbortSignal,  base = "/api",): Promise<TerminalEvent> {  const renew = async () => {    try {      const url = `${base}/runs/${runId}/presence`;      const res = await fetch(url, { method: "POST", signal });      // 410 is the server's presence_late: this reader came back after the grace window      if (res.status === 410) console.warn(`run ${runId}: presence lease lapsed`);    } catch {      // A failed heartbeat is not fatal on its own: the stall timer      // and the give-up budget decide    }  };  void renew(); // reattaching after a reload should reassert presence at once  const heartbeat = setInterval(renew, HEARTBEAT_MS);  let lastEventId: string | undefined;  let failures = 0;  let failingSince = 0;  try {    for (;;) {      try {        const terminal = await readOnce();        if (terminal) return terminal;        failures = 0; // clean end without a terminal event: reconnect at once        failingSince = 0;      } catch (err) {        if (signal.aborted || err instanceof RunGoneError) throw err;        failingSince ||= Date.now();        // Bounded, not forever        if (Date.now() - failingSince > GIVE_UP_AFTER_MS) throw err;        const delay = Math.min(500 * 2 ** failures++, 5_000) * (0.5 + Math.random());        await new Promise((resolve) => setTimeout(resolve, delay));      }    }  } finally {    clearInterval(heartbeat);  }  async function readOnce(): Promise<TerminalEvent | undefined> {    const attempt = new AbortController();    const abort = () => attempt.abort();    signal.addEventListener("abort", abort, { once: true });    let stall = setTimeout(abort, STALL_MS);    let terminal: TerminalEvent | undefined;    const parser = createParser({      onEvent(message) {        if (message.id) lastEventId = message.id;        const data = JSON.parse(message.data);        const e = { event: message.event, id: message.id, ...data } as RunEvent;        if (e.event === "done" || e.event === "error") terminal = e;        onEvent(e);      },    });    try {      const res = await fetch(`${base}/runs/${runId}/events`, {        headers: lastEventId ? { "last-event-id": lastEventId } : {},        signal: attempt.signal,        cache: "no-store",      });      if (res.status === 404) throw new RunGoneError(`run ${runId} expired`);      if (!res.ok || !res.body) throw new Error(`events failed: ${res.status}`);      const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();      for (;;) {        const { value, done } = await reader.read();        if (done) return terminal;        clearTimeout(stall); // any bytes count, including ": ping" comments        stall = setTimeout(abort, STALL_MS);        parser.feed(value);        if (terminal) {          await reader.cancel();          return terminal;        }      }    } finally {      clearTimeout(stall);      signal.removeEventListener("abort", abort);    }  }}

A stall timer is the client's own silent-drop detector. Because the server sends at least a ping every 15 seconds, 20 seconds without a single byte means the connection is dead even though fetch has not failed. So the client aborts that attempt and reconnects with the last ID it saw. I killed the Next.js server four seconds into a run and restarted it: the client reconnected by itself and assembled all 400 tokens in order. With an 8-second grace the same test ended abandoned, because the heartbeats failed during the outage too, which is the tolerance arithmetic again.

Once followRun does the work, the React hook is small. One line in the effect cleanup matters more than the rest.

// lib/use-run.ts"use client";import { useCallback, useEffect, useRef, useState } from "react";import { followRun, RunGoneError, startRun, stopRun, type RunEvent } from "./follow-run";type Status = "idle" | "streaming" | "complete" | "stopped" | "abandoned" | "failed";const STORAGE_KEY = "active-run";export function useRun() {  const [text, setText] = useState("");  const [status, setStatus] = useState<Status>("idle");  const runId = useRef<string | null>(null);  const reader = useRef<AbortController | null>(null);  const follow = useCallback(async (id: string) => {    reader.current?.abort();    const controller = new AbortController();    reader.current = controller;    runId.current = id;    setText(""); // a fresh reader replays the log from the start    setStatus("streaming");    const onEvent = (e: RunEvent) => {      if (e.event === "text") setText((t) => t + e.delta);    };    try {      const end = await followRun(id, onEvent, controller.signal);      if (end.event === "error" || end.reason === "shutdown") setStatus("failed");      else setStatus(end.reason);      sessionStorage.removeItem(STORAGE_KEY);    } catch (err) {      if (controller.signal.aborted) return; // unmount or a newer run took over      if (err instanceof RunGoneError) sessionStorage.removeItem(STORAGE_KEY);      setStatus("failed");    }  }, []);  // After a reload, reattach to the run this tab was following.  useEffect(() => {    const saved = sessionStorage.getItem(STORAGE_KEY);    if (saved) void follow(saved);    return () => reader.current?.abort(); // detach only; never stop the run from cleanup  }, [follow]);  const send = useCallback(    async (prompt: string) => {      const id = await startRun(prompt);      sessionStorage.setItem(STORAGE_KEY, id);      await follow(id);    },    [follow],  );  const stop = useCallback(async () => {    if (runId.current) await stopRun(runId.current); // the reader receives done: stopped  }, []);  return { text, status, send, stop };}

After a reload the in-memory text is gone, so the hook replays the log from the start instead of sending Last-Event-ID. Keep the ID for reconnects inside a living page. setText per delta is fine for a demo and wrong for a long answer. Batching renders and parsing partial markdown are covered in Frontend Architecture for GenAI, and they apply unchanged on top of this hook.

Sizing the grace window: simulating the three policies

I could not find a published production rate for how often chat users reload, switch networks or abandon a generation. So the numbers below come from a deterministic simulation, with the measured inputs and the assumed inputs kept apart.

• Measured: Claude Sonnet 5 outputs 76.2 tokens per second (Artificial Analysis, accessed 2026-09-15). That page is a live leaderboard and read 84.1 tokens per second a day later, which moves the presence-bound figures by a few percent and changes none of the orderings below. Its output price is USD 10 per million tokens (Anthropic pricing, 2026-09-15). A silent drop is not detected before the answer ends (the Docker test above).
• Assumed: 20% of generations are interrupted at a uniform point. Interruptions are split evenly between reloads (back in 2 to 12 seconds), network switches (back in 5 to 40 seconds, silent), tab closes and devices that go away silently. Heartbeats are every 5 seconds.

Wasted tokens are a budget line like any other, and agent cost governance is where that budget gets set. A repaid token is one the user must pay for again after an interruption destroyed or truncated the answer. Orphan Tail tokens are never read by anyone. Wasted cost counts both. Figures are per million generations.

Read the rows in pairs. For a four-second answer the lifetime policy barely matters, as long as it is not connection-bound: there is almost nothing left to generate when a user leaves. For a 105-second answer, connection-bound is worst on both columns at once.

A 5-second grace, the kind of number you would copy from a transport timeout, loses almost as many answers as connection-bound. Most of the value sits at the other end of the sweep: a 45-second grace loses none under these assumptions and still wastes 35% less than completion-bound, because it bounds the Orphan Tail of every user who is not coming back.

Where the ranking between the two published defaults flips

Connection-bound's cost is driven almost entirely by silent drops, and completion-bound's by users who never come back, so neither ordering is universal. These two sweeps hold the answer at 8,000 tokens and the grace at 45 seconds, and vary one assumption at a time. Wasted cost is per million generations.

Connection-bound overtakes completion-bound on cost at about a 1% silent share, when half of interrupted users come back. Read the second table and the ordering flips the other way: below roughly a quarter of users returning, connection-bound is the cheaper of the two published defaults, because it stops paying for people who left and there are few reloads left to re-bill. So a desktop product on stable networks whose users rarely come back can run connection-bound and pay less than a resumable stream would cost.

What does not flip is the rest. Connection-bound lost between 20,150 and 159,360 answers per million in these runs, and presence-bound lost none while costing the least in every row of both tables. That is the case for the third policy: it is not a compromise between the two defaults, it dominates them under every mix I measured.

So size the grace from your own users, and renew_presence already records what you need. Each presence_gap line is the time since the previous renewal: ordinary heartbeats cluster at the heartbeat interval, and reconnects form the tail. In the lab a reload appeared as presence_gap ms=5007 between beats of about 2,015 ms. Every presence_late line is a user who came back after the window closed. Set the grace to the heartbeat interval plus the 95th percentile of the reconnect tail, then watch how often presence_late fires. Swap your own rates into the simulation below before you trust my numbers.

"""Deterministic simulation of three generation-lifetime policies.Measured inputs: output speed (Artificial Analysis, Claude Sonnet 5,2026-09-15), output price (Anthropic pricing page, 2026-09-15), and the labresult that a silent drop is not detected before the generation ends.Assumed inputs (no published production rates were found) are markedASSUMED and swept."""import randomimport sysSPEED = 76.2  # tok/s, Claude Sonnet 5 (Artificial Analysis)PRICE = 10 / 1e6  # USD per output token, Claude Sonnet 5N = 100_000SEED = 7# ASSUMED: one generation in five is interrupted, at a uniform point in the stream.P_INTERRUPT = 0.20# ASSUMED: returning users come back 2-12s after a reload, 5-40s after a network switch.RETURN = {"reload": (2, 12), "switch": (5, 40)}def simulate(length, kinds, grace, renew=5.0, seed=SEED):    rng = random.Random(seed)    names = list(kinds)    weights = [kinds[k] for k in names]    policies = ("connection", "completion", "presence")    out = {p: {"lost": 0, "orphan": 0.0, "repaid": 0.0} for p in policies}    for _ in range(N):        if rng.random() >= P_INTERRUPT:            continue        kind = rng.choices(names, weights)[0]        # Tokens generated when the reader went away        done_at_leave = rng.random() * length        remaining = length - done_at_leave        returns = kind in RETURN        back_after = rng.uniform(*RETURN[kind]) if returns else None        silent = kind in ("switch", "device")  # no FIN reaches the server        # The lease was last renewed somewhere in the renew interval        # before the reader left        tolerance = grace - rng.random() * renew        c = out["connection"]        if silent:            c["orphan"] += remaining  # measured: the server never notices before the end        if returns:            c["lost"] += 1            # The user asks again and pays for the same prefix            c["repaid"] += done_at_leave        m = out["completion"]        if not returns:            m["orphan"] += remaining        p = out["presence"]        ran_on = min(remaining, max(tolerance, 0) * SPEED)        if not returns:            p["orphan"] += ran_on        elif back_after > tolerance and ran_on < remaining:            # Truncated: the prefix is kept in the log, the answer is not            p["lost"] += 1            p["repaid"] += done_at_leave + ran_on    return outdef silent_mix(silent_share, returning=0.5):    """Half the interruptions come back; `silent_share` of them leave no FIN behind."""    return {        "reload": returning * (1 - silent_share),        "switch": returning * silent_share,        "tab_close": (1 - returning) * (1 - silent_share),        "device": (1 - returning) * silent_share,    }def table(title, length, kinds, grace):    res = simulate(length, kinds, grace)    secs = length / SPEED    print(        f"\n{title}: {length} output tokens (~{secs:.0f}s), "        f"grace {grace}s, per million generations"    )    print(        f"{'policy':<12}{'answers lost':>14}{'orphan tail':>15}"        f"{'repaid':>15}{'wasted USD':>12}"    )    for policy, v in res.items():        k = 1_000_000 / N        wasted = (v["orphan"] + v["repaid"]) * k        print(            f"{policy:<12}{v['lost'] * k:>14,.0f}{v['orphan'] * k:>15,.0f}"            f"{v['repaid'] * k:>15,.0f}{wasted * PRICE:>12,.0f}"        )if __name__ == "__main__":    # ASSUMED, swept below    mix = {"reload": 0.25, "switch": 0.25, "tab_close": 0.25, "device": 0.25}    for length in (300, 2_000, 8_000):        table("even mix", length, mix, grace=20)    abandon_heavy = {"reload": 0.1, "switch": 0.1, "tab_close": 0.4, "device": 0.4}    table("abandon-heavy mix", 8_000, abandon_heavy, grace=20)    for g in (5, 10, 20, 45):        table("grace sweep", 8_000, mix, grace=g)    for share in (0.0, 0.01, 0.02, 0.05, 0.25, 0.5, 1.0):        table(f"silent {share:.0%}, half return", 8_000, silent_mix(share), grace=45)    for back in (0.1, 0.2, 0.3, 0.5, 0.8):        mix_back = silent_mix(0.5, returning=back)        table(f"silent 50%, {back:.0%} return", 8_000, mix_back, grace=45)

Production edge cases: multiple workers, hidden tabs, nginx and Cloudflare

Several uvicorn workers. The runs dictionary is per process, and nothing else is. Redis holds the lease, the stop flag, the owner heartbeat and the log, so a Stop or a reader can land on any worker. The owner key turns a crashed worker into an explicit producer lost error instead of a reader that waits forever.

Several tabs on one run. Any tab's renewal keeps the run alive, which is the behaviour you want. Closing one tab does not cancel an answer another tab is still showing.

Hidden tabs. Browsers slow timers in background pages. Chrome's intensive throttling applies once a page has been hidden for more than 5 minutes, the timer chain is at least five deep, and the page has been silent for 30 seconds. Then "the browser will check timers in this group once per minute", and a setInterval heartbeat qualifies, because each iteration extends the chain. A 5-second heartbeat can become a 60-second one. That means a grace below 60 seconds cancels runs in tabs hidden for more than five minutes. I think that is the correct outcome for a chat answer. If your product is a long agent run that users are expected to leave in the background, it is not, and completion-bound with a hard cap is the better policy for that workload.

Write volume. One XADD per delta is about 76 writes per second per active run at Sonnet 5 speed. If that matters for your Redis, coalesce deltas into one entry every 50 milliseconds; Last-Event-ID resume works the same. Each attached reader also holds a connection in a blocking XREAD, so size your connection pool for concurrent readers, not concurrent runs.

A leaked run ID is a cost lever. Anyone holding a run_id can renew its lease forever, which keeps a generation alive and billing, or call Stop on someone else's answer. Bind run_id to the authenticated session at every endpoint, and rate-limit the presence endpoint per session rather than per run.

When Redis itself fails. One line in lifetime_verdict carries the whole argument: a failed EXISTS returns None, never "abandoned". Read a Redis outage as absence and a blip cancels every live run at once. I restarted the container mid-run to check, and the supervisor logged lease_check_failed and kept generating. The log is a different matter, because it is the run's durability boundary. A restart without persistence loses the events, readers get a 404 on reconnect, and the run is gone. Run Redis with append-only file (AOF) persistence or a replica, and treat a lost log as a lost run rather than pretending otherwise.

nginx. X-Accel-Buffering is a response header that nginx reads from the upstream. The proxy module documentation describes it as "passing 'yes' or 'no' in the 'X-Accel-Buffering' response header field", so proxy_set_header X-Accel-Buffering no in your config sets a request header and does nothing. FastAPI's native SSE already sends the response header. proxy_read_timeout defaults to 60 seconds between two successive reads, which the 15-second ping covers.

Cloudflare. The proxy read timeout is 125 seconds, after which the client gets a 524. FastAPI's 15-second ping keeps a slow reasoning phase alive. Cloudflare Tunnel buffers responses unless the origin sends Content-Type: text/event-stream.

HTTP/2. Do not copy Connection: keep-alive or Transfer-Encoding: chunked into your streaming headers, even though several guides still suggest it. RFC 9113 requires that a message containing connection-specific header fields "MUST be treated as malformed". On HTTP/1.1, the other limit to remember is six connections per browser and origin, which several open chat tabs can reach.

Provider-side logs. OpenAI's background mode keeps the event log for you: create the response with background: true and stream: true, and resume with starting_after and the last sequence_number. It replaces the Redis Stream, not the lifetime decision. When the lease lapses you still call POST /v1/responses/{id}/cancel. OpenAI also warns that time to first token is higher for background responses. A lifetime policy also says nothing about a provider that degrades mid-answer, which is a question of retries, fallback and circuit breaking.

Disconnect-driven handlers elsewhere in your stack. If the stream drives a LangGraph agent rather than a single model call, What a Client Disconnect Commits shows how the handler's shape decides what a disconnect leaves behind. Its OnDisconnect policy has two values, abandon and complete. This article argues for a third.

Which lifetime policy fits your workload

The sweeps above decide this for you once you know two things about your traffic: how long an answer takes, and whether interrupted users come back.

Two of those rows are the published defaults, used where they are right rather than as a global setting. That is the whole argument: the policy is a property of the workload, not of the framework.

Production checklist for ChatGPT-style streaming

1. Write down the lifetime policy for each streaming endpoint. If the answer is "whatever the framework does", it is connection-bound.
2. For answers that finish in under about five seconds, use completion-bound with a Stop endpoint, and skip the lease.
3. For longer answers, use Presence-Bound Generation: a Redis lease the client renews, a supervisor that cancels when it lapses, and an explicit Stop.
4. Renew presence from the client, never from the SSE handler.
5. Set the grace window to the heartbeat interval plus the 95th percentile of your measured reconnect gap. Start around 45 seconds until you have data.
6. Cancel provider streams through aclosing() or the SDK's context manager, and confirm in a test that the provider stream actually closes.
7. Give every event a Redis Stream ID, validate Last-Event-ID before streaming, and replay from the start after a page reload.
8. Keep an owner heartbeat so a crashed worker produces a terminal error.
9. Proxy SSE through a Next.js route handler, not rewrites(). If you keep a rewrite, send Cache-Control: no-cache, no-transform from the backend or set compress: false.
10. Test streaming with curl --compressed or a real browser. Plain curl hides gzip buffering.
11. Keep a keep-alive ping under every idle timeout in the chain: 30 seconds for Next.js rewrites, 60 for nginx, 125 for Cloudflare.
12. Send an Idempotency-Key on every POST that starts a generation, and never retry that POST without one.
13. Bound the client's reconnect loop. Retrying forever hides a backend outage behind a spinner.
14. Run Redis with persistence or a replica, and never read a failed lease check as absence.
15. Do not trust end-of-stream usage for cancelled runs. Estimate their Orphan Tail from the deltas you logged.

The question this guide started with was who decides when a generation stops. In most codebases, the TCP stack makes that decision, and it is wrong in both directions: too quick for the user who reloads, and more than 15 minutes too slow for the one who walked away. A presence lease hands the decision back to the one party that knows whether a person is still waiting.

References

• FastAPI. Server-Sent Events (SSE). https://fastapi.tiangolo.com/tutorial/server-sent-events/
• FastAPI. Release 0.135.0 (2026-03-01). https://github.com/fastapi/fastapi/releases/tag/0.135.0
• FastAPI. Lifespan Events. https://fastapi.tiangolo.com/advanced/events/
• FastAPI source. fastapi/sse.py. https://github.com/fastapi/fastapi/blob/master/fastapi/sse.py
• Starlette source. starlette/responses.py, StreamingResponse disconnect handling (1.6.0, 2026-08-08). https://github.com/Kludex/starlette/blob/main/starlette/responses.py
• uvicorn source. uvicorn/protocols/http/h11_impl.py, ASGI spec_version 2.3. https://github.com/Kludex/uvicorn/blob/main/uvicorn/protocols/http/h11_impl.py
• ASGI. HTTP & WebSocket ASGI Message Format. https://asgi.readthedocs.io/en/latest/specs/www.html
• Linux kernel. IP Sysctl: tcp_retries2. https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html
• KristinZ. (2026, Aug 22). Your LLM App Is Wasting Money: What Happens When Users Close the Tab? DEV. https://dev.to/kristinz/your-llm-app-is-wasting-money-what-happens-when-users-close-the-tab-4k01
• Josh (Upstash). (2025, Apr 14). How to Build LLM Streams That Survive Reconnects, Refreshes, and Crashes. https://upstash.com/blog/resumable-llm-streams
• Vercel. resumable-stream. https://github.com/vercel/resumable-stream
• Vercel. AI SDK UI: Chatbot Resume Streams. https://ai-sdk.dev/docs/ai-sdk-ui/chatbot-resume-streams
• Vercel. Troubleshooting: Abort and resumable streams. https://ai-sdk.dev/docs/troubleshooting/abort-breaks-resumable-streams
• Quinn, M. (Ably). (2026, Jun 25). Stop vs disconnect: why canceling AI streaming is harder than it looks. https://ably.com/blog/stop-vs-disconnect-canceling-ai-streaming
• Dawson, A. (Ably). (2026, Mar 19). The missing transport layer in user-facing AI applications. https://ably.com/blog/agentic-ai-session-transport-layer-production-failures
• Ably. AI Transport: Agent presence. https://ably.com/docs/ai-transport/features/agent-presence
• OpenAI. Background mode. https://developers.openai.com/api/docs/guides/background
• OpenAI. openai-openapi issue #539: usage chunk on aborted streams. https://github.com/openai/openai-openapi/issues/539
• Anthropic. Streaming Messages. https://docs.anthropic.com/en/docs/build-with-claude/streaming
• Anthropic. Pricing. https://docs.anthropic.com/en/docs/about-claude/pricing
• Artificial Analysis. Claude Sonnet 5 (accessed 2026-09-15). https://artificialanalysis.ai/models/claude-sonnet-5
• Redis. XREAD. https://redis.io/docs/latest/commands/xread/
• Redis. XADD. https://redis.io/docs/latest/commands/xadd/
• Next.js. compress. https://nextjs.org/docs/app/api-reference/config/next-config-js/compress
• Next.js source. proxy-request.ts, default proxyTimeout (canary). https://github.com/vercel/next.js/blob/canary/packages/next/src/server/lib/router-utils/proxy-request.ts
• nginx. ngx_http_proxy_module. https://nginx.org/en/docs/http/ngx_http_proxy_module.html
• Cloudflare. Connection limits. https://developers.cloudflare.com/fundamentals/reference/connection-limits/
• Cloudflare. Tunnel troubleshooting. https://developers.cloudflare.com/tunnel/troubleshooting/
• Thomson, M. & Benfield, C. (2022). RFC 9113: HTTP/2, section 8.2.2. https://www.rfc-editor.org/rfc/rfc9113.html
• WHATWG. HTML Standard: Server-sent events. https://html.spec.whatwg.org/multipage/server-sent-events.html
• Chrome for Developers. (2021, Jan 18). Heavy throttling of chained JS timers beginning in Chrome 88. https://developer.chrome.com/blog/timer-throttling-in-chrome-88
• Azure. fetch-event-source. https://github.com/Azure/fetch-event-source
• rexxars. eventsource-parser. https://github.com/rexxars/eventsource-parser

Related Articles

Frontend AI

• CopilotKit in Production: Where the Abstraction Holds and Where You're on Your Own

Frontend Architecture

• Frontend Architecture for GenAI: Why Your React Patterns Don't Work Anymore

More Articles

• FastAPI + LangGraph: What a Client Disconnect Commits
• Building a Local Banking Sandbox: Why I Created DevBankSDK