Why streaming feels fast

When you ask ChatGPT or Claude something, the first words show up within a second or two and the rest keeps arriving while you read. The full answer might take fifteen seconds to write, but you spend most of those seconds reading, so the wait barely registers.

How fast the text arrives matters less than when it starts. A stream runs far ahead of anyone reading it:

Once the first words land, you never catch up to the model, so the only wait you notice is the one before the first word.

How RENEW's chat used to work

You'd send a message, a shimmering typing indicator would sit there for close to ten seconds, and then the whole reply would show up.

When I redesigned the chat screen in June, I added a word-by-word reveal so replies at least came in like they were being typed. It splits the reply into about 30 steps, 25 ms apart:

const chunkSize = Math.max(1, Math.floor(words.length / 30));
setTimeout(() => setVisibleCount((c) => c + chunkSize), 25);

It looks like streaming. It only starts once the entire reply has arrived, though, so all it really did was add most of a second to the end of the wait.

The path a message takes

Real streaming means sending each piece of the reply to the phone as the model writes it. Every hop between the phone and the model has to pass those pieces along as they come in:

1. The RENEW app
2. RENEW's PHP backend, behind Apache
3. Sprout, the service that runs the coach
4. The university's U-M GPT gateway
5. The model

The gateway was holding the stream

Sprout's coach used the chat completions API, and asking it for a stream returned a stream, technically. Every model I tried (GPT-4.1, GPT-5.2 and two GPT-6 models) did the same thing: nothing for several seconds, then the entire reply in one burst. GPT-4.1 sent 1,800 characters in 0.56 seconds after a 5.2 second wait.

The Responses API streams for real through the same gateway. For the same ~300-word reply:

So the coach moved to the Responses API. It's the same model with the same prompts and settings, so the risk was in the plumbing, which tests cover well. The Responses API saves responses on the provider's side by default, which you don't want in a health app, so the coach's client turns that off:

COACH_CLIENT_PARAMS = {"use_responses_api": True, "store": False, "stream_usage": True}

Two things that broke along the way

The content filter

Without streaming, a reply the gateway blocks comes back as an error. When streamed, the gateway sends its canned refusal as if it were reply text:

I'm sorry, but I cannot assist with that request.

Then it ends the stream with a content_filter flag, which LangChain drops. So a filtered reply looks exactly like the coach refusing. Sprout holds streamed text back at first, checks replies a sentence at a time, and treats any stream that doesn't end with a completed status as filtered.

Apache and PHP buffering

Apache needs flushpackets turned on for the PHP proxy. PHP also buffers output in 4096-byte blocks, so the relay has to flush each piece itself or small chunks sit in the buffer until there's enough to send.

Most of the wait came before the model

Streaming only speeds up the part where the coach is writing. With real streaming and real retrieval, the first words still took about 7.9 seconds, because of everything that runs first:

Getting closer to ChatGPT

For comparison I timed ChatGPT with web search on. It showed a first part of the answer at about 2 seconds and the rest about 5 seconds later. That became the target, and three changes got close:

• Run in parallel. The library search starts at the same time as the classifier instead of waiting for it.
• Less thinking where it isn't needed. The classifier and reranker run with reasoning turned off, and the reranker looks at 10 passages. The routing and retrieval evals didn't change.
• Say something useful while searching. When a message needs a web search, the coach writes a first part while the search runs, then the rest once results are in.

That first part took a few tries. The first version mostly restated the question. I tried dropping it and showing a status line instead, and one test reply then took 22.6 seconds to appear because everything waited on the search. The version that stuck lets the first part use stable general knowledge, and holds contacts, links and anything clinical until the results and the person's context are there.

Results

Median time to first text: