"Streaming, caching and other ways to make AI feel fast"

There is a version of your AI feature that feels twice as fast and costs nothing extra to run. The total processing time is identical; the difference is what the user sees while it happens. Most latency work on LLM features is not about making the model faster, you mostly cannot, it is about designing the wait. That is good news, because designing the wait is cheap.

Start with streaming, and be clear about what it is: a UX decision, not a performance optimisation. Streaming does not reduce total generation time by a single millisecond. What it changes is time to first word, which is the number users actually feel. Adults read silently at about 240 words a minute on average and models generate considerably faster than that, so once the first words arrive the user never catches up with the output. From their point of view the answer was ready the moment it started. A nine-second response that begins streaming at one second reads as fast; the same response delivered whole at nine seconds reads as broken.

Streaming has a real tension with structured output, and it is worth being honest about it. If your pipeline validates the model's JSON against a Zod or Pydantic schema before acting on it, and it should, you cannot validate a stream that has not finished. The patterns that work: stream prose to the user while generating the structured payload separately, or generate the validated payload first and stream a human-readable rendering of it. What you must never do is act on partial, unvalidated output because the stream made it available early. Show early; commit late.

Where you cannot stream, show progress that means something. A spinner communicates nothing except that the system has not crashed. A line that says "searching 34 documents, reading the 6 most relevant" changes the experience twice over: the user understands why they are waiting, and when the answer is wrong they can often see where it went wrong. Meaningful progress is also the cheapest observability you will ever ship, because users report failures with context instead of "it's stuck".

Optimistic interface work helps too. Acknowledge the click instantly, render the shell of the answer before the content exists, echo the user's request back into the empty structure. None of this shortens anything, but the perceived gap between action and response is what drives abandonment, and that gap closes the moment the interface visibly reacts.

The strongest technique is not waiting at all: precompute. The fastest LLM call is the one you already made. If users routinely ask for summaries of documents, generate the summary when the document arrives, not when someone asks. Classifications, embeddings, extracted metadata, all of it can move from query time to ingestion time, often overnight on a batch API at half price, the flat 50 per cent discount every major provider now offers for asynchronous work. You will spend compute on items nobody asks about, but at batch rates that is usually a bargain against a user staring at a spinner.

Response caching is narrower than people hope. Exact-match caching works well in programmatic pipelines where identical requests genuinely recur, and it works for the static parts of prompts. Semantic caching, returning a cached answer because the new question looks similar, is riskier than it sounds, because two similar questions can require different answers. Use it only where a plausible-but-slightly-off answer is cheap to tolerate.

Look at the pipeline shape as well. Most LLM pipelines run sequentially because sequential code is easier to write, not because the steps depend on each other. Retrieval and classification can often run concurrently; two independent extractions do not need to queue. I have cut end-to-end latency by 40 per cent on a pipeline without touching a single prompt, purely by drawing the dependency graph and parallelising what was independent.

One caution to end on: all of this masks latency rather than removing it, so keep measuring the real number underneath. If total processing time is quietly growing month on month, streaming will hide the decay right up until it cannot. Perceived speed buys you goodwill; it does not repeal the budget.

Designing AI features so the wait is invisible, and knowing when the honest answer is to make the job asynchronous instead, is part of the work I do for clients. If your AI feature works but feels slow, get in touch via the contact form on this site.