A demo tested with a handful of queries rarely surfaces a latency problem. A production system under real load, with a large context window, a chain of retrieval and model calls, or an underprovisioned deployment, often does, and the gap between “worked fine in testing” and “feels unusably slow to customers” is one of the more common reasons a promising pilot stalls at scale.
Where the delay comes from varies: a larger context window takes longer to process, an orchestrated multi-step system accumulates the latency of every step in the chain, and API-based models add whatever queueing and network time sits between you and the provider. Each of those is a different lever, and the fix depends on identifying which one is actually the bottleneck.
Some use cases tolerate a few seconds without issue; a live voice conversation or a real-time trading decision does not. Knowing which category a use case falls into, before committing to an architecture, is a design decision, not something to discover after launch.