Faster AI with controlled resources: cache, batching and quantization

Series II: From infrastructure to agentic projects in production. Episode 19 of 20.

A slow AI response can keep the user waiting in front of a form, but a fast and incorrect response does not solve the problem. For an SME, the useful decision is to optimize the component that actually consumes time and resources. The proposal for this episode: test prefix reuse before changing the model, hardware and numeric precision all at once.

We continue the fictional B2B portal for requests and documents. After accessibility and content discovery, we analyze the agent integrated into the application, which prepares a summary for the operator. The agents developing the portal have a different role; they can implement the experiment and tests. The scenario below does not describe results measured at a client.

Where time is consumed

Inference is the use of an already trained model to produce a result. In a language model, input and output are processed in tokens, units that can represent words or word fragments. The duration of a request may include queue wait, input processing and continuation generation.

Input processing is called prefill, and generating new tokens is called decode. TTFT (Time To First Token) measures the time to the first token and does not represent the duration of the complete response. For a form that displays the result only at the end, the total latency is what matters. For text displayed progressively, the continuation rate also matters.

Before any tuning, check whether the delay appears in the model, in reading documents or in an external service. A model cache does not fix a slow database query. We keep the instrumentation discussed in the episode on observability and measure these components separately.

What the prefix cache reuses

A cache stores information for reuse. The prefix is the initial part of the input. In vLLM, an engine for serving models, Automatic Prefix Caching reuses intermediate KV states (short for key/value) for common prefixes. It does not automatically reuse the final response.

The vLLM guide, updated on October 13, 2026, defines the benefit: reuse shortens prefill, but does not directly speed up new token generation. If the inputs do not share a prefix or if long response generation dominates, the gain may be reduced or absent. Matching concerns tokens and the engine's block mechanism, not just similar ideas.

For the portal, we propose including stable common instructions at the beginning, followed by the specific request. We do not move documents between companies just to fabricate reuse. Changing common rules receives a new version so the experiment can be tracked.

Separation of data remains mandatory

In a portal used by multiple companies, reuse must be restricted according to access. The vLLM documentation offers cache_salt, an additional per-request value that allows block sharing only between requests with the same value. The risks of inferring information from latency differences are also discussed.

For our scenario, we propose the application should assign the trust group boundary server-side, not accept an arbitrary value from the visitor. We check isolation between companies and applicable retention rules. This measure supplements authorization and does not replace it. Confidential documents are not copied into experiment logs.

Batching: capacity and waiting

Batching means processing multiple requests in batches. Continuous batching allows the engine to update the group of active requests in real time. The ability to process more requests does not automatically mean every user gets the first result faster.

The vLLM optimization guide explains the influence of concurrency and KV memory on execution. Insufficient KV memory can lead to interruptions and recomputation. The PagedAttention paper, presented at SOSP 2023, documents the role of memory management in model serving. Its comparative results come from 2023 experiments and not from our portal.

In the portal, interactive requests have a response goal different from summaries that can wait in the background. We measure them separately. We do not increase concurrency just to achieve higher throughput while ignoring the wait time of other users.

Quantization is a separate experiment

Quantization means using reduced numeric precision for certain model values or intermediate states. It can reduce required memory, but the effect depends on the method, hardware and implementation. vLLM documentation publishes a compatibility matrix; a format available on one accelerator is not necessarily suitable on another.

For KV memory in FP8, an 8-bit floating point format, the documentation recommends calibration on a dataset, meaning the conversion parameters are estimated using examples. There are also limitations of execution components. Less memory alone does not prove better latency or acceptable quality.

For the portal we recheck identifiers, sums, calendar dates and omissions from the summary. We keep the initial configuration in order to revert if needed. We do not enable quantization in the same test as prefix caching since we would no longer be able to attribute the observed effects clearly.

What we control in cloud and on-prem

On-prem means infrastructure managed in-house. The team controls the engine, configuration and resources, but is also responsible for operations. A managed cloud service may expose other options and limits; we do not assume it allows vLLM tuning.

For example, Google Cloud documentation distinguishes the implicit cache from the explicitly created cache, with retention policies, supported models and storage costs for explicit cache. Check the actual service and contract. Saving on certain tokens does not automatically mean saving on the process as a whole, and an available function does not guarantee reuse for the company's traffic.

Single experiment worksheet

The practical result of this episode is this worksheet proposed for the portal:

1. Hypothesis: the common prefix can shorten the time to the first token and, if input processing contributes significantly, also the time for the full summary. We set in advance the relevant threshold for the operator and the stopping conditions.
2. Fixed configuration: we record the model, engine version, hardware, input template, output limit and generation settings. We compare the same input, with reuse disabled and enabled, with no other changes.
3. Representative traffic: we use anonymized or synthetic requests in Romanian, with short and long documents. We include repeated and varied prefixes, concurrent load and samples between separate companies.
4. Cold and warm cache: we distinguish starts with no available prefix from runs where it can be reused. We repeat and alternate conditions; we do not report only the most favorable execution.
5. Indicators: we record TTFT and full duration, including p50, median and p95, the value at or below which 95 percent of measurements fall. We also track queue time, reuse, memory, errors and quality using the same criteria.
6. Decision: we document the accepted configuration, limits, responsible party and rollback. If reuse does not help real traffic or affects isolation or quality, we stop the change.

The reuse rate is not deduced solely from shorter time. We check indicators provided by the engine and separately record reserved and used memory. If we change the model or conversation template, we repeat the baseline measurement. A few identical responses do not demonstrate system capacity at peak times.

This sheet turns a speed promise into a verifiable decision. The final episode will connect the optimization to the portal’s complete architecture, with responsibilities and acceptance criteria for the entire flow.

Sources

Sources consulted as of October 19, 2026. Documentation latest changes; the implemented configuration must be checked in the installed version. The portal example and experiment sheet are proposals, with no claimed results or savings.

• vLLM: Automatic Prefix Caching, updated October 13, 2026
• vLLM: prefix cache mechanism and isolation
• vLLM: optimization and tuning
• vLLM: quantization and hardware compatibility
• vLLM: KV memory quantization and calibration
• vLLM: production indicators
• Google Cloud: context cache presentation
• Kwon et al.: PagedAttention, SOSP 2023, paper submitted September 12, 2023