Supermicro’s investor release dated November 5, 2026 describes a shared KV-cache solution with IBM and NVIDIA. Its “up to 56x” claim concerns time to first token at 130,000-token prompts. A separate throughput test in the linked IBM paper contains a particularly useful planning detail: the shared cache starts empty, and reuse builds during the run.

Conceptual illustration of an analyst comparing request groups beside abstract storage servers and a stopwatch; not an actual benchmark or dashboard.
Conceptual illustration of an analyst comparing request groups beside abstract storage servers and a stopwatch; not an actual benchmark or dashboard.

Read the cache condition beside the result

IBM reports that the throughput test sends 200 requests through 28 concurrent connections, choosing from 100 unique prompts. It explicitly includes the initial cold-start period. That is a different experiment from the paper’s single-request first-token comparison. Keep the request pattern, cache state and measured outcome attached to each result rather than merging their headline ratios.

NVIDIA defines time to first token as the interval from receiving a request to generating its first output token. Its glossary describes prefix caching as reusing computed KV state for common prompt prefixes. These definitions explain why a test’s reuse pattern matters: a result for repeated context does not establish the same result for a stream of unrelated inputs.

A fictional mixed-request calculation

Consider ten invented requests. Assume four cold requests each take 2 seconds to reach the first token, while six requests with reusable context each take 0.5 seconds. The total of those ten individual waiting times is 11 seconds: four times 2, plus six times 0.5. Their arithmetic mean is 1.1 seconds, not the 0.5-second value for the reusable-context group alone.

These are teaching inputs, not measurements of Supermicro, IBM, NVIDIA or a customer system. The sum is not a batch-completion time under concurrency. The example simply shows how selecting only the faster group changes the reported average.

Label the test sets

  • Cold or new context: record what was absent from the cache at the start.
  • Repeated context: document the shared prefix and observed reuse, rather than assuming a hit.
  • Mixed sequence: preserve request order and include the initial requests in the result.
  • For every set: retain input and output lengths, model, software versions, concurrency and failures.

Decide in advance which request mix resembles your intended application. A recurring assistant may revisit a shared document, while another workflow continually introduces new material. Run both relevant patterns on the same proposed configuration, then record the observed differences. Do not turn a laboratory maximum into a forecast for every user request.

Keep the wider outcome visible

Record the completed-response time and output checks alongside first-token latency. Review storage and network costs with the proposed hardware configuration before estimating operating savings. Faster access to reusable state answers one performance question; the buying decision still needs your workload’s results and actual quotation.

If you are also comparing inference hardware, the Google TPU 8tx and 8i Max guide separates security requirements from memory capacity and bandwidth bottlenecks.

Check the cache-test comparison

Primary sources

Supermicro investor release dated November 5, 2026

IBM technical paper: separate benchmarks and initial cache state

NVIDIA Dynamo glossary: first-token latency and prefix caching