Why the cost goal comes first

AWS introduced the SageMaker JumpStart Model Leaderboard on November 18, 2026. Its optimization goals select different deployment configurations: Balanced selects the lowest estimated cost per token, while Cost optimized selects the lowest hourly instance price. The use-case profile is a separate setting; changing it updates performance and cost metrics, not the quality benchmark scores. Estimated instance cost uses the default instance type's on-demand hourly rate and excludes transfer, storage, inference component costs and discounts. Treat that figure as a limited comparison input, not a complete bill.

Keep the comparison setup fixed

Before starting a pilot, write down the reader-facing outcome you need: for example, a useful ticket reply within your team's acceptable wait. Record the model and version, use-case profile, optimization goal, instance type and metric units. Keep your prompt set and settings fixed across the two candidates. If you change an optimization goal, put the new comparison in a separate row instead of silently replacing the first result. This makes a changed assumption visible to the next reviewer.

Suggested two-model record, to fill after testing

  • Model name and version
  • Use-case profile
  • Optimization goal
  • Instance type and runtime settings
  • Recorded metric and units
  • Date actually tested
  • Input and output lengths
  • Concurrency
  • Own answer-quality pass criteria

A simple helpdesk pilot example

A hypothetical helpdesk team might find that the configuration with the lower hourly price takes too long to produce a usable reply. Another configuration might finish sooner but fail the team's answer-quality rubric. Neither observation should be hidden behind a single cheapest label. In a proposed pilot, the team would assess both questions separately, then document the trade-off. This is an illustrative decision process, not a test performed here; no model, price or performance result is asserted.

Use the leaderboard as a record, not a shortcut

The record above is a proposed template, not a populated benchmark. Leave untested fields blank. Record the date and the conditions when an authorized pilot is actually run, and keep the original observations so another reviewer can reconstruct the comparison. A clear record also makes disagreement useful: someone can challenge the chosen workload or acceptance rule without confusing that disagreement with a measurement error.

AWS: Introducing the SageMaker JumpStart Model Leaderboard

Illustration of two blank comparison tables beside a laptop, coffee cup and plant on a wooden desk.
Illustration of two blank comparison tables beside a laptop, coffee cup and plant on a wooden desk.