On November 6, 2026, Toptal announced a collaboration between its ExpertCrowd platform and HUD and a release called ExpertCrowd-SWE. The accompanying results explain how the team combined repeated attempts into scores. If you are choosing a coding agent for a team, look at the tasks that resemble your own work and at how consistently the agent completes them.
Two agents can tie while behaving differently
In a fictional test, one agent earns 60 points on each of five attempts. A second earns 100, 100, 100, 0 and 0. Both average 60. If your workflow needs a predictable partial result, the first pattern may fit better. If it needs a complete fix and can cheaply verify and retry a failure, the second pattern may be useful. The numbers are illustrative and do not describe any model in the benchmark.
Read the scoring rules before borrowing the ranking
Toptal says its task scores average five valid runs. Its results page says runs that hit the step or output-token limit were excluded from scoring. That makes the definition of a valid run worth reading: in your own pilot, a time or budget limit may be part of the result your team cares about.
- Pick representative tasks from your own workflow, such as repairing a failing test or changing a small feature without breaking an existing one.
- Decide in advance what counts as complete, partial or failed. Record timeouts and incomplete attempts separately.
- Repeat the same tasks under comparable limits. Inspect the distribution of results and the failure cases before combining them into an average.
Use the published benchmark to decide what to test. Use your own pilot to decide how much review, time and retry capacity the tool would require.