How much quality do you trade for lower cost?
We tested phantom-1 against fixed-model strategies on 652 tasks: 295 everyday tasks and 357 difficult benchmark problems. These are results from October 5, 2026, before subsequent routing and model changes.
The tradeoffs, at a glance.
Everyday tasks · 295 questions
phantom-1 passed 91% for $2.438 in total test-set cost. Always using the frontier model passed 99% for $3.990. The router used 61% of the frontier strategy’s spending, but passed fewer tasks.
Hard benchmarks · 357 questions
phantom-1 passed 55% for $0.299. Always using the frontier model passed 86% for $6.790. Always using the free model passed 58%. The October 5 router was inexpensive on this set, but often chose models that were too weak.
What these results tell us.
For everyday work: The historical router handled most tasks while reducing spending relative to always using the frontier model. It did not match frontier accuracy.
For difficult work: Cost savings came with a substantial loss in pass rate. Some programming and math questions needed a more capable model than the router selected.
What changed afterward: We updated routing rules and replaced the free and standard models. The measurements on this page are from before those updates; we have not represented them as a new end-to-end benchmark. What’s changed
How we tested the router
We tested each of 652 tasks against the three models available in the benchmark. Each model’s answer was evaluated against the task’s scoring criteria. We then replayed different routing strategies over those observed results to compare accuracy and cost.
Test sets
Everyday (295 tasks): Formatting, small functions, bug fixing, algorithms, concurrency, math and long context.
Hard (357 tasks): BigCodeBench-Hard, LiveCodeBench, AIME and MMLU-Pro.
Execution and scoring
One sample per task and model at the configured default temperature, with the task as the sole message. Requests ran with prompt caching and the cheapest-provider preference enabled. Code was evaluated in a restricted sandbox; other answers were checked against task-specific expected formats and references. Failed calls that remained unsuccessful after retries counted as failures.
Cost calculation
Costs reflect what a Phantom caller would pay for the sampled tokens at the benchmark’s provider rates, including Phantom’s stated markup. These are retrospective test-set totals.
Historical model lineup
Free: Ling 3.1 Flash Free. Standard: DeepSeek V4 Flash. Frontier: Claude Sonnet 5.5. These were the models used in the October 5 measurements, not necessarily today’s lineup.
The tables come from the benchmark runs saved with Phantom’s code, read when this page is built.
Full methodology
- One sample per task and model, at the default temperature, with the task as the only message.
- Calls ran with phantom-1’s settings: prompt caching on, cheapest provider first. A call that still failed after retries counts as a fail.
- Code ran in a sandbox with no network and limits on time and memory, against the task’s tests: unit tests, or input and output cases for contest problems. Other answers had to match exactly: a number, a JSON value, a word or a multiple-choice letter.
- We kept a task only when its reference answer passed its own check.
- Costs are what a Phantom caller would pay for the same tokens: the upstream’s price plus Phantom’s markup.
- The results folder also holds three smaller runs, two smoke tests and a partial hard run. These tables don’t count them.
| Set | Tasks | What it holds |
|---|---|---|
| Everyday | 295 | Formatting, small functions, bug fixing, algorithms, concurrency, math and long context |
| Hard | 357 | BigCodeBench-Hard, LiveCodeBench, AIME and MMLU-Pro |
| Tier | Model in these runs | Everyday | Hard |
|---|---|---|---|
| Free | Ling 3.1 Flash Free inclusionai/ling-3.1-flash-free | 266/295 | 208/357 |
| Standard | DeepSeek V4 Flash deepseek/deepseek-v4-flash-0731 | 258/295 | 199/357 |
| Frontier | Claude Sonnet 5.5 anthropic/claude-sonnet-5.5 | 292/295 | 307/357 |
Everyday tasks: 295 requests
From formatting to coding and long-context tasks, the October 5 router passed 269 of 295 requests. Always using the frontier model passed 292 of 295. See the different costs and routing choices below.
How to read these results
- Pass rate:
- Tasks passed divided by tasks evaluated. The 95% interval is a Wilson confidence interval for the observed pass fraction.
- Total cost:
- Aggregate modeled customer charge for the test set.
- Cost vs. frontier:
- Total cost divided by the always-frontier baseline on the same task set (the Frontier cost % column).
- Free / standard / frontier picks:
- Number of tasks routed to each tier.
- Routed to a model that failed:
- The selected model failed a task that a stronger model passed (Misrouted down in the original data).
- Used a pricier model than necessary:
- A cheaper tested model passed the same task (Overpaid in the original data). This is a retrospective opportunity-cost label, not a billing error.
- Best-possible pick:
- An oracle comparison that picks the cheapest passing model with hindsight. It cannot be used directly for a live request without knowing the correct outcome first.
Scroll sideways to see every column.
| Strategy | Pass rate | 95% interval | Total cost | Frontier cost % | Picks | Routed to a model that failed | Used a pricier model than necessary | ||
|---|---|---|---|---|---|---|---|---|---|
| Free | Standard | Frontier | |||||||
Always free Ling 3.1 Flash Free | 90%266/295 | 86% to 93% | $0 | 0% | 295 | 0 | 0 | 26 | 0 |
Always standard DeepSeek V4 Flash | 87%258/295 | 83% to 91% | $0.107 | 3% | 0 | 295 | 0 | 34 | 266 |
Always frontier Claude Sonnet 5.5 | 99%292/295 | 97% to 100% | $3.990 | 100% | 0 | 0 | 295 | 0 | 278 |
Rules only phantom-1’s rules, replayed offline | 91%269/295 | 87% to 94% | $2.435 | 61% | 45 | 229 | 21 | 23 | 223 |
phantom-1 as deployed What production ran that day | 91%269/295 | 87% to 94% | $2.438 | 61% | 45 | 228 | 22 | 23 | 223 |
Best possible pick Oracle / hindsight: the cheapest model that passed | 99%292/295 | 97% to 100% | $0.235 | 6% | 269 | 12 | 14 | 0 | 0 |
Detailed findings
The widest gap was in algorithms: 28 of 45 against 44.
Most of the router’s cost went to long documents: phantom-1 paid $2.344 of its $2.438 on the 25 long-context tasks, and every model passed all 25.
The free model alone passed 90% at no cost, more than the standard model’s 87%. It was also the slowest: a typical answer took 6.5s, and one in twenty took longer than 74.3s, against 2.0s and 8.6s for frontier.
The best possible pick, a hindsight bound that picks the cheapest model that passed each task, passed 99% for $0.235, 6% of always-frontier’s cost. No live router can reach it without knowing the answers first.
Difficult benchmarks: 357 tasks
The hard set includes BigCodeBench-Hard, LiveCodeBench, AIME and MMLU-Pro. On the October 5 run, phantom-1 passed 198 of 357 tasks (55%). Always-frontier passed 307 (86%), and always-free passed 208 (58%). This was the clearest weakness in the historical routing rules.
Scroll sideways to see every column.
| Strategy | Pass rate | 95% interval | Total cost | Frontier cost % | Picks | Routed to a model that failed | Used a pricier model than necessary | ||
|---|---|---|---|---|---|---|---|---|---|
| Free | Standard | Frontier | |||||||
Always free Ling 3.1 Flash Free | 58%208/357 | 53% to 63% | $0 | 0% | 357 | 0 | 0 | 109 | 0 |
Always standard DeepSeek V4 Flash | 56%199/357 | 51% to 61% | $0.428 | 6% | 0 | 357 | 0 | 114 | 208 |
Always frontier Claude Sonnet 5.5 | 86%307/357 | 82% to 89% | $6.790 | 100% | 0 | 0 | 357 | 0 | 255 |
Rules only phantom-1’s rules, replayed offline | 55%198/357 | 50% to 61% | $0.299 | 4% | 93 | 261 | 3 | 115 | 172 |
phantom-1 as deployed What production ran that day | 55%198/357 | 50% to 61% | $0.299 | 4% | 93 | 261 | 3 | 115 | 172 |
Free unless rules say frontier Free, plus the rules’ frontier picks | 58%208/357 | 53% to 63% | $0.013 | 0% | 354 | 0 | 3 | 109 | 2 |
Free, then frontier Answer-key best case: frontier retries each task free failed | 88%313/357 | 84% to 91% | $4.855 | 72% | 208 | 0 | 149 | 0 | 47 |
Best possible pick Oracle / hindsight: the cheapest model that passed | 89%317/357 | 85% to 92% | $2.259 | 33% | 248 | 47 | 62 | 0 | 0 |
Weak-model picks
The historical router sent only 3 of the 357 hard tasks to the frontier tier, although 62 tasks passed only with the frontier model.
Free-model reliability
The free model’s final attempt failed without a usable answer on 79 tasks: 50 provider-unavailable responses and 29 timeouts. These counted as benchmark failures.
Why the oracle matters
The hindsight-only best-possible pick passed 89% at 33% of frontier spending. This illustrates the opportunity for better routing, not current achieved performance.
Detailed findings
phantom-1 as deployed passed 55% (50% to 61%). Its rules sent 81 of the 125 LiveCodeBench problems to the free model and kept all 60 AIME problems on standard.
On the 278 tasks every model answered, always-free passed 75%, phantom-1 63% and always-frontier 86%.
Running free first, and frontier only when free’s answer failed its check, passed 88% for 72% of frontier’s cost. That needs an answer key to check against, and a real request has none.
Where routing worked, and where it didn’t
Break down pass rates and costs by task type to see which requests the router handled well and which needed stronger models. These results use the same October 5 benchmark artifacts as the summary above. To compare the three models on their own, see free, standard and frontier by category.
Everyday tasks
Scroll sideways to see every column.
| Category | Always frontier | phantom-1 as deployed | Best possible pick (hindsight) | |||
|---|---|---|---|---|---|---|
| Passed | Cost | Passed | Cost | Passed | Cost | |
Formatting Generated extraction and reshaping jobs | 40/40 | $0.082 | 40/40 | $0 | 40/40 | $0 |
Small functions HumanEval and MBPP | 68/70 | $0.194 | 67/70 | $0.015 | 68/70 | $0 |
Bug fixing HumanEvalFix, and bugs written for the benchmark | 40/40 | $0.125 | 37/40 | $0.00687 | 40/40 | $0.00015 |
Algorithms APPS interview and competition problems | 44/45 | $0.556 | 28/45 | $0.028 | 44/45 | $0.205 |
Concurrency Questions written for the benchmark | 35/35 | $0.092 | 34/35 | $0.024 | 35/35 | $0.011 |
Math GSM8K, and MATH at level 5 | 40/40 | $0.121 | 38/40 | $0.020 | 40/40 | $0.019 |
Long context Generated Python modules and standard library files | 25/25 | $2.819 | 25/25 | $2.344 | 25/25 | $0 |
| All tasks | 292/295 | $3.990 | 269/295 | $2.438 | 292/295 | $0.235 |
Difficult benchmarks
Scroll sideways to see every column.
| Benchmark | Always frontier | phantom-1 as deployed | Best possible pick (hindsight) | |||
|---|---|---|---|---|---|---|
| Passed | Cost | Passed | Cost | Passed | Cost | |
BigCodeBench-Hard Python functions that call libraries | 36/60 | $0.459 | 21/60 | $0.031 | 38/60 | $0.131 |
LiveCodeBench AtCoder and LeetCode contest problems | 113/125 | $4.596 | 54/125 | $0.041 | 116/125 | $1.829 |
AIME The 2025 and 2026 exams | 59/60 | $1.248 | 31/60 | $0.191 | 59/60 | $0.254 |
MMLU-Pro Multiple choice across 14 subjects | 99/112 | $0.487 | 92/112 | $0.036 | 104/112 | $0.044 |
| All tasks | 307/357 | $6.790 | 198/357 | $0.299 | 317/357 | $2.259 |
What’s changed since October 5
These benchmark results are a historical snapshot. The routing rules and model lineup changed afterward, so the numbers above should not be read as a new benchmark of today’s phantom-1.
Routing changes
- Prompts longer than 20,000 tokens are routed to the standard tier instead of frontier.
- Contest-programming and olympiad-math tasks are directed to frontier.
- Code-writing requests are no longer classified as routine formatting merely because they mention JSON or extraction.
Model changes
- DeepSeek V4.1 Flash replaced DeepSeek V4 Flash as the standard model.
- Nemotron 3.5 Lightning Free replaced Ling 3.1 Flash Free as the economy model.
A separate comparison in the benchmark notes shows DeepSeek V4.1 Flash passing 273/295 everyday tasks versus 258/295 for DeepSeek V4 Flash, at reported costs of $0.353 and $0.107. This is not a rerun of the complete updated phantom-1 pipeline, so the 95% it suggests for phantom-1 is an estimate from replaying the same picks, not an observed end-to-end score.
Want to see what Phantom would choose?
Inspect the routing rules, test a prompt without calling a model, or use phantom-1 from your existing client.