Both runs are finished. Fabric passed 21 of 30 tasks, 70.0%, with GLM 5.3 Flash, and got exactly the same score with Kimi K3.
Runta's FrontierHarness ran twelve coding-agent harnesses through the same 30 tasks, all on one model, Kimi K3, and published the cost and pass/fail result of every run. I added Fabric to their chart.
Cost runs from expensive on the left to cheap on the right, so up and to the right is better. The lower panel zooms in on the ten harnesses bunched between $2.17 and $5.32. Tap the chart for the full-size image.
The numbers
| Harness | Pass rate | Passed | Cost per pass |
|---|---|---|---|
| Fabric · GLM 5.3 Flash | 70.0% | 21 / 30 | $0.37 |
| Fabric · Kimi K3 | 70.0% | 21 / 30 | $2.81 |
| Codex | 66.7% | 20 / 30 | $3.47 |
| DSH Creator | 63.3% | 19 / 30 | $3.28 |
| Claude Code | 63.3% | 19 / 30 | $18.34 |
| Pi | 60.0% | 18 / 30 | $2.43 |
| DSH PTC | 60.0% | 18 / 30 | $4.58 |
| DSH Standard | 60.0% | 18 / 30 | $3.46 |
| Oh My Pi | 56.7% | 17 / 30 | $4.75 |
| Kimi Code | 56.7% | 17 / 30 | $3.65 |
| DSH Minimal | 56.7% | 17 / 30 | $4.72 |
| Exo Harness | 53.3% | 16 / 30 | $1.05 |
| OpenCode | 50.0% | 15 / 30 | $3.24 |
| Hermes | 50.0% | 15 / 30 | $2.90 |
Only the official binary task reward counts as a pass. Cost per pass is total model cost divided by tasks passed, and for Fabric the cost covers only the graded attempt on each task.
How to read it
The published harnesses all ran Kimi K3 on the same 30 tasks: 21 from terminal-bench and 9 real-repository tasks from datacurve. Their costs come straight from the published data. FrontierHarness labels its x-axis "median cost per task", but the coordinates in its data equal total cost ÷ passes, so that's what I call it.
I ran Fabric twice. The first run used GLM 5.3 Flash as the main model, with DeepSeek V4 Flash on standby as a helper. The second used Kimi K3, with DeepSeek V4.1 Flash as the helper. I priced both at retail OpenRouter prices as of 23 September 2026, charging input, cached input and output tokens at their own list rates. Per million tokens, GLM 5.3 Flash lists at $0.15 input, $0.05 cached and $0.50 output; Kimi K3 at $3.00, $0.30 and $15.00; DeepSeek V4.1 Flash at $0.10, $0.01 and $0.50.
Different providers on OpenRouter charge different prices for the same model, so I used the price OpenRouter lists for the model itself.
Some attempts never got graded. The provider hit a usage limit, the task environment didn't come up, or a Fabric bug broke the run and I fixed it and ran the task again. Those attempts don't count toward the score, so I don't count their tokens either, and each task is priced on its graded attempt alone. On that basis the GLM run cost $7.87 for all 30 tasks and the Kimi run $58.98. Counting every attempt, graded or not, they cost $9.48 and $114.29.
A metering proxy records which model handled each request. Twenty of the GLM tasks went through Fabric's own request gateway instead, and it doesn't log a model id, so I priced those tokens at GLM's rate, the pricier of that run's two models. Four of the twenty left no token record at all and count as zero. The GLM figure is therefore a floor, and the real gap between the two runs could be somewhat smaller than the chart shows.
Caveats
The Kimi K3 run is the like-for-like comparison, with the same model, tasks and graders as every published harness. The GLM run used a different, much cheaper model, so part of its lead on the x-axis comes from the model's price rather than from Fabric.
With GLM 5.3 Flash, Fabric failed nine of the 30 tasks: arktype-json-schema-refs-dependencies, build-cython-ext, expr-try-catch-errors, kv-store-grpc, largest-eigenval, meriyah-explicit-resource-declarations, python-statemachine-state-data-scoping, sanitize-git-repo and scc-bounded-memory-spilling.
With Kimi K3 it also failed nine, not all the same ones: anko-typed-variable-bindings, arktype-json-schema-refs-dependencies, build-cython-ext, expr-try-catch-errors, gcode-to-text, katex-multicolumn-array-spans, kv-store-grpc, largest-eigenval and meriyah-explicit-resource-declarations.
Fabric ran in its own desktop app instead of the FrontierHarness runtime, though every grade comes from the official FrontierHarness graders. I did this comparison myself, and it isn't a leaderboard entry.
Kimi K3
Since every published harness used Kimi K3, this is the run I cared about most. Fabric passed 21 of 30 with it, 70.0%, one task ahead of Codex, the best published harness, and tied with Fabric's own GLM 5.3 Flash run.
On cost the runs are far apart. Kimi K3 came to $2.81 per passed task. GLM 5.3 Flash, the smaller and cheaper model, got the same score for $0.37.
The Kimi run also turned up two Fabric bugs. Kimi sometimes writes raw chat-template tokens into its tool calls, and Fabric's parser was dropping those calls. Separately, some commands ran fine but reached the model as empty output. I fixed both and reran the tasks they'd hit, and one of them, dna-insert, went from a fail to a pass. A provider hiccup broke a few more attempts, and I reran those tasks too.
Sources
All published results come from FrontierHarness by Runta. Because they published the cost and grade of every run, I could make this comparison at all, so thanks to them.
- Original chart and launch post: Introducing FrontierHarness Eval (Runta)
- Published data (eval-data.json)
- Source data and task definitions (GitHub)
GLM 5.3 Flash numbers as of 23 September 2026, Kimi K3 as of 24 September 2026.
Comments