Python, Rust and Go
36 Exercism tasks, 12 per language. Tests stay hidden until the agent finishes. Each harness ran every task twice per model: 72 runs. Sage 0.3.0 against OpenCode 1.19.5, on 17 November 2026.
Solved, out of 72
Total cost, 72 runs
Time per run
| Model | Harness | Solved | Python | Rust | Go | Cost | Time per run |
|---|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash | Sage | 44/72 | 13/24 | 17/24 | 14/24 | $2.73 | 3.8 min |
| OpenCode | 44/72 | 13/24 | 17/24 | 14/24 | $4.51 | 6.2 min | |
| GLM-5.3 | Sage | 31/72 | 8/24 | 15/24 | 8/24 | $8.86 | 4.6 min |
| OpenCode | 19/72 | 3/24 | 9/24 | 7/24 | $21.65 | 11.5 min |
- Same effort. Both harnesses asked the model for
lowreasoning effort. Sage raises it only after three surprises in a row. - Why OpenCode was slow on GLM-5.3. Each request wrote five to six times as much and took 30 s against 17 s. 18 runs hit the 20-minute or 60-request limit, against one of Sage’s.
- Isolated. Each harness ran under its own account, with no network except the model gateway and no way to read the hidden tests. OpenCode tried to download the official tests in most runs; every attempt was blocked.
- Reruns. Runs that ended on a provider error or a fault of our own machine were run again from scratch.
- Checked for leaks. On DeepSeek, three runs reproduced the hidden tests or solution almost in full, most likely from memory; without them Sage is 44/71 and OpenCode 42/70. On GLM-5.3, no solved run did.
Earlier: 24 project tasks
Bug fixes, small modules, skill tasks (security audits, dead-code cleanup, tests, performance, debugging, UI) and multi-file tasks with strict specs. Three runs per task on DeepSeek V4.1 Flash: 72 runs, on 12 November 2026, with an earlier Sage.
Solved, out of 72
Total cost, 72 runs
Time per run
Sage used 5.7 requests per run against OpenCode’s 10.8. Three changes measured that week, each on its own, are still how Sage works:
- Start with less reasoning
- Same accuracy in half the time.
- Allow whole-file rewrites
- Refused writes fell from more than ten per 48 runs to a few.
- Stick to the task’s own tests
- The test watcher no longer follows throwaway scripts, removing 14 to 40 false switches per 48 runs.
Limits and fairness
Gaps of two or three solved runs are within normal model variance. A miss that repeats on one task is the real signal. The results are from two models, DeepSeek V4.1 Flash and GLM-5.3, both through one gateway; other models may rank the harnesses differently.
Times and costs vary through the day, so compare only runs made side by side.
On your own code, results can differ from other harnesses in either direction: a benchmark is a sample, and your work is the test that counts.
The method, raw logs and per-run results for both tests are in the Sage repository, in docs/BENCHMARKS.md.
Try it on your own work
The fairest comparison is a task you already have. Download Sage, add a key for the model you use, and give it something real in a repository you can reset. Keep the effort on Auto: it starts low and rises by itself when the task needs it, which is where Sage saves the most.