I was bemused by Adrian Cockrofts' Retort - it looks like Claude Code mostly independently running evals on coding agent x model x target language combinations. The most recent report however, matches my anecdata. I have started on something like Retort earlier this year, but much smaller - custom tiny agent, a couple of small language models and about 10 programming languages with a number of small tasks (extract method, find code smells in a file etc). I parked it, because it looked like that would be significant work to turn into something explainable to more people. So it is great to see Retort blossoming.
The main takeaway for me from the latest one is that the number of turns taken by the agent seems to be a good predictor for speed of task completion, and Qwen 35B, that I use a lot, does as well as the better frontier models on this task (and better than some of the worse frontier models). My hypothesis (hope?) that a decent harness with a smaller model would perform well on incremental tasks seems to pan out.
Reading the readme of retort, the factors are free to label and create. It produces a number of synthetic 'blogs'. The optimal blog I find interesting, bedcause it compares anthropic frontier models with one model I use regularly (Qwen3.6-35B-A3B) and one that I have used successfully for legacy code analysis (Qwen3-Coder-Next 80B ).
Not all outcomes match my experience, Qwen3.6-35B-A3B does well enough with Elixir production code, although it struggles with macros and moving functions across modules. That, admittedly, is something more LLMs struggle with. Explaining that often takes more time than doing it by hand.
It would be interesting to see the same benchmarks with Deepseek V4 Flash, widely used on somewhat larger machines (128GB VRAM, or less VRAM + more RAM combinations) and cheap on Openrouter (I haven't gone over a dollar yet for a day of intensive use), and Qwen 3.6 27B (much slower in tokens per second on my Macbook pro M3 Max, but seems to get more complex tasks done than 35B. I tried it yesterday on a Docker deploy that 35B struggled with, and it fixed the two issues I had independently. Feedback loops do work, as long as the model is up to the task ).
Reading through the optimal blog I found the source of my bemusement. Retort also does some BDD / TDD / ATDD and 'neutral' (no tests?) comparisons. I believe these are all Synthetic TDD. I have used coding agents (mainly Claude Code up to april) extensively to generate tests, recently with the help of Allium. Without tests coding agents go off the rails quickly. And I am lazy, as a good programmer should be, so I like having tests generated and providing feedback loops. But synthetic tests do have limitations. They carry an LLMs interpretation of your intent, and may not drive the design.
It may be that design doesn't matter everywhere equally. I am now working on some code where it does, and explaining myself in code (with some assistance) is faster and more precise than talking to an LLM.
I am not sure how that kind of intent could fit in a benchmark like Retort. What do you believe? Does intent and design in tests matter? If so, where and when?