I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
Pretty cool to hear about your process - I've been thinking about this space a lot so have some context about pain points here.
I've found that building a benchmark is something that's easy-ish on the surface level, but has many layers of technical complexity the deep you go down the rabbit hole.
For example, a tangent on grading. Do you use test passing and/or LLM graders? If you're using an LLM, it's fairly easy to spin up a basic LLM grader. But - is that grader calibrated? Is it biasing towards same-model agents? How many results do you need to be above the noise floor? What is it even measuring, and is that aligned with what you care about? Does it have the right information to make those judgement calls?
I'm super curious how what you've been using this workflow for? Also happy to answer any questions / discuss further!
Yes, I'm with you 100% on the setup. You can go as deep as you want down the hole. At some point I had ask myself what I was really after with the benchmarks and where the line of diminishing returns was (and more importantly, was I really getting generalizable results).
I'm currently using a 2 tier grading system, one pass mechanical (LLM graded) one pass human. The grading ranges from -1 to 0 for the mechanical pass and -1 to 1 for the human pass. A model that does everything competently passed with a score of 0. Non-pass gets a -1. For the mechanical pass that's it. For the human pass there's a chance to get an extra point for exceptional work. Accepting the subjectivity in the human pass was part of the "what am I doing this for" point from above.
The scores then don't get read left-to-right and summed, but rather vertically per task and category. The temp setting as another commenter noted means you can't say anything definitive about the quality from a single run. What I think you can tell is 1) per task, who is the outlier in either direction and investigate, 2) how a task is generally handled by the models and 3) in the aggregate, who is dropping and gaining points consistently and does that change per task type. In the end, it's high touch. It doesn't give you one number to cut or keep a model. It tells you where to look and where not to.
You're the second person running an evaluation business to express interest, maybe I should start one as well ;) I will be reading your blog posts to see if there's anything I can adopt. If I was to give you the objection I would pose if this was an MBA brainstorming sessions I would ask:
Who are you targeting? and if the answer is engineers, what's to stop them from using the LLM to do it themselves? The problem space exists in the easy/hard domain; easy to set up, hard to refine because of nuance and specifics. Moreover, an engineer whose job might be under pressure because of LLMs would be highly motivated and highly capable to create a testing and evaluation setup and "own" this part of the process since it can't be replaced by an LLM.
Personally, I think this space is good consulting territory. There is hard-won knowledge that is valuable and sale-able, but beyond the basic setup, everything has to be tailored. It's also going to be hard to get people into a product moat. Git and IDEs give customers (if they are engineers) all the tools they need for this work and they already know how to use them.
One last note of your comment about same model agents. This is definitely something I have noticed. It's probably structural, but models absolutely prefer their own outputs to others. You can also move them toward agreement or disagreement by telling them that their model (in another session) produced some output. Kimi in particular will dunk on other models and promote what it thinks is its own output. Kimi is also highly sensitive to being bench marked. I have a specific audit script that goes through thinking traces to see if the model realizes it's being tested. Kimi figures it out more than any other model and will dig big time to find other results if you don't completely isolate it. Claude gets pissy if it thinks you are trying to benchmark it. Fable has inserted several times "not under testing" and "not a competitor" when left to its own committing. Fable is also very good at setting up open weights models and testing them. They must have spent a lot of time going for thoroughness and scientific method in training.
Yes at temperature 1 it would need to be averaged. The mechanical scoring is hidden specs through and if they don’t pass that it’s not often a result of temp, but instruction following or verification (different flavor of instruction following given what they are told to do). What’s surprising is just how well they do on this dimension even with the variance. On my tasks anyway, maybe Rails work isn’t that complicated. It’s probably not that I am an amazing ticket writer.
I have run multiple with the same model through issues to see the output. What differs run to run on the same model is style, likelihood to find subtle bugs and chance to correctly guess author intent. I don’t score them on those things though, only note them.
To me, multiple vertical samples is more useful to the coder who isn’t benchmarking these as their job. The average is abstract to you if you’re not going to pointing them at a ticket more than once, but seeing their output over 10 different tickets will give you a window into model tendencies, strengths and weaknesses and help you understand how to write for them. Overall the exercise has made me a better issue writer.
I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation.
Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.