Ali Husain Software engineer and product designer

02 Intelligence/Evalbench

Evalbench

Self-hosted app · open source

Names the language model to use for your own job, and shows the proof.

Evalbench's verdict page: Use Orion Mini. It gets 22 of 24 right for $0.12 per 1,000 answers, with five models plotted against the bar.Evalbench's verdict page: Use Orion Mini. It gets 22 of 24 right for $0.12 per 1,000 answers, with five models plotted against the bar.
Blind judging on a phone: an answer scored on the job's rubric, with nothing that names the model.Blind judging on a phone: an answer scored on the job's rubric, with nothing that names the model.
The verdict on a phone: Use Orion Mini, and the bar it had to clear.The verdict on a phone: Use Orion Mini, and the bar it had to clear.

Write down what a good answer is, run the models you are considering, and Evalbench names the one to use: the cheapest model that clears your bar, with every number behind the pick on the page. Answers are graded by automatic checks, up to Python tests in a sandbox with no network, and people can rate pairs of answers blind. Every run is frozen, so a regression shows the day it lands.

291
automated tests, plus sandbox checks in CI
7
check types, up to sandboxed Python tests
20
tasks in the core battery, graded automatically
0
API keys stored; read from the environment

Why it exists

A leaderboard says which model is best at someone else's tasks. A few chats say which one sounds best. Neither says which model to put behind one specific job.

In the demo's support job, one model gets 22 of 24 examples right and another gets all 24 at 26 times the price. A score table does not say whether 22 is enough.

What it does

I write the job down as a suite: real examples, each with checks an answer must pass. Evalbench runs every candidate and names the cheapest model that clears the bar, the share of examples it must get right.

Under the name sits the evidence: each model's score and cost, the pick's misses, and blind human ratings. The same verdict can fail a build.

How it is built

  • A rule checkable by hand

    The pick is the cheapest model at or above the bar. I kept the rule small so every number it uses is on the page.

  • One rule, two languages

    The rule runs in Python on the server and in TypeScript in the static demo. Both run the same cases file.

  • Runs freeze what they measured

    A run stores the suite text and every model's settings at the start, so a later edit cannot change an old result.

  • Model-written code stays boxed in

    In the Docker setup, model-written code runs in a separate container with no route to the internet and cannot start another process.

How I know it works

291
Tests run in CI: 142 Python, 19 end-to-end, 98 web unit and 32 that click through the built demo.
33
Cases for the verdict rule, run by both the Python and TypeScript versions.
2,924
Checks run in the demo workspace: scripted answers, graded by the real engine, every model's score pinned by a test.
Sandbox, checked
On every change to the main branch, CI tries to reach the internet from the sandbox container and checks that the attempt fails, and that the app refuses requests from it.

What was hard

Three right out of three

The first verdict rule judged each model on its latest run, whatever its state. A run cancelled three answers in could leave a weak, free model at three right out of three: 100% at no cost, so it could become the pick.

Now a model counts in a run only once it has answered everything. Until then it keeps the last run it finished. A test rewrites a run into that half-done state and checks the pick does not move.

Limits

  • On a small suite the pick can be wrong: 22 right of 24 could truly be 74% to 98%.
  • The sandbox is a container, not a virtual machine: it is not for an attacker who targets container escapes.
  • It does not look prices up, so cost is only as right as the prices entered.

Also in Intelligence

Open to remote work.

ali@gallowforge.dev