Skip to content
Gradient Notes logoGradient Notes
Evaluation

Evals before vibes: a practical starting point

You cannot improve what you only feel. A lightweight evaluation loop you can set up in an afternoon, and why it pays for itself in the first week.

Maya Okafor3 min read

Laptop on a desk showing code in a dark editor theme
Negative Space · CC0

Every team that ships an LLM feature goes through the same phase. Someone tweaks a prompt, tries it on three examples they remember, and declares it better. A week later a customer finds the case it broke. The problem is not the prompt; it is that the team has no shared definition of "better".

An evaluation set is that definition. It does not need to be big or clever. It needs to exist, be versioned, and be cheap enough to run on every change.

Start with fifty real examples

Pull fifty inputs from production logs (or from your own dogfooding if you have not launched). Pick them on purpose: a third should be the boring, common case; a third should be the cases you already know are hard; the last third should be weird — empty inputs, very long inputs, different languages, prompt-injection attempts.

For each one, write down what a good answer looks like. Not the exact text, but the properties: mentions the refund window, does not invent an order number, answers in Spanish. Those properties become your checks.

Use the cheapest grader that works

There are three kinds of checks, and you should reach for them in this order:

  1. Deterministic checks. Does the output parse as JSON? Does it contain the required field? Is it under 400 characters? These are free, fast and never flaky.
  2. Reference comparisons. For extraction and classification tasks, compare against a labelled answer with exact match or a simple similarity score.
  3. Model-graded checks. For open-ended answers, ask a second model a narrow yes/no question with a rubric. Keep the question specific: "Does the answer state the refund window as 30 days?" beats "Is this answer good?"

Make it run on every pull request

An eval that lives in a notebook gets run twice. Put it in CI. A small script that loads the cases, calls your pipeline, runs the checks and prints a table is enough. Fail the build when a previously passing case starts failing — regressions matter more than the headline score.

evals/run.pypython
import json, sys
from pipeline import answer
from checks import CHECKS

cases = [json.loads(l) for l in open("evals/cases.jsonl")]
failures = []
for case in cases:
    output = answer(case["input"])
    for name in case["checks"]:
        ok = CHECKS[name](output, case)
        if not ok:
            failures.append((case["id"], name))

print(f"{len(cases) - len({f[0] for f in failures})}/{len(cases)} cases passing")
for case_id, check in failures:
    print(f"  FAIL {case_id}: {check}")
sys.exit(1 if failures else 0)

Grow the set from incidents

Every time a bug reaches a user, add the input to the eval set before you fix it. Within a couple of months your fifty cases become three hundred, and they are the three hundred that matter for your product, which no public benchmark will ever tell you.

What changes on the team

The biggest benefit is social, not technical. Prompt changes stop being arguments about taste. Someone proposes a change, CI posts the before-and-after table, and the conversation moves to the two cases that got worse. That is a much better conversation to have.

“The eval set is the spec. Everything else is an implementation detail.”
— Something we now say in every design review

Keep reading