Write the eval before you pick the framework
Teams spend weeks choosing between LangGraph and CrewAI and zero days deciding how they'll know if it works. The framework question is downstream of a measurement question nobody asked.
A pattern worth naming: a team spends three weeks evaluating agent frameworks, picks one, builds for two months, and then cannot answer whether the thing is getting better or worse. They have opinions. They have vibes from a demo. They have no measurement.
The framework choice was never the important decision. It felt important because it was legible — a comparison table, a set of tradeoffs, a defensible conclusion. Measurement is illegible until you build it, so it loses.
Why this ordering is backwards
Framework choice is reversible. Migrating from CrewAI to LangGraph is a week of unpleasant work, and you'll have learned enough by then to make a better choice anyway.
The absence of an eval is not reversible in the same way. Every day without one, you accumulate changes whose effect you cannot attribute. Six weeks in, quality has drifted and there is no way to bisect it — you have a hundred commits, each of which might be the culprit, and no way to test any of them against a past state.
An eval is a ratchet. It prevents backslide. A framework is a convenience.
What a first eval actually looks like
The thing that stops people is imagining they need a sophisticated harness. They don't. Version one is a JSONL file:
{"input": "reset my password", "expect_tool": "send_reset_email"}
{"input": "cancel my plan", "expect_tool": "escalate_to_human"}
{"input": "why was I charged twice", "expect_tool": "fetch_billing_history"}
Twenty of these, hand-written in an afternoon from real user messages, plus a script that runs them and prints a pass rate. That's it. That's the eval.
It is crude, and it will catch the majority of regressions you'd otherwise ship, because most regressions are not subtle quality drift — they're the agent suddenly calling the wrong tool because someone reworded a prompt.
Three levels, in order
- Deterministic checks. Did it call the right tool? Is the JSON valid? Did it stay under the token budget? No model needed, runs in seconds, catches most breakage.
- Reference-based scoring. For tasks with a known answer, compare against it. Exact match where possible, embedding similarity where not.
- Model-graded rubrics. For genuinely open-ended output, have a model score against explicit criteria. Useful, noisy, and the last thing to build — not the first.
Most teams try to start at level three because it feels most sophisticated. It's the least reliable and the most expensive. Start at one.
If a change makes your eval pass rate go up, ship it. If you can't tell, you don't have an eval — you have a demo.
The honest objection
"Our task is too open-ended to evaluate."
Sometimes true. Usually it means the task hasn't been decomposed enough. An agent that "helps with research" can't be scored. But did it find the three sources a human expert would have found — that's scoreable, and it's most of what you actually care about.
If you genuinely cannot define success, that's not an evaluation problem. That's a product problem wearing an evaluation costume, and no framework will fix it.
The reordering
Week one: write twenty test cases and a script that runs them. Week two: build the simplest thing that passes some of them — a plain loop, no framework. Week three, when the loop is straining, then reach for orchestration, and you'll know which features you need because you'll have hit the walls yourself.
You'll pick a better framework in an hour with that experience than in three weeks of comparison tables without it.