How many evals are enough to launch?

Around 80-120 hand-labelled cases covering happy paths, ambiguous inputs and known failure modes catches most regressions before users do.

Do I need a vector database?

Not on day one. Full-text search over a well-chunked Postgres table is usually competitive until you cross a few hundred thousand documents.

AI Engineering

Evaluation harnesses, guardrails and cost ceilings — what separates a demo agent from one that survives real users.

Ovyro Editorial · Engineering desk · 2026-08-05 · 9 min read

Key takeaways

An agent without a scored eval set is an opinion. Freeze 100 real user requests, label the correct outcome, and run them on every prompt change.

Score three things separately: did it pick the right tool, did it pass correct arguments, and did the final answer satisfy the request.

Schema-validate every tool call, cap the loop at a fixed number of steps, and make each tool idempotent so a retry cannot double-charge a user.

Route cheap classification to a small model and reserve the frontier model for the reasoning step that actually needs it.

Track spend per conversation and hard-stop at a ceiling. Monthly budgets tell you about a fire after the building is gone.

How many evals are enough to launch?: Around 80-120 hand-labelled cases covering happy paths, ambiguous inputs and known failure modes catches most regressions before users do.
Do I need a vector database?: Not on day one. Full-text search over a well-chunked Postgres table is usually competitive until you cross a few hundred thousand documents.

#agents#llmops#evaluation

Share this essay

What 4,200 engineers across Bengaluru, Hyderabad and Pune actually ship with — and what they quietly abandoned this year.

A three-pass method for going from abstract to intuition on any machine learning paper in under an hour.