Running evaluations
In AEL, the Agent Engineering Language, prompts will be treated as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback. ael eval (command names may change before launch) will run an evaluation suite against a prompt or model change, before you promote it.
Status
Planned for AEL Beta 0.0.1. AEL is not available yet.
Usage
ael eval --suite support-regression
ael eval --suite support-regression --json
| Option | Meaning |
|---|---|
--suite <suite> | The evaluation suite to run. |
--json | The report in machine-readable form. |
What a suite holds
A suite will be a set of cases. Each case will give the input; the scripted or recorded model responses, or a live model binding; the expected typed outcome, or the scores that count as success; tool calls that must never happen; and limits for the case. Evaluations and prompt releases describes suites in full.
What it runs
| Cases | Model responses | Result |
|---|---|---|
| Contract cases | Scripted responses. | Will have to pass exactly: types, tool lists, denied permissions, retries, repairs and budgets. |
| Regression cases | Responses recorded from a real model earlier, played back with no network and no credentials. | Will have to pass exactly. A recording that no longer matches its request, types or tools will be marked incompatible, never replayed against the wrong request. |
| Live cases | A real model, called live. | Scores, measured over a sample. Run only when you ask for them. |
Live cases will declare their sample size, how many times each case repeats, and ceilings for time, tokens and cost. A live run will stop at its ceilings.
The report
ael eval will compare the candidate, your working version, with a baseline, and report the differences in a short summary for people and, with --json, in machine-readable form. It will keep three kinds of difference apart:
- contract regressions: a case that passed exactly now fails;
- drift: live scores have moved beyond what chance explains;
- missing coverage: a behaviour has no case that tests it.
The report will name the exact prompts, model bindings, tools and configuration it evaluated, so a result always belongs to one version of the agent. The command will end with a failure status when a contract or regression case fails.
What the results mean
Scores from a live model are observations with uncertainty, not proof that an agent is correct in every case, and model output is not reproducible run for run. Pin model versions where the provider allows it, and re-run live cases after every provider or model change to catch drift early.
Before promotion
A typical release of a prompt change:
ael eval --suite support-regression
ael prompt diff --baseline <release>
ael deploy plan
The deployment plan will show the evaluation results together with what the release changes. Comparing prompts and Deployment plans describe the next two steps.
Not settled yet
How a suite is written, and how you will choose the baseline and the model for live cases, will be described when the evaluation format is final.
Related
- Evaluations and prompt releases: versions, suites, promotion and rollback.
- Testing: prompt evaluations beside the tests that must pass exactly.
- Running tests: the project's tests.