Evaluations and prompt releases
AEL, the Agent Engineering Language, will treat prompts as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback, while active runs stay pinned to their version. Changing a system prompt will be a change to what your agent does, and it will get the same care as a change to its code.
Status
Planned for AEL Beta 0.0.1. AEL is not available yet.
What identifies a version
An agent's behaviour will depend on more than its prompt. Its version will be identified by:
- the exact text of every prompt it resolves, and its input builders;
- its input and output types, tools and permissions;
- its model bindings and their parameters;
- its configuration and its rules for building context.
A prompt-only change will give the agent a new version, even when no code changes. Each run will record the version it was admitted with.
Three layers of tests
| Layer | Model responses | What it shows |
|---|---|---|
| Contract tests | Scripted responses. | Exactly whether types, tool lists, denied permissions, retries, repairs and budgets behave as specified. They must pass exactly. |
| Regression tests | Responses recorded from a real model earlier. | Whether a change breaks behaviour that worked, with no network and no credentials. |
| Behavioural evaluations | A real model, called live. | How well the agent does its job on representative cases. |
- A recording will be tied to the requests, types and tools it was made with. When those change, the recording will be marked incompatible instead of being replayed against the wrong request.
- A live evaluation will declare its sample size, how many times each case repeats, and limits on time, tokens and cost.
- Evaluation results will describe how the agent performed on its cases. They are not proof of how a model will behave in every case, and model output is not reproducible run for run.
Evaluation suites
A suite will be a set of cases. Each case will give:
- the input;
- the scripted or recorded model responses, or a live model binding;
- the expected typed outcome, or the scores that count as success;
- tool calls that must never happen;
- limits for the case.
Good suites cover normal work and the hard cases: missing information, contradictory input, invalid model output, tool errors, instructions hidden in retrieved documents, a budget running out and a delegated agent failing. Scores for live evaluations will be able to include success on the case, evidence quality, unnecessary tool calls and time.
Running a suite
You will run a suite from the command line (command names may change before launch):
ael eval --suite support-regression
ael eval will compare the candidate, your working version, with a baseline, and report the differences in a short summary for people and, with --json, in machine-readable form. The report will keep three kinds of difference apart:
- contract regressions: a case that passed exactly now fails;
- drift: live scores have moved beyond what chance explains;
- missing coverage: a behaviour has no case that tests it.
Drift checks
A hosted model can change without any change on your side. Re-running your live evaluations regularly, and after every provider or model change, will show drift early. Pin model versions where the provider allows it; each run will record the model identity the provider returned.
Comparing prompts
ael prompt diff will show how a prompt changed against a released one:
ael prompt diff --baseline <release>
Promotion and rollback
A new version will reach your deployment through a reviewed plan:
ael deploy plan
ael deploy apply --plan <plan>
- The plan will show what changes: prompts, model bindings, tools and their permissions, and whether each change is compatible with what is deployed, together with the evaluation results.
- Applying the plan will stage the new version, check that it is ready, then switch new runs to it.
- Runs already under way will stay pinned to the version they were admitted with, to the end.
- Rolling back will be one step, to the previous compatible version.
- A change that widens a tool's permissions will need a new, authorized plan: a prompt update alone will never be able to widen them.
Deployment and rollback describes staged deployments in full. Running evaluations, Comparing prompts and Deployment plans describe the commands.