Testing
Tests in AEL, the Agent Engineering Language, will be AEL code, checked like the rest of your project. The command line will run them, and the standard library will give them what an agent needs to be tested without a network, a model account or a device on the desk.
Status
Planned for AEL Beta 0.0.1. AEL is not available yet.
Running tests
ael test will run a project's tests (command names may change before launch). Give it a folder or a file to run only the tests there.
ael test
ael test src/nodes
Like the other commands, it will take --json for machine-readable results.
Tests that give the same result every time
A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The library will let a test fix each of these:
| What varies | What a test will use instead |
|---|---|
| A model's answers | Mock responses you write, or responses recorded from a real model and played back. |
| The clock | A controlled clock that moves only when the test advances it. |
| Random choices | A fixed seed. |
| Devices and sensors | Simulated inputs, such as sensor readings or serial data, fed to the same application code. |
| Scheduling | A chosen order for concurrent work, so a failure can be reproduced. |
- Recorded responses will play back without credentials or a network. A recording will stop matching when the request, the output type or the tools change, and the test will say so rather than accepting a response to a different question.
- A controlled clock will let a test cross a timeout, a schedule's time-zone change or a timer's wraparound without waiting for it.
- A test will be able to make things fail on purpose: a full mailbox, a cancellation, a timeout, a fault that the supervisor handles.
The application code will stay the same. What changes will be what it is connected to.
What to check
- Results: an operation's
Result, including each error your code handles. - Agent behaviour: which tools a run called and which it did not, what it returned, and whether it stayed within its budget of attempts, repairs and tokens.
- Fallbacks and checks: that a missing system prompt, a malformed output or a failed output check leads where you expect.
Prompt evaluations
A prompt will be business logic, and a change to it will change what an agent does. Prompts will be treated as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback, while active runs stay pinned to their version.
ael eval will run an evaluation suite against a prompt or model change before you promote it:
ael eval --suite <suite>
- A suite will list inputs, the typed outcome each should produce, tool calls that must not happen, limits and how answers are scored.
- Deterministic cases will run against mock or recorded responses. Cases against a live model will run only when you ask, with a sample size, a number of repeats, and time, token and cost ceilings that you set.
- A report will compare the new prompt or model with the one it replaces, and keep exact regressions apart from changes in a live model's behaviour.
- Scores from a live model are observations with uncertainty, not proof that an agent is correct.
Evaluations and prompt releases describes suites, promotion and rollback, and Running evaluations and Comparing prompts the commands.
Record and replay
You will record control and orchestration runs and replay them to find the first point where behaviour diverged. Replay will use the recorded responses from outside the program, so it reproduces your program's decisions; it will not make a live model's output repeatable. See Reporting, logs and replay.
Not yet described
How you will declare a test, and the names of the testing operations, will be published when the library is final.