Running tests
In AEL, the Agent Engineering Language, ael test (command names may change before launch) will run a project's tests. Tests will be AEL code, checked like the rest of your project and held to the same limits as every other source file.
Status
Planned for AEL Beta 0.0.1. AEL is not available yet.
Usage
ael test
ael test src/nodes
ael test src/nodes/summarize.node.ael
ael test --json
| Argument or option | Meaning |
|---|---|
[path] | Run only the tests in one folder or one file. Without a path, every test of the project in the current folder. |
--json | Results in machine-readable form. |
What it does
- It will check the project first, through the same checks as
ael check. A project that fails them will run no tests. - It will run the tests it finds, and report each one that fails with the place in your source.
- It will end with a failure status when any test fails, so an automated build stops there.
Tests without a network
A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The standard library will let a test replace each of these: mock model responses you write, or responses recorded from a real model and played back; a controlled clock; a fixed random seed; simulated devices and sensors; and a chosen order for concurrent work.
Recorded responses will play back without credentials or a network. A recording will stop matching when the request, the output type or the tools change, and the test will say so. This is how prompts will become releasable business logic, with recorded-model regression tests you can run on every change.
Testing describes what a test can replace and check.
Tests and evaluations
ael test will run tests that must pass exactly. To measure how well an agent does its job against a live model, with a sample size and limits on time, tokens and cost, use ael eval instead.
Not settled yet
How you will declare a test, and how ael test will find tests in a folder, will be published when the standard library is final.
Related
- Testing: writing tests.
- Running evaluations: evaluation suites against prompt and model changes.
- Evaluations and prompt releases: the three layers of agent tests.