Durable runs and approval
Runs in AEL, the Agent Engineering Language, will be durable: they will pause, resume, wait for human approval, survive restarts and reconcile side effects safely. A long-running agent will keep its tenant, its budget and its record of what it has done through all of these.
Status
Planned for AEL Beta 0.0.1. AEL is not available yet.
The life of a run
| State | Meaning |
|---|---|
| Queued | Admitted, with its budget reserved, and waiting to start. |
| Running | Doing work. |
| Waiting | Waiting for a reply, a timer or more input. |
| Awaiting approval | Paused until a person approves, rejects or revises a proposed action. |
| Cancelling | Stopping its child work and cleaning up. |
Every run will end in exactly one final state:
| Final state | Meaning |
|---|---|
| Succeeded | It produced its typed output. |
| Partial | It produced part of its output, and recorded what is unfinished. Allowed only when the output type accepts partial results. |
| Failed | It ended with a typed error, including a rejected approval. |
| Timed out | Its deadline passed, including an approval that expired. |
| Exhausted | Its budget ran out. |
| Cancelled | It was cancelled. |
A final state will never change, and you will not be able to resume a finished run. To continue its work, you will start a new run that refers to the original one and keeps its budget. A run's result will be stored before its reply is delivered, so delivering a reply again will never run the work again.
Pausing and resuming
A durable run will save checkpoints as it goes. A checkpoint will hold:
- where the run is in its workflow, and the version of its state;
- effects it has started but not confirmed;
- its budget reservations and approvals;
- the prompt, configuration and model binding it was admitted with.
A checkpoint will hold the decisions and actions the run took, never a model's private reasoning. A resumed run will continue with the prompt and configuration it started with, or move to a new version only through an explicit, tested migration.
Human approval
A node will be able to pause for a person's approval, or for more input, before it acts. Approval will be your policy, where you want it; it will not be required for every action.
- An approval will be bound to the exact proposed action: its arguments, the policy version, who may approve and an expiry time.
- The approver will be able to approve, reject or revise. A revised proposal will need a new approval.
- A stale, replayed or altered approval will be refused.
- On approval, the run will check the caller's permissions again before acting, and draw on the same budget.
- A rejection will end the run as failed; an approval that expires will end it as timed out.
Surviving restarts
- Runs, timers, deadlines and child work will be recovered after a restart or a crash.
- Only one worker will own a run at a time. A worker that comes back after another has taken over the run will not be able to continue it.
- Work whose deadline passed while the program was down will end as timed out.
- When a client reconnects, it will receive the run's events and result: work that was already paid for will not be started again.
- A retried run will keep the original deadline and budget, and the record of side effects it already caused.
Side effects
Some effects happen outside your program: a payment, a message, a change in another service. AEL will not be able to make such an effect happen exactly once, and will say so. Instead, runs will make effects safe to reconcile:
- Idempotency keys will let a service recognize a repeated request for the same effect.
- Receipts will record what a service confirmed.
- Unknown outcomes will stay visible, when a reply was lost after an effect may have happened, until your code or a person resolves them.
- Compensation steps that you write will be able to undo or offset an effect, where the other service allows it.
Cancelling or restarting a run will never claim that an effect was undone.
Where durable state lives
Durable runs will need storage for their checkpoints. A deployment that only answers short requests will be able to do without it: its runs will start and finish within one request. Memory and state describes how state is scoped and kept.