Designing Write-Ahead Approval Logs for High-Risk LLM Actions

The first time most teams realize they need a write-ahead approval log is not during architecture review. It is after an agent has already done something expensive, irreversible, or embarrassing.

A common pattern looks like this: a team launches an internal operations copilot that can read tickets, query dashboards, draft SQL, trigger cloud workflows, and call internal admin APIs. In staging, it looks great. Engineers ask it to restart services, finance asks it to refund credits, support asks it to deactivate compromised accounts, and IT asks it to rotate secrets in test environments. The model appears cautious. The tool wrappers add some validation. There is a “are you sure?” confirmation for a few actions. Everyone feels reasonably safe.

Then production traffic arrives.

One day the agent receives a support case: “Customer says they were charged twice. Please refund the duplicate invoice and re-enable their premium plan.” The agent retrieves the customer record, sees a failed payment retry, a manual credit memo, and a temporary suspension. It decides the “corrective” action is to issue a refund and reactivate the plan. It calls the refund API and the entitlement API in sequence. The first succeeds. The second fails because the plan was intentionally suspended for abuse review, not billing error. Support notices only after the customer posts screenshots. Finance now has an unauthorized refund. Risk wants to know who approved the action. Engineering wants to know what the model saw. Legal wants an audit trail. Nobody has a satisfying answer.

In the logs, there is a prompt, some tool arguments, and a 200 response from the refund API. But crucial details are missing or inconsistent:

What exact action did the agent intend before execution?
Was the human asked to approve that exact action or a paraphrased summary?
What policy checks were evaluated, and against which context?
Could the second tool call have been blocked after the first one succeeded?
Can the team replay the exact decision path in a sandbox?
Can they cancel queued actions not yet executed?
Which actions are compensatable, and what rollback playbook applies?

This is the point where many teams start bolting approvals onto the side of an agent loop. They add a Slack confirmation, or a modal in the admin app, or an allowlist in the tool router. Those controls help, but they are not enough when the system can produce side effects across heterogeneous tools, asynchronous workflows, and multiple approval domains.

The missing primitive is a write-ahead approval log.

A write-ahead approval log, or WAAL, borrows the most useful idea from write-ahead logging in databases: before a side effect happens, the system durably records the intended action in a canonical, reviewable, replayable form. Execution only proceeds against that logged intent after approval and policy checks. If approvals change, policies tighten, or operators need to intervene, they do so against a stable record of intent rather than an ephemeral model thought or UI confirmation string.

For high-risk LLM systems, that primitive becomes the difference between “the model decided something weird” and “we have a controlled execution substrate with auditability, cancellation, replay, and rollback semantics.”

The pattern: the agent is not the transaction boundary

The core pattern is simple:

The model proposes one or more actions.
The system normalizes those proposals into canonical action intents.
Each intent is written durably to an approval log before execution.
Policy engines, humans, and automated risk checks evaluate the logged intent.
An executor performs only approved log entries.
Every state transition is appended to the log: proposed, validated, approved, denied, expired, dispatched, succeeded, failed, compensated, canceled.
Replay, rollback, and audits operate from the log, not from model text.

This sounds obvious. In practice, many teams mistakenly treat the model turn as the unit of control. They inspect assistant messages, parse tool calls, maybe require confirmation, then immediately invoke tools. That design breaks down for high-risk actions because the model turn is not a stable transaction boundary.

Why? Because real production systems are messy:

One user request can lead to multiple dependent tool calls.
Tool calls can be delayed, retried, or fan out into downstream jobs.
Human review may happen minutes or hours later.
Policies may change between proposal time and execution time.
The same action might need approval from different parties depending on amount, data sensitivity, geography, or customer tier.
Some operations are reversible, some are not, and some are only partially compensatable.

If you do not externalize intended actions into a durable log before execution, you cannot reliably govern those realities.

Why the naive approach fails

Let’s make the failure modes concrete.

1. “Human in the loop” without canonical intent is theater

A lot of teams implement approval by showing a human a natural-language summary generated by the model:

“I’m about to refund invoice INV-44821 and reactivate the account.”

That is not approval-grade. It is a lossy paraphrase. It may omit parameters, preconditions, identity context, linked records, risk flags, or downstream side effects. The human might approve the summary while the actual tool call includes additional arguments or a broader scope. If challenged later, you cannot prove the approver saw the exact executable action.

Approval must bind to canonical intent, not assistant prose.

2. Inline tool execution makes cancellation and replay nearly impossible

If the agent directly calls tools as part of a streaming reasoning loop, you lose control over timing. There is no stable “pending” state for operations teams to review, pause, or cancel. There is no canonical artifact to replay in a staging environment. If you later discover a policy bug, there is no queue of not-yet-executed actions to invalidate.

3. Policy checks on prompts are brittle

Some systems run safety or policy checks on the user request, model response, or raw tool arguments. That is better than nothing, but policies need a normalized action schema to be effective. “refund_customer” might be low risk under $10, medium risk over $100, forbidden for certain account states, and require dual approval for regulated customers. Prompt text is too inconsistent a substrate for those determinations.

4. Retry semantics become dangerous

Without a write-ahead log and explicit idempotency, a transient network error can turn into duplicate side effects. The model may reissue the same action with slightly different arguments. An orchestrator may retry the request without knowing whether the downstream side effect already occurred. A robust WAAL design makes the intended action addressable and idempotent before any external call is attempted.

5. Multi-step workflows create split-brain outcomes

High-risk actions often span systems: issue refund, revoke token, notify customer, open compliance case, update CRM. If you execute these inline from the agent, partial failures are unavoidable and hard to reason about. You need a log that captures both the high-level planned action set and the per-step execution state, plus compensating actions where possible.

6. Audits require durable state transitions, not best-effort logs

Compliance and incident review care about who requested an action, what context was used, which policy version evaluated it, who approved it, when it executed, and what happened after. Application logs are noisy, sampling can drop records, and unstructured traces are hard to reconstruct. You need an append-only state machine around intended actions.

What a better approach looks like

The better approach is to separate decision proposal from side-effect execution, and to put a durable approval log in the middle.

At a high level, the architecture looks like this:

Agent runtime: The LLM and orchestration loop that interprets user requests and proposes actions.
Intent normalizer: Converts model-generated tool requests into a canonical action schema.
Write-ahead approval log: The system of record for all proposed high-risk actions and state transitions.
Policy engine: Evaluates logged intents against rules, risk models, identity, resource state, and environment context.
Approval service: Manages human and automated approvals, escalation, expiration, dual-control, and delegation.
Execution dispatcher: Pulls only approved log entries and invokes tool executors with idempotency and concurrency controls.
Tool executors: Deterministic adapters around side-effecting systems.
Compensation/rollback orchestrator: Executes defined compensating actions when needed.
Audit and replay plane: Enables forensic reconstruction, sandbox re-execution, and simulation.

The crucial boundary is this: the LLM never directly causes the side effect. It can propose, justify, and gather context, but actual execution is driven by a non-LLM subsystem reading approved intents from the log.

The approval log as a state machine

Your WAAL should behave like an append-only state machine, not just a table with a current status field.

A useful lifecycle for each action intent might include:

PROPOSED: Intent created from an agent proposal.
NORMALIZED: Tool name and arguments canonicalized.
VALIDATED: Schema, preconditions, and static checks passed.
POLICY_PENDING: Awaiting policy evaluation.
POLICY_REJECTED: Rejected by automated policy.
APPROVAL_REQUIRED: Needs human or multi-party approval.
APPROVED: Required approvals collected.
DENIED: Explicit human or policy denial.
EXPIRED: Approval window elapsed.
DISPATCHED: Sent to executor.
EXECUTION_SUCCEEDED: Side effect completed.
EXECUTION_FAILED_RETRYABLE: Temporary failure.
EXECUTION_FAILED_TERMINAL: Permanent failure.
COMPENSATION_PENDING: Awaiting rollback or compensating action.
COMPENSATED: Compensation succeeded.
CANCELED: Prevented before execution.

In practice, you will also want immutable events attached to each transition:

actor identity
time
policy version
model version
prompt/session trace references
resource version or snapshot hash
idempotency key
approver rationale
execution receipt
compensation receipt

That event history is the foundation for auditability and replay.

Canonical intent schema: the most important design choice

Most WAAL projects succeed or fail based on the intent schema. If the schema is fuzzy, policies become unreliable and approvals become ambiguous.

A canonical intent record should include at least:

intent_id: globally unique identifier
trace_id / session_id: links to the broader agent interaction
proposed_by: agent identity, model version, user identity
action_type: normalized verb like refund_invoice, disable_account, rotate_secret
target: resource identifiers and types
arguments: canonical parameters in typed form
risk_level: computed or assigned classification
justification: concise model or operator rationale
preconditions: facts assumed true at proposal time
policy_context: environment, tenant, geography, account state, amount bands
approval_requirements: who or what must approve
expires_at: after this time, revalidation required
execution_constraints: ordering, rate limits, maintenance windows
idempotency_key: stable execution key
compensation_plan: compensating action type if available
redaction_views: fields visible to approvers vs auditors vs operators
artifacts: supporting evidence, snapshots, query results, retrieved documents

A strong schema lets you ask operational questions directly:

Show all rotate_secret intents over the last 30 days that bypassed human approval due to break-glass policy.
Replay all denied refund_invoice actions in sandbox against the current policy version.
List all executed intents where the target resource version changed between proposal and dispatch.

Those are production questions. If your data model cannot answer them, your control plane is too weak.

Approval design: humans, policy engines, and dual control

A WAAL is not just for human approvals. It is the coordination substrate for multiple approval authorities.

Automated policy approval

Many actions should be auto-approved if low risk and policy-compliant. Example:

restart stateless service in staging
grant read-only access for temporary debug session under 1 hour
refund under $5 for non-regulated, non-flagged accounts

Automated approval keeps latency and operational overhead manageable.

Human approval

Use human review for actions where the risk comes from incomplete world state, cross-functional judgment, or non-machine-verifiable context. For example:

account reactivation with abuse signals
high-value credits or refunds
production secret rotation during business hours
privilege changes involving sensitive data

Dual control and split approvals

Some actions should require multiple approvals from different domains, such as finance plus support, security plus infrastructure, or on-call engineer plus service owner. The approval service should support policy-defined approver sets and quorum rules.

Break-glass execution

In emergencies, you may permit policy-bounded break-glass execution with stronger audit requirements, shorter expiration, and mandatory post hoc review. Crucially, break-glass should still create a write-ahead log entry before execution. Emergencies are where auditability matters most.

Policy evaluation: treat it like code, not prompt engineering

For high-risk actions, policy should live outside the model in deterministic systems.

A practical policy stack often combines:

static schema validation
deterministic business rules
RBAC/ABAC checks
resource state checks from source systems
anomaly/risk scoring models
environment guards like prod vs staging
rate and amount thresholds
maintenance window rules

Teams often ask whether to use OPA, Cedar, custom rules engines, SQL-based controls, or application logic. The answer is usually a combination.

OPA/Rego: strong for infrastructure-style policy and decoupled rule evaluation; more operational overhead.
Cedar: excellent for authorization-style policies with principal/action/resource semantics; narrower scope but very clean.
SQL/application logic: fast to adopt; tends to sprawl and becomes hard to audit if not disciplined.

My recommendation: use a central policy decision point with explicit versioning, even if some checks remain application-side initially. The important thing is that every approval decision references a policy version and input snapshot.

Execution architecture: make executors boring

A recurring anti-pattern is letting the agent stay “in the loop” during execution. That increases unpredictability. For high-risk actions, your executors should be boring, typed, deterministic services.

A clean execution flow is:

Dispatcher polls or subscribes to approved intents.
Dispatcher acquires a lease/lock on the intent.
Executor rechecks freshness-sensitive preconditions.
Executor invokes downstream system with idempotency key.
Executor records receipt, output, and resulting resource version.
Dispatcher appends terminal or retryable state.
If needed, compensation workflow is registered.

This separation delivers several benefits:

You can cancel intents before dispatch.
You can pause execution by action type or tenant.
You can replay or dry-run intents in sandbox.
You can migrate model providers without changing execution controls.
You can apply backpressure and concurrency limits independent of the agent.

Replay, revalidation, and cancellation

One of the most valuable properties of WAAL is operational control after proposal time.

Replay

Replay should mean at least three different things:

Decision replay: Re-run policy evaluation against the original intent and context.
Simulation replay: Execute the intent against mocks or a sandbox.
Forensic replay: Reconstruct the timeline of proposal, approval, and execution for incident review.

These are different capabilities and need different stored artifacts.

Revalidation

For stale approvals, the dispatcher should revalidate preconditions before execution. Example: if a support-approved refund intent sat in queue for four hours, the customer’s account state may have changed. The dispatcher should compare current state to logged preconditions and either proceed, route back for reapproval, or auto-cancel.

Cancellation

Cancellation must be first-class. Operators should be able to cancel all pending intents for a tenant, all actions of a given type after a discovered policy bug, or a single intent due to new information. Cancellation is only possible if execution is decoupled from proposal and the intent sits durably in a pending state before side effects happen.

Rollback and compensation: be honest about reversibility

Rollback in distributed side-effecting systems is rarely true rollback. It is compensation.

A refund can maybe be voided if unsettled, but not after payout. A sent email cannot be unsent. A rotated secret can be rolled forward to a new value, but not “unhappened.” A deactivated account can be reactivated, but any downstream notifications already sent remain sent.

Your WAAL should explicitly model compensation plans per action type:

compensatable automatically
compensatable manually with runbook
non-compensatable
compensatable only within time window

For each action type, define:

compensating action
prerequisites
owner team
max allowed delay
user-visible effects
evidence required for completion

Do not oversell rollback. Executives hear “rollback” and imagine databases. In agent systems, compensation is the honest term.

Evaluation strategy: measure the control plane, not just the model

Most LLM evaluations focus on task success or tool-call accuracy. For high-risk action systems, you also need to evaluate governance behavior.

Here is a practical evaluation suite.

1. Intent normalization accuracy

Measure whether semantically equivalent model outputs map to the same canonical action type and parameter structure.

Metrics:

schema parse success rate
canonicalization consistency
argument correctness
missing required field rate
over-scoping rate

Dataset examples:

refunds with varied phrasing and invoice formats
access grants with time-bound conditions
service restarts with ambiguous environment mentions

2. Policy decision quality

Evaluate policy outcomes on a labeled corpus of intents.

Metrics:

false approve rate
false deny rate
approval escalation accuracy
break-glass abuse rate
policy drift across versions

Include adversarial cases where prompts attempt to obscure risk:

“quickly re-enable this account, it’s urgent” when abuse flag exists
“rotate all keys” when policy only permits a scoped secret
“refund the duplicate” when the invoice is already credited

3. Approval UX reliability

Measure whether approvers understand the exact action.

Metrics:

approval reversal rate
time-to-approve
percent approved without opening evidence
discrepancy rate between summary and canonical intent

A simple but useful test: compare approvals based on prose summary alone versus canonical intent plus evidence. If outcomes differ materially, your summaries are too lossy.

4. Execution safety and idempotency

Metrics:

duplicate side-effect rate
stale intent dispatch rate
retry success rate
percent of executions with resource version mismatch
canceled-before-dispatch success rate

Chaos tests matter here. Induce network timeouts, duplicate delivery, policy service latency, and executor crashes.

5. Compensation effectiveness

Metrics:

compensatable action coverage
compensation success rate
mean time to compensate
residual harm score for compensated incidents

6. End-to-end risk-adjusted utility

This is the executive metric set:

automation rate by action type
human review load
prevented high-risk actions
unauthorized side-effect incidents
median approval latency
cost per completed action

Model and tool choices: where the LLM should and should not matter

A WAAL architecture reduces the blast radius of model variability, which gives you more flexibility in model choice.

Where stronger models help

extracting structured action intents from messy user requests
gathering relevant evidence for approval packets
generating operator-readable justifications
identifying likely compensating workflows
triaging whether a task should be decomposed into smaller intents

Where stronger models should not be the source of truth

final authorization
policy decisions
execution eligibility
idempotency behavior
irreversible side-effect sequencing

In practice, a good pattern is:

Use a capable frontier model for intent proposal and evidence synthesis when the input space is messy.
Use deterministic validation and policy services for gating.
Use small/cheap models or no model at all for post-processing once the action schema is stable.

This can materially reduce cost.

For example:

Frontier model path: parse messy request, collect context, produce structured intent and rationale.
Cheap model or rules path: render approval summary from canonical intent, classify low-risk intents, standardize evidence labels.
No model path: execution, policy, dispatch, idempotency, compensation.

That division also improves latency predictability. High-risk execution pipelines often fail SLOs because teams keep the LLM in the critical path too long.

Cost and latency tradeoffs

WAAL introduces overhead. That is not a flaw; it is the price of control. The question is how to spend that overhead intelligently.

Cost drivers

extra storage for append-only event logs and artifacts
policy evaluation infrastructure
approval UX and notification systems
sandbox/replay environments
human review time
executor engineering for idempotency and compensation

Latency drivers

intent normalization
evidence retrieval
policy checks against source systems
human approval wait time
revalidation at dispatch

How to keep it practical

Tier actions by risk: Only send genuinely high-risk actions through full approval workflows.
Use auto-approval for bounded low-risk actions: Keep humans focused where they add value.
Precompute policy context when possible: Cache account risk status, environment metadata, and ownership mappings.
Bundle evidence: Approvers move faster when context is packaged once, not fetched ad hoc.
Set expiration windows: Prevent stale approvals from becoming hidden latency bombs.
Keep execution adapters simple: Complexity in executors drives both latency and incident rates.

A useful mental model is payments fraud infrastructure, not chatbot UX. You are building a risk control plane around side effects.

Implementation details that matter more than teams expect

1. Use append-only events plus materialized views

Store immutable events as the source of truth. Build current-state views for fast querying. This gives you auditability without making operators query raw event streams for every dashboard.

2. Version everything

At minimum, version:

intent schema
policy bundles
tool contracts
executor code
model/provider configuration
approval workflow definitions

When incidents happen, “which version made this decision?” should be answerable in minutes.

3. Snapshot or hash critical evidence

If approvals depend on retrieved documents, account state, or query results, store snapshots or content hashes. Otherwise your replay attempts will silently drift because the underlying systems changed.

4. Treat tool contracts like public APIs

Every high-risk tool should have:

typed inputs
explicit side-effect description
idempotency support
timeout semantics
retry policy
compensating action metadata
least-privilege credentials

Do not let agent tools remain thin wrappers around arbitrary scripts.

5. Separate “intent approved” from “safe to execute now”

Approval validates the action under known context. Execution still needs freshness checks. This distinction prevents stale approvals from causing bad side effects.

6. Build redaction-aware approval packets

Approvers need enough evidence to make decisions, but not everyone should see raw PII, secrets, or internal abuse notes. The WAAL should support audience-specific views over the same underlying intent.

7. Add bulk controls

In incidents, operators need coarse levers:

pause all executions for tool X
cancel all pending intents for tenant Y
require human approval for all actions above risk level Z
disable break-glass globally

If your only control is editing prompts or redeploying the agent, you are under-instrumented.

A concrete reference architecture

Here is a production-oriented design that works well for many teams.

Control plane

Intent API: receives normalized action proposals from agent runtime.
Event store: append-only durable log, often on relational DB + immutable event table, or Kafka plus relational index.
State projector: materializes current intent state views.
Policy service: synchronous decision API with versioned bundles.
Approval service: workflow engine for human/auto approvals, notifications, escalation.
Audit service: search, reporting, evidence export.
Replay service: sandbox execution and policy replay.

Data plane

Dispatcher: consumes approved intents.
Executor workers: one per tool domain or action family.
Compensation workers: perform compensating actions.
Source-of-truth connectors: fetch fresh state for revalidation.

Storage choices

A relational database is often the right first choice for the WAAL metadata because approvals, filters, and audits are relational in nature. If throughput gets high, you can pair it with Kafka for event fan-out. Many teams prematurely reach for event-sourcing stacks when PostgreSQL plus outbox patterns would have been enough.

A practical starting stack:

PostgreSQL for intents, approvals, event log, materialized views
Kafka or SQS for dispatch queues
OPA/Cedar/custom service for policy
Temporal/Durable Functions/Argo/Step Functions for compensation workflows if needed
Object store for evidence artifacts and snapshots

Why not let the workflow engine be the log?

You can use a workflow engine, but do not let it become the only approval ledger. Workflow histories are often awkward for audit search, policy replay, and cross-intent controls. Keep a domain-specific approval log even if workflow infrastructure powers orchestration.

Failure scenarios to design for

If you are serious about WAAL, test these scenarios before production:

policy service outage during approval
duplicate dispatch messages
executor crash after downstream success but before receipt writeback
approver acts on an expired intent
target resource changes between approval and execution
compensation action fails halfway
tool contract version changes while intents are pending
redaction bug exposes sensitive evidence to broad approvers
model proposes a broad-scope wildcard target not allowed by schema
queue backlog causes approvals to become stale

For each scenario, decide the safe default. In high-risk systems, the safe default is usually fail closed on execution while preserving operator visibility and recovery paths.

Anti-patterns to avoid

A few anti-patterns show up repeatedly.

“We log after execution”

That is not write-ahead. It is post hoc observability. Useful, but insufficient.

“The approval is just a chat confirmation”

Natural-language confirmations are too ambiguous for high-risk side effects.

“The model decides if approval is required”

No. The model may suggest risk level, but approval requirements should be determined by policy over canonical intents.

“One giant execute_business_action tool”

This destroys policy precision and approval quality. High-risk tools should be narrow and typed.

“We’ll add rollback later”

Compensation design changes how you decompose actions and define tool contracts. It cannot be bolted on cleanly afterward.

“All actions go through the same approval path”

That creates too much latency and reviewer fatigue. Risk-tiered workflows are essential.

A practical adoption path

You do not need to build the perfect control plane on day one. A staged rollout works better.

Phase 1: Wrap the top five risky actions

Pick the small set of side-effecting tools that matter most: refunds, access grants, secret rotation, account deactivation, production deploy/restart. Define canonical intent schemas and require write-ahead logging plus approval before execution.

Phase 2: Externalize policy and idempotency

Move approval rules out of prompts and into versioned policy services. Add idempotency keys and explicit executor state transitions.

Phase 3: Add replay and cancellation

Build operator controls for canceling pending intents and replaying decisions in sandbox. This is where incident response starts getting much easier.

Phase 4: Add compensation and bulk controls

Implement compensating workflows for the most damaging action types. Add system-wide pause switches and per-tenant controls.

Phase 5: Evaluate and optimize

Measure automation rate, review burden, false approvals, and latency by action type. Then tighten schemas, simplify tools, and tune which actions require human review.

The takeaway

If your LLM system can trigger high-risk side effects, the core problem is not just model quality. It is transaction control.

A write-ahead approval log gives you a durable, canonical, reviewable representation of intended actions before they happen. That single design move unlocks the controls production teams actually need: exact approvals, deterministic policy checks, idempotent execution, cancellation, replay, auditability, and honest compensation.

The most important mindset shift is this: the agent should propose actions, not directly perform them. Execution belongs to a governed substrate that records intent first and acts second.

Teams that skip this usually end up with fragile approval theater, hard-to-replay incidents, and side effects they cannot explain. Teams that invest in it end up with something much more valuable than a safer agent. They build a control plane for AI-driven operations.

And in high-risk environments, that control plane is the product.