Build a crash-proof AI agent in Postgres, then crash it

To make an AI agent survive a crash, save each finished step to a table and skip saved steps when it runs again. That stops repeated model calls, but on its own it doesn't stop repeated side effects. When we killed our test agent after it had placed an order but before it had saved that step, the re-run placed the order a second time. The fix is an idempotency key: an ID for the step that the system taking the action can recognise. With both in place, a crashed and resumed run ended in exactly the same state as one that never crashed. Each checkpoint cost about 0.6 ms.

We've written before about why agents belong in durable background workflows. This post is the hands-on version: about 65 lines of Node and one Postgres table, then a deliberate crash at the worst possible moment to see what actually holds up.

The agent

The test agent reorders groceries in six steps: plan the order (model call), check the pantry, choose what to buy (model call), place the order, send a confirmation email, and summarise (model call). We stubbed the model so we could count exactly how many times it was called; durability doesn't depend on which model you use.

The journal is one table, keyed by run and step number:

create table agent_steps (
  run_id text  not null references agent_runs(id),
  step   int   not null,
  name   text  not null,
  output jsonb not null,
  primary key (run_id, step)
);

Every step goes through one function. If the step already has a row, it returns the saved output without running anything. If not, it runs the step and saves the result. The two crash() lines are how we killed the process at chosen points:

// Run each step at most once per run. On a re-run, finished steps return their saved output.
export async function step(runId, n, name, fn) {
  if (MODE !== "naive") {
    const { rows } = await db.query(
      "select output from agent_steps where run_id = $1 and step = $2", [runId, n]);
    if (rows.length) return rows[0].output;
  }
  const key = MODE === "durable" ? `${runId}:${n}` : null;   // idempotency key
  const output = await fn(key);
  crash(`after-effect:${n}`);
  if (MODE !== "naive") {
    await db.query(
      "insert into agent_steps (run_id, step, name, output) values ($1, $2, $3, $4)",
      [runId, n, name, JSON.stringify(output)]);
  }
  crash(`after-checkpoint:${n}`);
  return output;
}

The agent itself reads like an ordinary script, which is the point. Each line is a step that runs at most once:

const plan    = await step(runId, 1, "plan",    () => model(runId, 1, "plan this week's order"));
const pantry  = await step(runId, 2, "pantry",  async () => ({ low: ["atta", "toor dal", "paneer", "spinach"] }));
const list    = await step(runId, 3, "choose",  () => model(runId, 3, `order for ${pantry.low}`));
const order   = await step(runId, 4, "order",   (key) => placeOrder(pantry.low, key));
const email   = await step(runId, 5, "email",   (key) => sendEmail(`Order ${order.orderId} placed`, key));
const summary = await step(runId, 6, "summary", () => model(runId, 6, "summarise for the user"));

Step outputs have to be JSON, because they're stored as JSON. That's a useful constraint: anything a later step needs must be written down, not held in a variable that a crash would lose.

Where to crash it

There are two interesting places for step 4 to die:

1. After the step was saved. The order exists and the journal knows it.
2. After the order was placed but before the step was saved. The order exists, but as far as the journal is concerned, step 4 never ran.

The second is the one that matters. It's a window of a few milliseconds, but a deploy, an out-of-memory kill or a function timeout can land in it, and across thousands of runs some of them will.

We ran three versions of the agent against both crash points. Each crashed once and was then run again to completion:

A run with no crash makes 3 model calls, places 1 order and sends 1 email. Only the last design matches that at both crash points.

The missing piece: idempotency keys

The journal can only record that a step finished. It can't know whether a step half-finished, because the crash happened before it could write anything down. Only the system being changed knows whether the order already exists. So the step passes it a key that is stable across retries (run-1:4), and the order store refuses to create a second order with the same key:

async function placeOrder(items, key) {
  const { rows } = await db.query(
    `insert into orders (items, idempotency_key) values ($1, $2)
     on conflict (idempotency_key) do nothing returning id`, [JSON.stringify(items), key]);
  if (rows.length) return { orderId: rows[0].id };
  const existing = await db.query("select id from orders where idempotency_key = $1", [key]);
  return { orderId: existing.rows[0].id };
}

The Postgres documentation is plain about what ON CONFLICT DO NOTHING does: it "simply avoids inserting a row as its alternative action". The unique index does the real work. On a retry, the insert does nothing and the step returns the order it already made, so step 5 emails the right order number.

Most payment and messaging APIs support the same pattern. Stripe's API reference explains why it exists:

If a connection error occurs, you can safely repeat the request without risk of creating a second object or performing the update twice.

Note the fine print in that same page: keys can be removed "after they're at least 24 hours old". If a crashed run might sit for longer than that before it's resumed, check the step's effect directly rather than relying on the key alone.

What it costs

On our test machine (Node 20 and Postgres 14 on the same laptop), saving a new step took a median of 0.64 ms (1.34 ms at the 99th percentile), and replaying a saved step took 0.24 ms. Over a network to a managed database you'd add a round trip or two. A single model call takes far longer than either, so the journal is not where an agent's time goes.

Rules we'd follow

1. One row per step, keyed by run and step number. The primary key is what makes "at most once" true.
2. Every side effect gets a key derived from the run and step, never a fresh random ID, which would be different on the retry.
3. If the side effect is in your own database, do it in the same transaction as the checkpoint. Then the crash window disappears entirely, with no key needed.
4. Keep steps deterministic around the journal. If step order depends on the time of day or a random number, a re-run can line steps up against the wrong saved rows. Put anything variable inside a step, so it's saved.
5. Test the crash, don't assume it. Our own earlier post said progress is never lost. That's true of the journal. It took deliberately killing the process to see that it isn't true of the order.

Our opinion

Checkpointing an agent is easy; making its side effects safe to repeat is the actual work. Most guides to durable agents stop at "save each step", and our test shows exactly where that falls short. The duplicate didn't come from the model or the framework. It came from a few milliseconds between two writes.

We'd also argue against reaching straight for a workflow engine. Engines are worth it for long-running, many-step, multi-service work. For an agent with a handful of steps that already uses Postgres, one table and one function get you most of the way, and you can read every line of it. Either way, the idempotency keys are still your job. No engine can make someone else's API safe to call twice.

How Ashvara helps

We build the backend that agents run on: the step journal, the idempotency keys on every action that costs money or sends a message, and the tests that crash it on purpose before production does.

That's the backend and API development side of our AI solutions work. If you have an agent that sometimes does things twice, or that you're nervous to restart mid-run, tell us what it does. The crash test above takes an afternoon, and it tells you where you stand.

Sources: our test agent (Node 20, PostgreSQL 14, stubbed model, each scenario run once from a clean database); Stripe API reference: idempotent requests; PostgreSQL documentation: INSERT and ON CONFLICT. Timings are from a local database and will be higher over a network.