Skip to content

Roll back a release

Three commands, three different situations. Choosing the wrong one is the usual way an incident gets worse.

SituationCommandWhat it does
A deploy was interrupted and you want to finish itob resumeContinues from durable phase state
A deploy was interrupted and you want it undoneob abortReverts the interrupted deploy
A deploy completed but the release is badob rollbackActivates the previous release

Start by asking the server, not your memory:

Terminal window
ob status --output json

ob status exits non-zero on divergence, so it is safe to use as a gate in a script.

Terminal window
ob rollback

Every release has a strict manifest state and an explicit predecessor. Rollback activates the current serving release’s eligible predecessor, verifies it, and records the transition; it does not select a directory by timestamp or name. deployment.retain_releases (default 5) bounds ordinary history while always protecting current, predecessor-chain, and checkpoint-referenced releases.

Concretely:

  • Supporting services are not rolled back. A service runs in its own Compose project and outlives every release, so no deploy and no rollback stops it or removes its volume. That protects your data; it also means a schema migration is still applied after you roll the application back.
  • A job with data_effect: migration or destructive constrains the gate. That is what the field is for.
  • A job whose result evidence is missing becomes changed=unknown and halts before workload replacement rather than proceeding on an assumption.

Both read the append-only journal, strict release manifests, and the durable activation checkpoint. Recovery removes only containers owned by the exact interrupted release, restores and verifies its predecessor, and clears the checkpoint last. A failed postcondition leaves the checkpoint retryable instead of hiding divergence.

Terminal window
ob resume --output ndjson
ob abort --output ndjson

Activation splits the two halves of a deploy. Interrupted before it, the release never took effect and ob resume replays the phases that remain. Interrupted after it — retention, schedule sync or the post_deploy hook failed while the new release was already current and healthy — there is nothing to replay: ob resume completes only those remaining steps, skipping any the journal already records as done, and the release keeps serving throughout. The manifest carries a failed operation outcome in the meantime, so a healthy release and an unfinished operation stay distinguishable.

ob resume re-verifies the serving release first, then finalizes only when the recorded activation, the current release, the serving manifest and its recorded predecessor, the absence of an open activation checkpoint, and the live workload labels all agree. One disagreement returns finalize_refused: no step runs and no release, symlink or workload changes.

Only the newest deploy is recoverable. Once a later deploy reaches a terminal state of its own — finished, aborted, or automatically rolled back — the older interrupted one is history rather than pending work, and resuming it would re-activate a release the host has already moved past. ob status stops reporting it; ob audit still shows the run.

A manual ob rollback is journaled under the release it restores, so it does not settle an interrupted deploy that way. That deploy stays visible, and resuming it is refused on its superseded manifest before any workload is touched.

Two refusals have an override, and each grants exactly one thing.

migration_gate_closed — the interrupted release ran a job or hook whose data effects are rollback-unknown, so ob abort will not roll the application back on its own. Fix forward and ob resume, or, if you know the old code reads the current data correctly, take responsibility explicitly:

Terminal window
ob abort --break-migration-gate --output ndjson

A stale operation lock — a previous runner died holding it. Inspect the holder in ob status first; the lock records who took it and when. If it is genuinely abandoned:

Terminal window
ob abort --break-lock --output ndjson

--break-lock is on six commands: deploy, bootstrap, abort, job run, service apply and proxy apply. Not every mutating command has it — resume, destroy and secrets push do not. It breaks the application and host locks only. The protection lock has no override and clears on TTL expiry or when the same operation returns.

Recovery onto another host is a different workflow

Section titled “Recovery onto another host is a different workflow”

Rolling deployment can avoid interruption while the host is healthy. It cannot make a failed host available. Recovery onto another host is a distinct, evidence-backed workflow — not failover, and not something rollback reaches.

Terminal window
ob audit # human table of recorded operations
ob logs web # one workload or Onebox-run service
ob status --output json

The journal is append-only, so the record of a failed operation survives the operation that replaced it.