Roll back a release
Three commands, three different situations. Choosing the wrong one is the usual way an incident gets worse.
| Situation | Command | What it does |
|---|---|---|
| A deploy was interrupted and you want to finish it | ob resume | Continues from durable phase state |
| A deploy was interrupted and you want it undone | ob abort | Reverts the interrupted deploy |
| A deploy completed but the release is bad | ob rollback | Activates the previous release |
Start by asking the server, not your memory:
ob status --output jsonob status exits non-zero on divergence, so it is safe to use as a gate in a
script.
Rollback activates the previous release
Section titled “Rollback activates the previous release”ob rollbackEvery release has a strict manifest state and an explicit predecessor. Rollback
activates the current serving release’s eligible predecessor, verifies it, and
records the transition; it does not select a directory by timestamp or name.
deployment.retain_releases (default 5) bounds ordinary history while always
protecting current, predecessor-chain, and checkpoint-referenced releases.
What rollback does not do
Section titled “What rollback does not do”Concretely:
- Supporting services are not rolled back. A service runs in its own Compose project and outlives every release, so no deploy and no rollback stops it or removes its volume. That protects your data; it also means a schema migration is still applied after you roll the application back.
- A job with
data_effect: migrationordestructiveconstrains the gate. That is what the field is for. - A job whose result evidence is missing becomes
changed=unknownand halts before workload replacement rather than proceeding on an assumption.
Resume and abort work from durable state
Section titled “Resume and abort work from durable state”Both read the append-only journal, strict release manifests, and the durable activation checkpoint. Recovery removes only containers owned by the exact interrupted release, restores and verifies its predecessor, and clears the checkpoint last. A failed postcondition leaves the checkpoint retryable instead of hiding divergence.
ob resume --output ndjsonob abort --output ndjsonActivation splits the two halves of a deploy. Interrupted before it, the release
never took effect and ob resume replays the phases that remain. Interrupted
after it — retention, schedule sync or the post_deploy hook failed while the
new release was already current and healthy — there is nothing to replay:
ob resume completes only those remaining steps, skipping any the journal
already records as done, and the release keeps serving throughout. The manifest
carries a failed operation outcome in the meantime, so a healthy release and an
unfinished operation stay distinguishable.
ob resume re-verifies the serving release first, then finalizes only when the
recorded activation, the current release, the serving manifest and its recorded
predecessor, the absence of an open activation checkpoint, and the live workload
labels all agree. One disagreement returns finalize_refused: no step runs and
no release, symlink or workload changes.
Only the newest deploy is recoverable. Once a later deploy reaches a terminal
state of its own — finished, aborted, or automatically rolled back — the older
interrupted one is history rather than pending work, and resuming it would
re-activate a release the host has already moved past. ob status stops
reporting it; ob audit still shows the run.
A manual ob rollback is journaled under the release it restores, so it does not
settle an interrupted deploy that way. That deploy stays visible, and resuming it
is refused on its superseded manifest before any workload is touched.
When recovery refuses
Section titled “When recovery refuses”Two refusals have an override, and each grants exactly one thing.
migration_gate_closed — the interrupted release ran a job or hook whose
data effects are rollback-unknown, so ob abort will not roll the application
back on its own. Fix forward and ob resume, or, if you know the old code reads
the current data correctly, take responsibility explicitly:
ob abort --break-migration-gate --output ndjsonA stale operation lock — a previous runner died holding it. Inspect the
holder in ob status first; the lock records who took it and when. If it is
genuinely abandoned:
ob abort --break-lock --output ndjson--break-lock is on six commands: deploy, bootstrap, abort, job run,
service apply and proxy apply. Not every mutating command has it — resume,
destroy and secrets push do not. It breaks the application and host locks
only. The protection lock has no override and clears on TTL expiry or when the
same operation returns.
Recovery onto another host is a different workflow
Section titled “Recovery onto another host is a different workflow”Rolling deployment can avoid interruption while the host is healthy. It cannot make a failed host available. Recovery onto another host is a distinct, evidence-backed workflow — not failover, and not something rollback reaches.
Inspecting what happened
Section titled “Inspecting what happened”ob audit # human table of recorded operationsob logs web # one workload or Onebox-run serviceob status --output jsonThe journal is append-only, so the record of a failed operation survives the operation that replaced it.