Schedule a job

workloads:
  nightly-dump:
    role: Job
    image: postgres:17
    command: ["sh", "-c", "pg_dump \"$POSTGRES_URL\" | gzip > /backups/$(date -u +%F).sql.gz"]
    dataEffect: None
    needs: [postgres]
    volumes: [{name: backups, path: /backups}]
    schedule:
      cron: "0 2 * * *"
      timezone: Europe/Berlin
      timeout: 45m
      catchUp: true

It becomes a host timer

A schedule is translated into a timer on the host. It fires without any Onebox process running and survives a reboot. There is no daemon, no listening port, and nothing to keep alive.

That design decision has a visible consequence elsewhere: encrypted environment entries are decrypted into the release when it is staged and stay there, because a timer firing at 02:00 must resolve the values the deploy resolved.

Each run is a systemd oneshot with a wall-time limit. timeout defaults to 1h; when it expires, systemd terminates the run and records timeout as its result. Set a longer duration for jobs that legitimately need it.

catchUp defaults to true: if the host was off at the scheduled time, the timer runs once after it returns. Set it to false for time-sensitive work that should be skipped rather than run late.

Retry inside one firing

By default a failed run is not retried; the next attempt is the next cron elapse, which is the right answer for anything that fires every few minutes. For a job whose failures are usually transient, a rate limit or an upstream that was busy for a moment, declare a bounded retry:

schedule:
  cron: "0 * * * *"
  timeout: 45m
  retry: {attempts: 3, backoff: 30s, maxBackoff: 10m}

attempts counts every attempt including the first, so the default of 1 is today’s single run. After a non-zero exit the runner sleeps backoff, doubles it after each further failure, and never sleeps longer than maxBackoff. Every attempt runs inside the same timer firing, under the same locks and the same timeout; when the timeout expires the run ends, with no further attempt.

Validation refuses a retry whose worst-case backoff is not smaller than the timeout, because the last attempt could never start and the record would say it did. With deployLock: Exclusive the deploy lock is held through the sleeps, so a deploy waits for the run to finish; with deployLock: Pinned the release lease is held instead and every attempt runs the same release.

Deployment coordination is exclusive by default

An exclusive host-fired job takes an application-wide kernel lock for its whole container run. A pinned job takes that rendezvous as a shared reader until it establishes its immutable release lease. Different pinned jobs may still start together. Every operation that establishes Onebox’s fenced application lock takes the rendezvous as an exclusive writer while publishing its ownership. Readers and writers wait briefly for a handoff; the wait is shortened for a job whose own timeout is near the ten-second default. A timer that still collides does not modify Docker beside a deploy: it records a skipped run with the reason and exits cleanly. An application lock older than its TTL is treated as expired, exactly as a deploy treats it, so a runner that died mid-operation cannot silence a timer forever.

The target must provide flock (part of util-linux on supported Linux hosts). Onebox refuses to install or run schedules when that serialization primitive is missing.

For a long-running job that does not change shared data, opt into a pinned release instead:

dataEffect: None
schedule:
  cron: "0 2 * * *"
  timeout: 6h
  deployLock: Pinned

The runner briefly meets deployment under the same scheduling mutex, resolves current to one immutable release, and takes a shared lease on that release. It then releases the application-wide mutex and keeps only a per-job lock, so a data-effect-free deployment can proceed but a second instance of the same job cannot overlap. Deployments containing migration, destructive, or unknown-effect jobs, and deployments containing untyped lifecycle hooks, wait for the pinned job to finish. Other application operations also remain exclusive. Release cleanup preserves the leased Compose document, environment files, and release-bound mounts until the job exits. ob status reports the policy, the pinned release, start time, and timeout while it runs.

This is deliberately fail-closed. deployLock: Pinned is accepted only for a Onebox-rendered job declaring dataEffect: None. Migration, destructive, unknown-effect, and adopted-Compose jobs remain exclusive because pinning their files cannot prove that a concurrent deployment is safe for the data or external files they use. Destroy and every other non-deploy application operation also refuse while any pinned release is leased.

Reconcile after upgrading Onebox

The ob binary is agentless: upgrading it on the operator workstation does not silently connect to or mutate a target. Existing timers therefore keep the unit content written by the previous runner until the next deploy or an explicit schedule apply:

ob schedule apply

This rewrites every declared timer, service, runner, and failure notifier from the current contract, and removes units for jobs no longer declared. It runs under the same application lock, schedule mutex, fence, and journal boundary as a deploy, but it does not stage or activate a release. Run it after upgrading Onebox when an application may not be deployed again soon.

Run records are written by the unit, so a host whose units predate them has none until this command or a deploy rewrites them. Until then ob status falls back to what systemd retained, reports a failure as it always did, and says that no run record exists yet.

Every run leaves a record

When a run ends, for any reason, the unit’s ExecStopPost writes one record to the host journal with the syslog identifier onebox-run and the job’s unit in its ONEBOX_UNIT field, so journalctl SYSLOG_IDENTIFIER=onebox-run ONEBOX_UNIT=onebox-job-<job> -o cat on the host is the raw history:

{"run":"a3f9…","job":"nightly-dump","trigger":"timer","operation":"","release":"20260905-140000-ab12cd","started_at":"2026-09-05T02:00:01Z","finished_at":"2026-09-05T02:04:37Z","duration_s":276,"attempts":2,"exit_status":0,"outcome":"success","reason":"","inputs":{}}

run is systemd’s invocation id, so the record and the run’s own log share a key. outcome is one of success, failure, timeout, or skipped. A skip is a firing that met a running instance of the same job, timed out waiting for the application scheduling rendezvous, or found an application operation holding the deploy lock. The runner writes the reason and exits before any container starts, so the unit is not failed and the job’s own exit status is never mistaken for a skip. Skips are recorded because a job that is silently never running looks exactly like one that works: one skip is timing, and three in a row are reported by ob status as an issue.

A skip is news about timing, not about the job, so it never clears a failure: ob status keeps reporting the newest run that actually happened, and says that nothing has run since. A job that selects skipped in notify is told its run did not happen, not that it failed.

Read the records back from the workstation:

ob schedule list                      # every job, its timer state and next elapse
ob job history nightly-dump      # records, newest first; -n 50 for more
ob job logs nightly-dump         # the journal of the newest run
ob job logs nightly-dump --run a3f9…

ob status reads the same records. Each scheduled job’s line carries the next elapse, the last outcome with its duration and attempt count, and, while a run is in progress, the attempt it is on. A last outcome of failure or timeout is reported as divergence:

schedule nightly-dump last run failed: timeout (exit 143) ⚠

Retention is the journal’s. A host whose journal lives in memory keeps records only since its last boot; ob status says so on the job’s line, and the fix is a persistent journal (/var/log/journal), which the supported images have by default.

Notifications per outcome

schedule.notify selects which outcomes send the configured notifications. The default is [Failure, Timeout], which is what the failure notifier always did. Add Success for a job whose completion someone waits for, and Skipped for a job whose firings collide often enough that silence would hide it:

schedule:
  cron: "0 2 * * *"
  notify: [Failure, Timeout, Skipped]

Each selected outcome sends a bounded, fail-open POST to every webhook whose own on list accepts that class of outcome. The payload carries the run id as its deploy_id, so ob job logs <job> --run <id> finds the run; it carries nothing else about the run, because notifications cross the host trust boundary and diagnostics stay on the host. Onebox writes the webhook handler root-only beside the unit, so credentials in webhook paths do not appear in ExecStart or ExecStopPost. A delivery failure is recorded in the unit’s journal and never replaces the job’s own result. The webhook must be reachable from the target host; timer delivery uses its network path, not the operator workstation’s.

Cron is translated exactly, or refused

0 2 * * *      ✓  every day at 02:00
0 2 * * 1      ✓  every Monday at 02:00
0 2 1 * 1      ✗  schedule_untranslatable

The last one declares a day-of-month and a day-of-week. Cron treats that as “either matches” — it fires on the 1st and on every Monday. Onebox refuses it at load rather than running on days nobody chose.

timezone takes an IANA zone name and defaults to UTC. Generated timers use one-second accuracy so systemd does not coalesce distinct cron minutes into its default one-minute wake-up window.

Jobs still declare a data effect

dataEffect is required on every job, scheduled or not. A nightly report is None; a nightly prune is Destructive. The rollback and abort gates read it, and a job that lies about it defeats them.

Stopping a job without deleting it

ob schedule pause nightly-dump --reason "upstream is returning garbage"
ob schedule resume nightly-dump

Pausing stops the timer and nothing else. The units stay installed and later deploys keep updating them, so a fix still lands on the host while the job stays stopped. A run already under way is left alone: this stops the next firing, it does not kill work in progress.

The pause is recorded on the host, not in the project, which is what lets it survive a deploy. Reconciliation will not start a paused timer, so a pause cannot be undone by someone else’s release. --reason is required, and is kept with the operator and the time, because a job that is deliberately not running is indistinguishable from one that is broken:

schedule nightly-dump PAUSED — by ops@example.com; since 2026-09-06T10:00:00Z; upstream is returning garbage

ob status prints that line whether or not anything else is wrong, and does not count the stopped timer as divergence. It does still report everything else about the job: a unit that will not load, or a run that failed, is printed beside the pause and still fails the command, because a pause explains a stopped timer and nothing more. A stopped timer with no pause behind it is also still a fault. ob schedule list shows a paused job’s timer as paused rather than inactive, so the table does not read the same for a job somebody stopped and one that broke.

Both ob schedule pause and ob schedule resume are journaled and appear in ob audit with the operator. Neither will record a change that did not happen: pausing an already-paused job is refused rather than overwriting who stopped it and why, and resuming a job nobody paused is refused rather than logged. Both take --break-lock, because a running job holds the application lock for its whole run and a crashed deploy leaves one behind.

Resuming starts the timer again; the next run is the next scheduled elapse. It does not run the job now, and does not make up firings missed while paused — ob job run is how you ask for an immediate run.

Deleting a job from the project clears its pause along with its units, so a name reused later does not come back stopped for a reason from a previous life.

Run one now with inputs

A scheduled job may declare inputs: named parameters that reach the container as environment variables. A timer firing uses the defaults; an operator may run the job now and override them.

workloads:
  source-sync:
    role: Job
    image: ghcr.io/acme/ingest:1.8.2
    command: ["./ingest", "sync"]
    dataEffect: None
    inputs:
      SOURCE:
        enum: [catalog, prices, reviews]
        default: catalog
        description: Which upstream to sync.
      SINCE:
        pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})?$'
        default: ""
        description: Only records changed since this date. Empty means the stored cursor.
    schedule:
      cron: "0 * * * *"
      deployLock: Pinned

ob job run source-sync --input SOURCE=prices --input SINCE=2026-09-01

Names are upper-case identifiers outside Onebox’s ONEBOX_ namespace and may not repeat an env key. Each input declares exactly one of enum or pattern, and a default that satisfies it; a pattern matches the whole value. Whatever the pattern allows, a value may not contain a double quote, a backslash, or a control character, and is at most 256 bytes. Those rules are what let the runner hand values to the container without escaping anything.

ob job run validates every value on the workstation, refuses if the unit is already running or an operator run is still pending, journals the request as schedule_run with the operator and the inputs, and starts the unit. It holds the application lock only while writing and journaling, because the runner skips a run that meets that lock. The host record of the run carries the operation id, so ob audit and ob job history join on it. The command follows the host-supervised unit by default; --detach returns once the unit accepts the run.

Only a job with dataEffect: None accepts inputs. Every operator invocation uses a sealed plan; migration jobs additionally retain their approval and backup-report gates and remain attached so Onebox can capture result evidence.

The runner tells an operator activation from a timer firing by the TRIGGER_UNIT variable systemd sets on timer activations, which that project introduced in version 252. Ubuntu 24.04 and Debian 12 qualify; Ubuntu 22.04 and Debian 11 do not.

Scheduled jobs still run on those older hosts, unchanged. Two things narrow:

• A job that declares inputs is refused, by ob preflight and by ob deploy, before anything is staged. Without TRIGGER_UNIT the next timer firing would read the file meant for an operator’s run.
• ob job run is refused for the same reason. The timer keeps firing.

The records on such a host say trigger: unknown, because the runner cannot observe what started it and will not guess.

Keep application workflows in the application

Onebox runs the job’s container command. A retry starts that command again; it does not resume from a completed step. A reboot does not resume interrupted work. Make repeated attempts safe in the application and store any progress there. A command can perform several operations internally; the application owns their ordering, checkpoints, and outputs.

Running one by hand

ob job plan nightly-dump --out ob-job-plan.json
ob approve --plan ob-job-plan.json --out ob-job-approval.json
ob job run --plan ob-job-plan.json --approval ob-job-approval.json

When the operator job also declares schedule, Onebox submits this run to the same installed systemd unit used by its timer. The host therefore owns the container, timeout, overlap lock, retries, and run record if the SSH session or operator terminal disappears. The command follows the unit by default and shows elapsed time. Ctrl-C stops following but does not stop the host job; use --detach to return immediately after the unit accepts it. Inspect either form with ob job history <job> and ob job logs <job>.

Jobs without schedule keep the direct foreground path. Migration jobs also stay attached because their result evidence is part of the approved operation.

ob job plan accepts a job when operatorRun is Allowed. This property is independent of deployment: deploymentPhase decides whether the job runs in the deploy graph, while operatorRun decides whether an operator may invoke it. By default, phase None allows operator runs; release-phase jobs disable them unless explicitly enabled.

A declared deploymentPhase: None job remains in the digest-pinned release runtime but never joins the deploy graph. Its job plan binds the current serving release, runtime digest, immutable image, data effect, target, and expiry. Automation supplies the saved plan and its separately recorded local confirmation; migration jobs may also need the exact plan-bound backup report.

For an undeclared emergency command, the escape hatch remains:

ob exec --reason "incident investigation" <workload|service> -- <command>

ob exec is an audited escape hatch

Nothing it changes belongs to a release and Onebox cannot describe, roll back, or make its effect idempotent. The journal stores the reason, target, operator, outcome, and command digest, but never the command bytes or passthrough output. The reason is durable metadata, so do not put a secret in it.

For a job that participates automatically in a deploy, choose a release phase:

deploymentPhase: PreRelease
operatorRun: Allowed            # optional: also permit ob job plan/run

The three properties are independent. deploymentPhase controls deploy hooks, schedule controls timer activation, and operatorRun controls explicit invocation. A job declaring schedule: and deploymentPhase: PreRelease runs both at its cron time and on every deploy. For a timer-only job, leave deploymentPhase at None; set operatorRun: Disabled too if operators must not start it explicitly.

See workloads for every field.