Draining for a planned reboot

Cordoning a running daemon so it fires no schedules and claims no new jobs while in-flight work finishes, then polling whether it is safe to reboot.

A kernel update or hardware maintenance means rebooting the control node. exocomp serve already drains on shutdown — it stops the scheduler, gives in-flight jobs a grace window, then cancels whatever is left (Running the daemon). But that drain is a side-effect of the process exiting, and the grace window cancels work it does not like. For a planned reboot you want the opposite: cordon the running daemon ahead of time, let in-flight work finish on its own clock, and be told when it is safe to reboot.

exocomp drain is that action. It is one operator on one node (invariant 1); there is no failover partner and none is needed — this is planned maintenance, not a crash.

What a cordon does

A cordoned daemon:

• fires no new schedules — the scheduler is paused, not torn down. A cron occurrence that passes while cordoned is not lost: because a suppressed fire never stamps last_run, lifting the cordon runs the same catch-up a daemon does after downtime, so the missed occurrence fires once (coalesced), never once per missed tick.
• claims no new jobs — a row that reaches the queue while cordoned stays queued. It is claimed once the cordon lifts, or swept to interrupted on the next boot if the reboot happens first.
• lets in-flight jobs finish — a job already running is never cancelled by a cordon. This is the key difference from shutdown's grace window. A backup leg writing its backup_run row, a tofu apply pushing state — all run to their own completion.

"Safe to reboot" is the conjunction the CLI computes for you: cordoned AND nothing still in flight. A queued-but-unclaimed job does not block a reboot (it is swept to interrupted and re-drivable); a running job does.

Driving it

drain controls a running daemon, so it goes over the operator API, the same channel --remote uses for every other write (Driving a remote daemon). It needs a selected remote — the control node targets its own daemon:

exocomp --remote self drain
cordoned: yes  in-flight: 2  safe to reboot: not yet

Cordon and wait for the two in-flight jobs to finish, then reboot:

exocomp --remote self drain --wait
waiting: 2 job(s) in flight
waiting: 1 job(s) in flight
cordoned: yes  in-flight: 0  safe to reboot: yes

--wait polls until safe or --timeout (default 10m) elapses; a timeout exits non-zero, so a reboot script does not proceed on a daemon still finishing work. --status reports readiness without cordoning. Lift the cordon (a reboot postponed, say) with --resume:

If the goal is to restart the daemon itself rather than reboot the host, exocomp restart does the same drain and then re-execs the on-disk binary in place, so no reboot and no systemd restart is involved.

exocomp --remote self drain --resume
cordon lifted; the daemon is taking work again

Every verb honours --output json; the readiness envelope is maintenance.readiness, and a script branches on its safe_to_reboot field:

exocomp --remote self drain --output json
{"schema_version":1,"kind":"maintenance.readiness","data":{"cordoned":true,"in_flight":0,"safe_to_reboot":true}}

A cordon is in-memory daemon state: a reboot clears it, so the daemon comes back up taking work normally. There is nothing to un-set after the reboot.

After the reboot: interrupted jobs

A reboot cuts off whatever was still running, and sweeps whatever was still queued. Both land in interrupted — a terminal state that is never auto-retried (the sweep). List them to see what to re-drive:

exocomp --remote self drain interrupted
ID  STATE        RUNBOOK   KIND    TRIGGER   EXIT  QUEUED
41  interrupted  nightly   backup  schedule  -     2026-08-22 03:00:00Z

drain interrupted reads them newest-first. It works locally too (reading the store directly), for the common case of an operator on the freshly-rebooted box. Re-drive the ones that still matter with exocomp jobs redrive (dry-run first, then --yes); nothing re-runs on its own. See Recovering jobs a restart cut off.

The mechanism, and why not a signal

SIGHUP reloads schedules and SIGTERM drains-and-stops, but a signal cannot carry the in-flight count back, so it cannot answer "safe yet?". The cordon is therefore an operator-scoped API action on the Decision 0009 listener, gated exactly like plan/apply: a read-only token is refused, and a browser session may not drive it. It is deliberately not a queue job — a cordon is control-plane state, not a run, so it has no ledger entry.