Offline dependency caches
Vendoring ansible collections and OpenTofu providers so a run does not depend on Galaxy or the tofu registry being reachable.
Disaster recovery must not depend on Ansible Galaxy or registry.opentofu.org being
reachable. Two caches, both inside
the 0700 data dir and both covered by the ordinary backup snapshot, make that true.
Ansible collections
A source's collections are vendored per source, mirroring the venv model
(Decision 0015): into
<data dir>/cache/collections/<source>/<requirements-hash>/, where <source>
is the same slug+hash key a source's uv venv uses and the requirements file is
.exocomp/requirements.yml, overriding a repo-root requirements.yml (the same
precedence as the Python manifest). A ready marker (.exocomp-galaxy-ready) in
the dir records that the resolve already happened.
The ansible driver resolves the source's collections at claim time, before the
play runs: a matching ready marker is a no-op with no Galaxy dial; a new hash
runs ansible-galaxy collection install -r <req> -p <dir> as a subprocess of
the claimed job, fail-closed on an unresolvable requirements file. The
daemonless inline drain path resolves the same way before enqueue. A playbook
needs no flag, no environment variable, and no network to use a vendored
collection; it only needs to have been vendored once.
exocomp playbook vendor installs, through the queue, the collections named
by its --requirements file into that file's project's hash-keyed per-source
dir:
exocomp playbook vendor --requirements requirements.yml
The shared <data dir>/cache/collections dir survives only as a fallback
for collections no source declares; the driver appends the per-source dir ahead
of it so a source's pin wins. Nothing writes the shared dir anymore, but it
stays readable so the offline guarantee never regresses for an existing
deployment.
Deleting a source removes its cache/collections/<source> (and
cache/python/<source>) — only when no other registered source resolves to the
same root, since a shared root means a shared key. A requirements-hash change
provisions a new dir and prunes that source's old hashes with the same keep-N +
since policy as venvs. The shared fallback is never purged by one source's
lifecycle.
community.sops is worth vendoring on every install: the design itself depends on it
(disaster-recovery.md's S3 credential story assumes sops-encrypted
files are decryptable offline, and ansible-galaxy collection install warns, harmlessly,
that a path outside its configured collections paths "will not be picked up" — that
warning describes running ansible-galaxy directly, not a playbook run through
exocomp, which sets the path itself).
Python dependencies
A source that declares Python dependencies (requirements.txt, or a
pyproject.toml with PEP 621 / [dependency-groups] deps) resolves into a
per-source uv venv at <data dir>/cache/python/<source>/<manifest-hash>
(Decision 0011). uv's own download
cache lands at <data dir>/cache/python/uv-cache, so both travel with the
backup.
The resolver runs under the job sandbox, not the daemon. uv resolves and
installs a project's declared deps, and a manifest can steer that — a
pyproject.toml build backend is arbitrary code, run at install time, derived
from an untrusted source. So when the sandbox tier is in effect the uv
invocation is wrapped in the same nono launcher every job runs under
(Decision 0005): uv is granted write on its
venv and cache, read on its tool tree and the manifest, and HTTPS to fetch
packages, and is denied the data dir, host.key, keyring.age, and the
database by the same profile guard a run's grants pass through. A resolve that
tries to read a denied path fails closed, leaving no half-built venv. A host
that cannot enforce the sandbox resolves in the daemon context, loudly, the same
way any job degrades there.
Index posture and no flag injection. A pyproject.toml's
[project].dependencies are PEP 508 package requirements — a package name comes
first, never a dash. A specifier that begins with - is therefore a pip option
smuggled into the dependency list (--index-url=… to redirect the resolver's
index, -e /path to install an editable tree), and the vendoring step refuses
it rather than pass it to uv as a flag; the specifiers it does
accept are handed to uv after a -- separator as defence in depth, so nothing
from the manifest is ever read as an option. A requirements.txt, by contrast,
is uv's own input format and its in-file options are the operator's declaration,
honoured as written — pin the index and hashes there when a private index or
hash-gating is wanted, and keep the source a trusted (ideally --signed) one.
OpenTofu providers
workspace plan/apply sets TF_PLUGIN_CACHE_DIR to <data dir>/tofu/plugin-cache, so
a provider a previous run already downloaded is not fetched again.
That cache alone does not make a run network-independent: OpenTofu queries the
registry's discovery document to resolve a provider even when a cached copy would
already satisfy the request, so a plain TF_PLUGIN_CACHE_DIR run still needs a route to
registry.opentofu.org. The driver (internal/executor/tofu.go) also supports an
OfflineProviders mode that restricts resolution to that cache directory alone, with no
registry method declared at all — one directory serves both roles, since tofu's
filesystem-mirror method reads the exact layout TF_PLUGIN_CACHE_DIR already writes.
A lockfile written online carries only zh: (registry) hashes, which the filesystem
mirror cannot satisfy with no network, so an offline init would fail verification against
it. In OfflineProviders mode the driver first runs tofu providers lock -fs-mirror <cache> -platform=<current>, which writes platform-correct h1: hashes into
.terraform.lock.hcl sourced from the mirror alone — the hashes the following offline
init can verify.
That mode is proven end to end (TestTofuPlanRunsOfflineFromPluginCache,
internal/executor/tofu_offline_test.go: one online apply, then a network-denied init and
plan from the same cache). The surface is exocomp workspace offline <name> on|off:
a workspace flipped on resolves every tofu run's providers from the mirror alone,
with no registry access, and a provider missing from the cache fails the run rather
than falling back to the network. Flip it once the cache is known warm — the
restore-drill plan above is exactly the offline run this posture forces.
Verifying offline resolution
Both proofs run for real, in a network-denied unshare -rn user+network namespace (no
sudo; user namespaces are the mechanism, verified on the target host):
go test ./internal/executor/... -run 'OfflineFromVendoredCollection|PlanRunsOfflineFromPluginCache' -v
Each test brings up whatever it needs (a real ansible-galaxy/age/sops fixture, or
exocomp's own HTTP state backend) inside the child namespace itself — a namespace's
loopback is its own, unreachable from any other namespace — and asserts a real network
dial to the registry it depends on fails before trusting the result.