I run a k3s cluster at home and treat it like production. Everything gets backed up. Almost none of it had ever been restored, which is the normal state of affairs everywhere I've ever worked. So I automated restore testing (not restores... testing) and gave the job to an AI operator. This is how it's scoped, because that's the part that made it a sane decision instead of a scary one.
What it does
The operator takes a backup, brings it back in an isolated throwaway namespace, confirms the data is real and consistent, writes a dated pass or fail, then deletes everything it created. Production is never touched. Real recovery stays a human call... I'm not automating disaster recovery, I'm automating the proof.
One caveat I'd rather volunteer than have someone catch: the scratch namespace lives inside the production cluster. The isolation is policy and permission, not hardware... a fence, not a moat. Fine for a test-restore that evaporates in minutes; not for anything heavier. More on where that's going below.
The permission model
Read-only by default. The operator can't restore anything until a small, scoped permission is granted, and that grant is mine to make and revoke. No standing credential either... short-lived tokens minted per job and expired minutes later.
The piece I'd point at for anyone doing this: I split the capability from the binding.
- The capability (the Role and its permissions) lives in version control. Declared, reviewable, dormant, with nothing bound to it. A Role with no binding grants nothing.
- The binding (the thing that switches the power on) is applied out-of-band, deliberately kept out of GitOps, so the reconciler can't reinstate a permission I revoked.
- Teardown verifies rather than assumes: it re-checks that the privilege is actually gone and fails loudly if it isn't.
A teardown that fires a delete and trusts it is how you end up with standing access you think you revoked.
What's working
The lanes that only ever read run unattended, start to finish. The etcd drill is the clean example: it pulls a snapshot from object storage with a key scoped read-only to exactly one bucket, restores it to scratch, verifies, tears down, weekly, with an alarm if it goes quiet. Nothing to gate, because there's almost nothing that key could do. More than thirty restores now, zero failures, most of the register covered. The rule that fell out of it: an agent gets a key only when the key is narrow enough that handing it over doesn't matter.
Where I'm stuck, and want you to poke holes
For the drills that have to build things (restore a Longhorn volume, stand up a scratch DB), the operator needs a real write grant, and today I apply that grant by hand before the run. That's the one manual step left, which means a "fully automated" weekly drill is really a weekly appointment. I don't want to hand-approve it forever, and I don't want to leave a standing grant sitting there either.
The design I'm circling: a separate, deterministic policy broker (not another AI making judgment calls) that approves a narrow, time-windowed elevation for a specific request and forces de-escalation inside a few minutes. What I've got so far:
- The request is structured, not prose: {which identity, which role, what window, which drill}. No model output touches the decision. The agent requests; it never authorizes itself.
- Deny-wins and fail-closed. Broker down or request malformed means the drill fails and alerts, never fails open.
- Two independent expiries: a TTL on the binding and a short-lived token, so either one failing alone still kills access.
- De-escalation is confirmed by checking whether the binding still exists, not by trusting the remover's exit code.
- The grant path itself is guarded by an admission policy, so an out-of-shape or never-expiring grant is inadmissible, not just logged after the fact.
- The broker's rules live in git behind my merge, so the agent lane can't edit its own approval rules.
- Break-glass and anything high-blast-radius stay human.
As far as I can tell this is just the NIST PDP/PEP model pointed at an agent, and the human-facing JIT-access tools already do exactly this. But I can't find anything that ships it agent-native for Kubernetes, which makes me wonder what I'm missing. The failure mode I already see: inside a legitimate window, a prompt-injected operator holds a legitimate grant and spends it with someone else's intent. Shape-constraining the grant shrinks that to "can only touch drill-labeled scratch objects," but that's a reduction, not a cure.
So: where does this bite me?
(Separate track, less interesting to this sub: the fence-not-a-moat problem gets solved by moving the build-type drills onto a genuinely separate cluster on an isolated network, so prod is actually absent rather than fenced off. That same rig is where I'll finally rehearse a full-cluster rebuild from nothing, which I can't yet prove.)