https://kubernetespodcast.com/episode/269-ai-usage/

Check out our latest episode: Navigating AI Policies in Kubernetes, with Kat Cosgrove and Natali Vlatko!

When people talk about AI in Kubernetes, the conversation is often focused on the user side, but for open source maintainers, there's of course a lot to unpack on how AI is changing oss contribution! How does AI impact the community, and what are the new rules of engagement?

We sat down with Kat Cosgrove (SIG Docs Technical Lead, SIG Release Subproject Lead, and Kubernetes Steering Committee member) and Natali Vlatko (SIG Docs Co-Chair, TODO Group Steering Committee member, and OSPO at Cisco) to unpack the recently published Kubernetes AI usage policy for contributors.

Topics covered include:
- Why you can't call an AI tool your "co-author" for legal reasons (because the AI tool can't sign the CLA)
- The rise of "AI slop" PRs and why AI-generated comments are causing trouble for OSS maintainers & reviewers.
- And of course, paths for new contributors to get involved sustainably!

Tune in for an insider's perspective into how AI is impacting contribution to one of the world's largest OSS projects!

I'm the founder of Slipway. Short version: nft list ruleset is not the whole truth on a box running Tailscale. Tailscale's NAT rules live in iptables-legacy, and it cost us two days of etcd TLS rejects to find that out.

Let me set some context. I am running an application on AKS cluster where I have a file in memory that i need to upload to Azure Blob Storage on two different paths.

So I am doing those uploads concurrently.

I am using go, and there are two ABS client created and the upload is done in 2 diff goroutines.

Now what we are seeing is that one upload happens in 1-2 minute, and the other one takes 20mins to finish.

This has occurred enough times that we cant call it intermittent, but still it doesn't happen always.

Upon getting the Blob storage logs, we found that the individual API requests are taking about a minute for the slow upload, while it is taking abt e seconds for the fast one. They confirmed it is a network throughput issue rather than a disk write issue.

We looked at the pod network io metrics ob grafana and saw a drop in output rate during the upload window of the 2nd file.

The drop was from 300KB/s to 30KB/s.

Can someone help me figure out what might cause this to happen?

NOTE: What is consistent in all the instances of this issue is that everytime, there is one file upload that is faster and one is slower.

The same code is doing the upload sith same configurations. The file is same. What is different is the dst file path and the network port.

We ran into this evaluating inference frameworks for a production LLM workload: clean benchmarks hid exactly the failure modes that matter at the infra layer.

Bursty concurrency and mixed prompt lengths exposed GPU memory pressure and latency spikes that never showed up under steady synthetic load.

We ended up building a testing process around soak tests and simulated failure scenarios (node loss, sudden load spikes) instead of trusting a single throughput number. Wrote up what we found and the process we now use before committing to a framework.

When VAP came out I was really excited about it, however operationalizing it on existing clusters looks to be non-trivial and I'm curious how people are managing it.

Specifically, on existing clusters I'm thinking of how to determine the impact of new VAP policies on existing workloads before you roll them out. You could roll them out in Warn mode but users only see the issue on admission so running workloads remain blissfully unaffected by a newly introduced policy.

You could use something like Kyverno to manage them in Warn/Inform and run them as background processes so you get all the info of which workloads would violate the new VAP policies. This is a really nice feature in Kyverno but if you are not using Kyverno already it's another on-cluster tool to deal with.

Are there any existing tools out there that will pre-validate VAP before it gets deployed, ideally off-cluster, or has a way of tracking VAP results that doesn't involve trolling through the audit logs?

Same boat as half of this sub I think. When the cutover hit last August we repointed Postgres, Redis, Mongo and Rabbit at bitnamilegacy as a stopgap and told ourselves we'd sort it properly later. It's July and later never came.

The legacy images don't get patched though and Bitnami's own line is that repo isn't meant to stick around, meaning we're sitting on unpatched infra that could also just vanish on us. Redis is a whole separate headache now with Valkey being the fork.

Small team, I really don't want to end up owning ten hand-rolled operator setups by myself. is it just service by service or how did y'all get all the way through this and land somewhere decent.

One of my engineers (im the CEO of Portainer) asked if I had the time to help him better understand Kubernetes architecure and components… and so rather than just explain it once, I figured why not share it wider.

So, https://kubeschool.portainer.io was born :-)

I run a k3s cluster at home and treat it like production. Everything gets backed up. Almost none of it had ever been restored, which is the normal state of affairs everywhere I've ever worked. So I automated restore testing (not restores... testing) and gave the job to an AI operator. This is how it's scoped, because that's the part that made it a sane decision instead of a scary one.

What it does

The operator takes a backup, brings it back in an isolated throwaway namespace, confirms the data is real and consistent, writes a dated pass or fail, then deletes everything it created. Production is never touched. Real recovery stays a human call... I'm not automating disaster recovery, I'm automating the proof.

One caveat I'd rather volunteer than have someone catch: the scratch namespace lives inside the production cluster. The isolation is policy and permission, not hardware... a fence, not a moat. Fine for a test-restore that evaporates in minutes; not for anything heavier. More on where that's going below.

The permission model

Read-only by default. The operator can't restore anything until a small, scoped permission is granted, and that grant is mine to make and revoke. No standing credential either... short-lived tokens minted per job and expired minutes later.

The piece I'd point at for anyone doing this: I split the capability from the binding.

• The capability (the Role and its permissions) lives in version control. Declared, reviewable, dormant, with nothing bound to it. A Role with no binding grants nothing.
• The binding (the thing that switches the power on) is applied out-of-band, deliberately kept out of GitOps, so the reconciler can't reinstate a permission I revoked.
• Teardown verifies rather than assumes: it re-checks that the privilege is actually gone and fails loudly if it isn't.

A teardown that fires a delete and trusts it is how you end up with standing access you think you revoked.

What's working

The lanes that only ever read run unattended, start to finish. The etcd drill is the clean example: it pulls a snapshot from object storage with a key scoped read-only to exactly one bucket, restores it to scratch, verifies, tears down, weekly, with an alarm if it goes quiet. Nothing to gate, because there's almost nothing that key could do. More than thirty restores now, zero failures, most of the register covered. The rule that fell out of it: an agent gets a key only when the key is narrow enough that handing it over doesn't matter.

Where I'm stuck, and want you to poke holes

For the drills that have to build things (restore a Longhorn volume, stand up a scratch DB), the operator needs a real write grant, and today I apply that grant by hand before the run. That's the one manual step left, which means a "fully automated" weekly drill is really a weekly appointment. I don't want to hand-approve it forever, and I don't want to leave a standing grant sitting there either.

The design I'm circling: a separate, deterministic policy broker (not another AI making judgment calls) that approves a narrow, time-windowed elevation for a specific request and forces de-escalation inside a few minutes. What I've got so far:

• The request is structured, not prose: {which identity, which role, what window, which drill}. No model output touches the decision. The agent requests; it never authorizes itself.
• Deny-wins and fail-closed. Broker down or request malformed means the drill fails and alerts, never fails open.
• Two independent expiries: a TTL on the binding and a short-lived token, so either one failing alone still kills access.
• De-escalation is confirmed by checking whether the binding still exists, not by trusting the remover's exit code.
• The grant path itself is guarded by an admission policy, so an out-of-shape or never-expiring grant is inadmissible, not just logged after the fact.
• The broker's rules live in git behind my merge, so the agent lane can't edit its own approval rules.
• Break-glass and anything high-blast-radius stay human.

As far as I can tell this is just the NIST PDP/PEP model pointed at an agent, and the human-facing JIT-access tools already do exactly this. But I can't find anything that ships it agent-native for Kubernetes, which makes me wonder what I'm missing. The failure mode I already see: inside a legitimate window, a prompt-injected operator holds a legitimate grant and spends it with someone else's intent. Shape-constraining the grant shrinks that to "can only touch drill-labeled scratch objects," but that's a reduction, not a cure.

So: where does this bite me?

(Separate track, less interesting to this sub: the fence-not-a-moat problem gets solved by moving the build-type drills onto a genuinely separate cluster on an isolated network, so prod is actually absent rather than fenced off. That same rig is where I'll finally rehearse a full-cluster rebuild from nothing, which I can't yet prove.)

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

almost vendor call i see starts with "well, technically we're a CNAPP, but really it covers CSPM too" and so we all know that half the room checks out.

so here is Quick breakdown: CSPM is config checks. like Open buckets, missing encryption, overly permissive rules..... Static, infrastructure-focused. and so CNAPP is CSPM plus CWPP plus CIEM plus container scanning rolled into one. we know that gartner shifted a few years ago to treat CSPM as a component inside CNAPP...and not a separate category.

so imo The real difference day-to-day isn't feature count..but the correlation. i mean CSPM gives you three separate alerts...like vulnerable container, internet-exposed resource, admin role. and the CNAPP tells you it's the same attack path. That's where the alert fatigue drop comes from.

The costly trap i think is buying standalone CSPM in 2026 means buying a shrinking category, then stitching on separate CWPP, CIEM, and scanning tools. More vendors, might mean more integration work...more gaps where context doesn't carry across.

but the one question that cuts through is that ... Does this correlate findings across config, identity, and workload into a single attack path, or am I going to be piecing together three dashboards?

Every new service starts with another Dockerfile. A year later, every

repo builds containers differently. Keeping them secure and consistent

becomes a platform team nightmare.

Cloud Native Buildpacks solve this — centralized builds, SBOM

generation by default, OS patches without full rebuilds. No Dockerfile

per service.

Short explainer on how they work under the hood.

Are you using Buildpacks in production? If not — what's stopping you?

At my recent talk, I took down an entire region mid-demo while an app was actively writing to the database. Then I promoted a replica cluster in another region using mariadb-operator and cut traffic over.

Regional outage? Survived. ✅

The talk shows how to run MariaDB as a true first-class citizen on Kubernetes, covering the full lifecycle of the database, multiple topologies and day 2 operations. The epic finale 🍿 is the fun part: a multi-cluster, multi-region demo with a live failover across three KIND clusters, all driven by the operator:

• eu-south: the primary region
• eu-central: bootstraps from a physical backup taken on the primary, and joins as a replica cluster
• envoy: load balancing client traffic to the active region leveraging Envoy Gateway and the Gateway API

I run an app that continuously writes to the database, then take down the entire primary region, promote the replica cluster with a single kubectl patch, and flip the Envoy backend weights to cut traffic over. The app keeps running and resumes writing against the new region once the cutover completes, seamlessly and cloud-natively.

Want to try it yourself? The full demo and step-by-step instructions are ready to go:

🛠️ Demo README: Check the architecture diagram and run it locally in your machine.

📺 Watch the talk

🦭 mariadb-operator on GitHub: ⭐ to support the project would be greatly appreciated!

I'm the maintainer, happy to share additional details about the new multi-cluster topology!

Im new to tech and recently started learning operators and controllers. With AI writing every code, which is faster than human, is it rational for me to learn about the kubernetes now ?

If you are in NYC and available on Wednesday evening, come hear guest speaker Noam Levy talk end-to-end testing with Kubernetes and networking magic. 🪄

➡️ If you can make it, please RSVP at https://luma.com/u4vja8rx

See you soon!

Hi everyone, let's go.

I am migrating from Beanstalk to EKS, and I wanted to clear up some doubts.

Which components/tools do you consider essential in a cluster? Currently, the cluster I am developing has:

• Karpenter (x86/ARM)
• Kyverno
• KEDA
• ESO (Fetches secrets from Parameter Store)
• Kong API Gateway (used as Ingress)
• ArgoCD for GitOps

What do you recommend? The CNI is the default VPC CNI, is it worth switching to Cilium or Calico? Bearing in mind that this will be my first experience building a cluster—I've managed them before at other companies, but I don't have much experience creating one from scratch.

During a Kubernetes incident (say users start getting 503s), what's your typical debugging workflow?

Mine usually looks something like:

Ingress → Service → Endpoints → Pods → Readiness → Events → Logs

I'm curious whether experienced platform engineers and SREs follow a similar flow or if there are better heuristics to narrow down the root cause faster.

What are the first 3–5 things you check, and what part of the process do you find the most repetitive?

I was watching this video from r/kodekloud on YouTube

and something the speaker said confused me.

He said: "Most people know how to deploy pods, but almost no one really knows how to secure them."

I work in cloud infrastructure, and in my experience, most customers put their entire cluster behind the cloud provider's security services (such as built-in security scanning) and a Palo Alto firewall.

So I'm trying to understand what he means. If those security controls are already in place, what is still considered "securing the pods"? What am I missing? Are there Kubernetes-specific security practices that these tools don't cover?

Inherited a fleet of EKS clusters and the utilization numbers are almost funny. Cluster wide CPU request versus actual usage is sitting around 25 percent. We're paying for a few node groups worth of headroom that's not being used, because every team padded their requests "to be safe" a year ago and never looked at them again.

The obvious move is to right-size the requests and let the bin-packer do its job. But the second I touch a team's limits I own any latency blip for the next quarter, so nobody wants to be the one who does it.

I've looked at VPA and a couple of the cost tools and half of them just show me the same graph I already have telling me we're overprovisioned. I know we're overprovisioned. Meanwhile the bill sits there being a third bigger than it needs to be, everyone agrees it's a problem in the abstract but no one wants to own the fix.

A simple walkthrough on how to use KAI Scheduler for topology-aware placement and gang scheduling to keep distributed GPU training inside one region, zone, or rack on Kubernetes

https://hrittikhere.com/posts/kubernetes-topology-aware-scheduling-kai

Looking for feedback from folks running apps that require mTLS that use API Gateway.

What controller are you using? Traefik, Envoy, etc.

Any pain points or gotchas you might have ran into?

Any other feedback or recommendations welcome!