I am a pentester, mostly web apps and internal networks. I'm currently learning Kubernetes, and one thing keeps bothering me. In K8s, I see scanning everywhere through kube-bench, Trivy, Falco, and admission policies; however, nothing about actually testing these guardrails.

How are people validating that these actually fire? Is it through manual attempts, or is there tooling I've missed?

https://kubernetespodcast.com/episode/269-ai-usage/

Check out our latest episode: Navigating AI Policies in Kubernetes, with Kat Cosgrove and Natali Vlatko!

When people talk about AI in Kubernetes, the conversation is often focused on the user side, but for open source maintainers, there's of course a lot to unpack on how AI is changing oss contribution! How does AI impact the community, and what are the new rules of engagement?

We sat down with Kat Cosgrove (SIG Docs Technical Lead, SIG Release Subproject Lead, and Kubernetes Steering Committee member) and Natali Vlatko (SIG Docs Co-Chair, TODO Group Steering Committee member, and OSPO at Cisco) to unpack the recently published Kubernetes AI usage policy for contributors.

Topics covered include:
- Why you can't call an AI tool your "co-author" for legal reasons (because the AI tool can't sign the CLA)
- The rise of "AI slop" PRs and why AI-generated comments are causing trouble for OSS maintainers & reviewers.
- And of course, paths for new contributors to get involved sustainably!

Tune in for an insider's perspective into how AI is impacting contribution to one of the world's largest OSS projects!

Let me set some context. I am running an application on AKS cluster where I have a file in memory that i need to upload to Azure Blob Storage on two different paths.

So I am doing those uploads concurrently.

I am using go, and there are two ABS client created and the upload is done in 2 diff goroutines.

Now what we are seeing is that one upload happens in 1-2 minute, and the other one takes 20mins to finish.

This has occurred enough times that we cant call it intermittent, but still it doesn't happen always.

Upon getting the Blob storage logs, we found that the individual API requests are taking about a minute for the slow upload, while it is taking abt e seconds for the fast one. They confirmed it is a network throughput issue rather than a disk write issue.

We looked at the pod network io metrics ob grafana and saw a drop in output rate during the upload window of the 2nd file.

The drop was from 300KB/s to 30KB/s.

Can someone help me figure out what might cause this to happen?

NOTE: What is consistent in all the instances of this issue is that everytime, there is one file upload that is faster and one is slower.

The same code is doing the upload sith same configurations. The file is same. What is different is the dst file path and the network port.

We ran into this evaluating inference frameworks for a production LLM workload: clean benchmarks hid exactly the failure modes that matter at the infra layer.

Bursty concurrency and mixed prompt lengths exposed GPU memory pressure and latency spikes that never showed up under steady synthetic load.

We ended up building a testing process around soak tests and simulated failure scenarios (node loss, sudden load spikes) instead of trusting a single throughput number. Wrote up what we found and the process we now use before committing to a framework.

When VAP came out I was really excited about it, however operationalizing it on existing clusters looks to be non-trivial and I'm curious how people are managing it.

Specifically, on existing clusters I'm thinking of how to determine the impact of new VAP policies on existing workloads before you roll them out. You could roll them out in Warn mode but users only see the issue on admission so running workloads remain blissfully unaffected by a newly introduced policy.

You could use something like Kyverno to manage them in Warn/Inform and run them as background processes so you get all the info of which workloads would violate the new VAP policies. This is a really nice feature in Kyverno but if you are not using Kyverno already it's another on-cluster tool to deal with.

Are there any existing tools out there that will pre-validate VAP before it gets deployed, ideally off-cluster, or has a way of tracking VAP results that doesn't involve trolling through the audit logs?

Same boat as half of this sub I think. When the cutover hit last August we repointed Postgres, Redis, Mongo and Rabbit at bitnamilegacy as a stopgap and told ourselves we'd sort it properly later. It's July and later never came.

The legacy images don't get patched though and Bitnami's own line is that repo isn't meant to stick around, meaning we're sitting on unpatched infra that could also just vanish on us. Redis is a whole separate headache now with Valkey being the fork.

Small team, I really don't want to end up owning ten hand-rolled operator setups by myself. is it just service by service or how did y'all get all the way through this and land somewhere decent.

One of my engineers (im the CEO of Portainer) asked if I had the time to help him better understand Kubernetes architecure and components… and so rather than just explain it once, I figured why not share it wider.

So, https://kubeschool.portainer.io was born :-)

I run a k3s cluster at home and treat it like production. Everything gets backed up. Almost none of it had ever been restored, which is the normal state of affairs everywhere I've ever worked. So I automated restore testing (not restores... testing) and gave the job to an AI operator. This is how it's scoped, because that's the part that made it a sane decision instead of a scary one.

What it does

The operator takes a backup, brings it back in an isolated throwaway namespace, confirms the data is real and consistent, writes a dated pass or fail, then deletes everything it created. Production is never touched. Real recovery stays a human call... I'm not automating disaster recovery, I'm automating the proof.

One caveat I'd rather volunteer than have someone catch: the scratch namespace lives inside the production cluster. The isolation is policy and permission, not hardware... a fence, not a moat. Fine for a test-restore that evaporates in minutes; not for anything heavier. More on where that's going below.

The permission model

Read-only by default. The operator can't restore anything until a small, scoped permission is granted, and that grant is mine to make and revoke. No standing credential either... short-lived tokens minted per job and expired minutes later.

The piece I'd point at for anyone doing this: I split the capability from the binding.

• The capability (the Role and its permissions) lives in version control. Declared, reviewable, dormant, with nothing bound to it. A Role with no binding grants nothing.
• The binding (the thing that switches the power on) is applied out-of-band, deliberately kept out of GitOps, so the reconciler can't reinstate a permission I revoked.
• Teardown verifies rather than assumes: it re-checks that the privilege is actually gone and fails loudly if it isn't.

A teardown that fires a delete and trusts it is how you end up with standing access you think you revoked.

What's working

The lanes that only ever read run unattended, start to finish. The etcd drill is the clean example: it pulls a snapshot from object storage with a key scoped read-only to exactly one bucket, restores it to scratch, verifies, tears down, weekly, with an alarm if it goes quiet. Nothing to gate, because there's almost nothing that key could do. More than thirty restores now, zero failures, most of the register covered. The rule that fell out of it: an agent gets a key only when the key is narrow enough that handing it over doesn't matter.

Where I'm stuck, and want you to poke holes

For the drills that have to build things (restore a Longhorn volume, stand up a scratch DB), the operator needs a real write grant, and today I apply that grant by hand before the run. That's the one manual step left, which means a "fully automated" weekly drill is really a weekly appointment. I don't want to hand-approve it forever, and I don't want to leave a standing grant sitting there either.

The design I'm circling: a separate, deterministic policy broker (not another AI making judgment calls) that approves a narrow, time-windowed elevation for a specific request and forces de-escalation inside a few minutes. What I've got so far:

• The request is structured, not prose: {which identity, which role, what window, which drill}. No model output touches the decision. The agent requests; it never authorizes itself.
• Deny-wins and fail-closed. Broker down or request malformed means the drill fails and alerts, never fails open.
• Two independent expiries: a TTL on the binding and a short-lived token, so either one failing alone still kills access.
• De-escalation is confirmed by checking whether the binding still exists, not by trusting the remover's exit code.
• The grant path itself is guarded by an admission policy, so an out-of-shape or never-expiring grant is inadmissible, not just logged after the fact.
• The broker's rules live in git behind my merge, so the agent lane can't edit its own approval rules.
• Break-glass and anything high-blast-radius stay human.

As far as I can tell this is just the NIST PDP/PEP model pointed at an agent, and the human-facing JIT-access tools already do exactly this. But I can't find anything that ships it agent-native for Kubernetes, which makes me wonder what I'm missing. The failure mode I already see: inside a legitimate window, a prompt-injected operator holds a legitimate grant and spends it with someone else's intent. Shape-constraining the grant shrinks that to "can only touch drill-labeled scratch objects," but that's a reduction, not a cure.

So: where does this bite me?

(Separate track, less interesting to this sub: the fence-not-a-moat problem gets solved by moving the build-type drills onto a genuinely separate cluster on an isolated network, so prod is actually absent rather than fenced off. That same rig is where I'll finally rehearse a full-cluster rebuild from nothing, which I can't yet prove.)

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Im new to tech and recently started learning operators and controllers. With AI writing every code, which is faster than human, is it rational for me to learn about the kubernetes now ?

If you are in NYC and available on Wednesday evening, come hear guest speaker Noam Levy talk end-to-end testing with Kubernetes and networking magic. 🪄

➡️ If you can make it, please RSVP at https://luma.com/u4vja8rx

See you soon!

Hi everyone, let's go.

I am migrating from Beanstalk to EKS, and I wanted to clear up some doubts.

Which components/tools do you consider essential in a cluster? Currently, the cluster I am developing has:

• Karpenter (x86/ARM)
• Kyverno
• KEDA
• ESO (Fetches secrets from Parameter Store)
• Kong API Gateway (used as Ingress)
• ArgoCD for GitOps

What do you recommend? The CNI is the default VPC CNI, is it worth switching to Cilium or Calico? Bearing in mind that this will be my first experience building a cluster—I've managed them before at other companies, but I don't have much experience creating one from scratch.

I was watching this video from r/kodekloud on YouTube

and something the speaker said confused me.

He said: "Most people know how to deploy pods, but almost no one really knows how to secure them."

I work in cloud infrastructure, and in my experience, most customers put their entire cluster behind the cloud provider's security services (such as built-in security scanning) and a Palo Alto firewall.

So I'm trying to understand what he means. If those security controls are already in place, what is still considered "securing the pods"? What am I missing? Are there Kubernetes-specific security practices that these tools don't cover?

Inherited a fleet of EKS clusters and the utilization numbers are almost funny. Cluster wide CPU request versus actual usage is sitting around 25 percent. We're paying for a few node groups worth of headroom that's not being used, because every team padded their requests "to be safe" a year ago and never looked at them again.

The obvious move is to right-size the requests and let the bin-packer do its job. But the second I touch a team's limits I own any latency blip for the next quarter, so nobody wants to be the one who does it.

I've looked at VPA and a couple of the cost tools and half of them just show me the same graph I already have telling me we're overprovisioned. I know we're overprovisioned. Meanwhile the bill sits there being a third bigger than it needs to be, everyone agrees it's a problem in the abstract but no one wants to own the fix.

A simple walkthrough on how to use KAI Scheduler for topology-aware placement and gang scheduling to keep distributed GPU training inside one region, zone, or rack on Kubernetes

https://hrittikhere.com/posts/kubernetes-topology-aware-scheduling-kai

Looking for feedback from folks running apps that require mTLS that use API Gateway.

What controller are you using? Traefik, Envoy, etc.

Any pain points or gotchas you might have ran into?

Any other feedback or recommendations welcome!

sat down to review our cloud bill this morning and honestly felt my soul leave my body. The amount we are paying just for cross-az data transfer and nat gateways for our EKS clusters could literally fund another engineering team

Everyone pushes managed kubernetes like it's the holy grail of modern infra but at what cost? we're abstracting away the hardware just to get completely robbed by bandwidth fees. tbh im at the point where I just want to go back to deploying on raw bare metal. I was pricing out some alternatives on a hardware website that does instant dedicated deployments just to compare, and the difference is actually insulting. You can get a 16-core ryzen rig with unmetered bandwidth for less than what amazon charges us just to look at our vpc flow logs

setting up rke2 or kubeadm is basically trivial now anyway. Cluster API is solid. Why did we all collectively agree to hand over our wallets to the big 3 cloud providers? starting to think the whole "cloud native" premium is just a massive industry scam

So im working with Kustomize Overlays( FluxCD). i understand you setup your base infra with overlays per environment and then use patches for environment specific configurations such as replicas, urls, etc..

Some of these patches become very complex and large to the point i think its just easier to not have a base and just have each environment have its own configuration files. How much dry am i buying?

I've also seen the recommendation to use PostBuild subsituteFrom with ConfigMaps from each environment . My question is when should this be used over patches?

Sorry for the dumb questions and rant.

We've got signing in place.. cosign on everything that gets built, but I keep coming back to the question of what that signature actually proves versus what we assume it proves. It confirms the image wasn't tampered with after we signed it and that it came from our pipeline.

It says nothing about whether the base image we started from was itself trustworthy, or whether a compromised dependency got pulled in earlier in the build, before we ever ran the sign step. Signing our own output doesn't touch any of that, it just draws a line and says everything after this point is us.

The upstream side is where it actually gets uneven. Base image signatures are something we can realistically verify against sigstore before a build starts, but most of our language-level dependencies don't have signing adoption in the same way, so we're leaning on lockfile hashes and pinned checksums there instead, and that's a much weaker guarantee against something like a compromised registry or a dependency confusion attack.

For people who've actually hardened this end to end: are you verifying base image signatures before the build even starts, and what are you actually relying on for dependency integrity given how uneven signing support still is across ecosystems? Is any of this enforced at the pipeline level, rejecting a build outright, or is it still more of an audit-after-the-fact thing?

Got something working? Figure something out? Make progress that you are excited about? Share here!