Let me set some context. I am running an application on AKS cluster where I have a file in memory that i need to upload to Azure Blob Storage on two different paths.

So I am doing those uploads concurrently.

I am using go, and there are two ABS client created and the upload is done in 2 diff goroutines.

Now what we are seeing is that one upload happens in 1-2 minute, and the other one takes 20mins to finish.

This has occurred enough times that we cant call it intermittent, but still it doesn't happen always.

Upon getting the Blob storage logs, we found that the individual API requests are taking about a minute for the slow upload, while it is taking abt e seconds for the fast one. They confirmed it is a network throughput issue rather than a disk write issue.

We looked at the pod network io metrics ob grafana and saw a drop in output rate during the upload window of the 2nd file.

The drop was from 300KB/s to 30KB/s.

Can someone help me figure out what might cause this to happen?

NOTE: What is consistent in all the instances of this issue is that everytime, there is one file upload that is faster and one is slower.

The same code is doing the upload sith same configurations. The file is same. What is different is the dst file path and the network port.

https://kubernetespodcast.com/episode/269-ai-usage/

Check out our latest episode: Navigating AI Policies in Kubernetes, with Kat Cosgrove and Natali Vlatko!

When people talk about AI in Kubernetes, the conversation is often focused on the user side, but for open source maintainers, there's of course a lot to unpack on how AI is changing oss contribution! How does AI impact the community, and what are the new rules of engagement?

We sat down with Kat Cosgrove (SIG Docs Technical Lead, SIG Release Subproject Lead, and Kubernetes Steering Committee member) and Natali Vlatko (SIG Docs Co-Chair, TODO Group Steering Committee member, and OSPO at Cisco) to unpack the recently published Kubernetes AI usage policy for contributors.

Topics covered include:
- Why you can't call an AI tool your "co-author" for legal reasons (because the AI tool can't sign the CLA)
- The rise of "AI slop" PRs and why AI-generated comments are causing trouble for OSS maintainers & reviewers.
- And of course, paths for new contributors to get involved sustainably!

Tune in for an insider's perspective into how AI is impacting contribution to one of the world's largest OSS projects!

We ran into this evaluating inference frameworks for a production LLM workload: clean benchmarks hid exactly the failure modes that matter at the infra layer.

Bursty concurrency and mixed prompt lengths exposed GPU memory pressure and latency spikes that never showed up under steady synthetic load.

We ended up building a testing process around soak tests and simulated failure scenarios (node loss, sudden load spikes) instead of trusting a single throughput number. Wrote up what we found and the process we now use before committing to a framework.