Throwaway for obvious reasons.

SWE/SRE background, ~7 years experience, applying across engineering and adjacent roles. Laid off earlier this year in a company-wide cut.

I tracked everything since the layoff, so here is the actual funnel:

- 240+ applications
- ~50 companies replied with anything human at all (screen or better)
- 33 reached a first round
- 9 reached a second round
- 4 reached a final round
- 1 offer, which stalled in negotiation
- Everything else: rejected at resume screen, req closed, or ghosted

Mostly just wondering if others are in the same boat. The numbers made me question myself a lot until I laid them out like this and realised most of it never even reached a person. Curious how everyone else's search has been going this year, and how you are holding up.

In my last role, we ended up multi cloud mostly by accident, major workloads on AWS, a big legacy system on Azure, some experiments on GCP, and a handful of SaaS platforms that each came with their own identity and billing surface, another provider in every way that mattered. Nobody planned a unified strategy, it just evolved that way. I have since talked to teams who went multi cloud on purpose, picking providers deliberately for specific managed services, so this is not only an accidental sprawl story, though many of what we learned came from cleaning up the accidental version.

We had to figure out which bits were in the right place for real reasons, like latency or compliance requirements and which were just historical accidents. We tried to get identity and backups into something resembling a common pattern, while accepting that each provider has its own quirks. Terraform helped once we agreed on conventions.

The other key piece was visibility: tooling that showed us what was under IaC across every provider we ran and what wasn't, plus a read on cost and risk for each. Without that, multi cloud felt like we were flying blind.

For anyone who is intentionally or unintentionally multi cloud: have you found a way to make it feel like a strategy instead of entropy or is it one control plane stacked on another?

I'm an SRE at a shop shipping a lot of LLM features and our incident response is basically broken for anything non-deterministic.

Our standard flow is built for the usual suspects like latency, errors, and uptime. If the 5xx rate is low and P99 is fine, our dashboard stays green. But LLMs are giving us soft failures constantly where the API returns a 200 OK and the latency is perfect, but the model is just outputting complete garbage.

Nobody gets paged, but the CX team is losing their minds because users are getting hallucinations. I'm struggling with the severity logic here. Is a 5% drop in eval scores a P3? Does that warrant a 3 AM wake-up call, or do we just treat it as drift for the morning?

I'm trying to wire online eval scores into our alerting pipeline so quality is a first-class metric and not just something we manually hunt down in a trace. We're using Braintrust for the traces, but the bridge between a bad trace and an actionable alert is basically non-existent right now. I need to know how to turn a production hallucination into a high-confidence signal without the noise.

Has anyone actually solved this quality vs. availability mess in prod? How are you deciding what level of dumbness actually deserves a page? I don't want to overengineer the alerts, but I'm tired of CX being the only ones who know the site is broken.

Hi everyone,

I’ve been observing the AI boom across different sectors, and it feels like SRE isn't getting the same level of hype or rapid integration as Software Engineering (SWE) or Cybersecurity. While AI tools for SRE definitely exist, their progress seems slower, and their impact on the job market feels less disruptive so far.

As a Junior , I’m trying to wrap my head around this. It got me wondering:

1. Is SRE inherently more "AI-proof"? Does the high stakes of infrastructure and the "human-in-the-loop" necessity for critical incidents make it harder for AI to take over?

2. The "Invisible" AI: We see AI tools in the space, but they don't seem to have a clear impact on hiring or daily workflows yet. Am I missing something, or are we genuinely in a more "secure" niche compared to pure coding roles?

I’d love to hear your perspectives—especially from those who have been in the industry for a while. Is our field special, or am I just being overly optimistic?

I have got an offer to join Nvidia on their SRE team which revolves around maintaining or keeping up with the GPU Infra in their compute Infrastructure, and another offer from Palo Alto Networks as a swe, so I just wanted to know like which one to choose, I'm a new grad btw, please let me know your perspective guys

We're getting pushed to use AI for more ops work. At the same time I keep seeing posts here about an agent wiping a prod db or deleting the backups (the Railway/Cursor one especially). Feels backwards that we won't give a mid-level engineer write access to prod, but we'll happily point an agent at it.

How people actually handle the thing that touches prod, human or agent:

• When something needs to change prod (run a runbook, restart a service, rotate a cred, drain a node), how do you control who or what is allowed to do it? Jenkins jobs, break-glass/PIM, Teleport ...?
• Has anyone let an AI agent actually run things in prod, not just read? If so, are you relying on the tool's own guardrails or something you set up yourself?
• The bit I keep getting stuck on: keeping RBAC and the audit trail the same across every tool that can touch prod. Have you got it sorted?

How's everyone handling this?

First of all, I hope a post like this is fine here if not I will gladly delete it.

For the past 1,5 years I have been designing Kubernetes clusters in a hub-spoke topology using Cluster API and CRDs. That is exactly what I want to do, but the brand is non existent and pay is mid.

I now got an offer from a better-known company for notably more money, but the role is owning dev tooling that runs on K8s (CI/CD, code scanning etc.), not building the cluster layer itself.
Long-term I want to stay in platform work, I wonder if someone has some opinions on my situation:

• Does ~2 years off cluster-level work hurt your shot at getting back into it later?
• Take the money/brand now, or hold out for the deeper technical role and stay within my lane?

Pretty much the title. He said "no hand-holding at all" and that he dislikes it, so I'm wondering what he actually meant by that.

I think I'm going to be assigned to him, and I have no idea what he wants from me. Does it mean "shoo, shoo, do it on your own, don't bother me unless the sky is falling apart"? Or is it more along the lines of "okay, yes, I can help, but only after you've exhausted your resources on your own"?

To make matters worse, he seems exhausted as heck and was so busy he could barely find time to set up a meeting with me and had to cancel not two but three times. I've got a really, really bad gut feeling, guys. Maybe I'm overthinking it, but it reads like someone who wants a person who'll train themselves and wouldn't have time to look after a junior.

There is another devops in the team but he barely speaks English and i am seriously considering learning his language since I am a polyglot just so that I can talk to someone less... aloof because that SRE seems lowkey pissy as hell. He is also suspiciously young for a lead and gives a certain tech bro vibe that seems hard to get along with.

I'm fresh out of college, so I'm already anticipating a lot of hardship and some serious studying for hours every day... which I actually enjoy, honestly. How do I survive his style without ending up out on the street?

A practical write-up on deadline budgeting, retry timing, clock skew tolerance, and expiration safety.

 Our Trivy scan throws a few hundred findings a week. Almost none of them matter. They're packages baked into the base image that the service never even loads.

Everyone stopped reading the report months ago, obviously. Which means the week a real one lands it slips through with the rest.

Tried severity tuning, a VEX file. The allowlist has just become another thing to maintain. The real problem is the base itself, with hundreds of packages that came with it.

Right now the gate passes everything anyway. Want it fixed before it costs us something.

In SRE book the chapter on “load balancing within datacenter” talks about lame duck state, backend subsetting and load balancing policies. While reading lame duck state I could relate it to pre-stop hooks in Kubernetes and it makes sense for a process to serve remaining requests before termination but stop accepting new requests.

My question is how subsetting and techniques about load balancing policies (weighted round robin etc) are used. I would really appreciate any response from engineers who have used this knowledge in practice.

I use AWS and I feel like there’s always something missing from my monitoring setup. How do you ensure you have everything in place and don’t miss anything critical?

We're working through how to add AI-assisted mitigation to our on-call workflow, while referencing Google SRE's white-paper from May. I noticed it's more concrete and more complicated at the same time.

The architecture has three pieces, AI Operator for autonomous mitigation, Actus as an execution control plane, and IRM Analyzer for continuous readiness evaluation against historical incidents. The Actus piece is just confusing, The mitigation agent can't exceed what Actus allows, even when the agent's own reasoning suggests otherwise. Actus is an architectural constraint, baked into the control plane which is very different from a permission model or a flag you configure per environment.

The IRM Analyzer evaluates readiness nightly against past incidents, so there's an actual record of where the agent failed. This help earn trust through measurement.

The honest question here is what a non-Google version of Actus looks like. We don't have dedicated infrastructure for a separate execution control plane. The constraint we have today is just the on-call engineer reviewing before anything runs. That works until the volume doesn't let it.

Whitepaper: sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!

Every place I've worked has had at least one. Mine right now is a \~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?

In a lot of places I've worked, "cloud infrastructure services" ends up meaning one person is the brain of the whole system. They know which dashboards matter, which Terraform repo is trusted, and how that strange VPC layout came to be. Other people can make changes, but all serious questions drift back to this one person.

It feels convenient at first because decisions are quick and you don't need long discussions to understand the setup. Over time, it turns into a liability. On‑call pressure targets one or two people, docs are never fully current, and the team starts relying more on pings than on their own understanding of the system.

By the time leadership notices there is a single point of failure, it is usually after a close call: that person is away during a bad incident, or hints they might leave. At that stage, spreading knowledge and responsibility is much harder because complexity has already grown around them.

If you've avoided turning one engineer into the "human runbook" for your cloud infra, what did you put in place early, processes, rotations, or ownership models, that worked in real life rather than only in a policy doc?

Preface: I'm trying to grow my team, and that requires a bit more structure. (Was 2 people, is now 5 + me, the manager.) But I struggle with aligning the structure of "Software engineering for Products" to the structure of "Software engineering for Operations". A lot of the misalignment for me comes from the fact that books, articles, and instructional videos are all written about Sprints with Product engineering in mind.

Also, though I've been doing operations work in an SRE-like way since 2008, I've really only ever used kanban. I've been on teams that use a sprint-like organization of the workflow but that was really more like "kanban but we call it sprints".

So, questions:

• How do you organize your project work?
  • 2 week Sprints with grooming/refining, planning, and retros?
    • Who is the "Product manager" for you? Who does the grooming and planning?
  • Kanban?
    • How do you groom/refine?
    • How do you prioritize?
• How do you incorporate reactive work during sprints?
  • on-call person handles most of it?
  • Just leave padding for reactive work? like a sprint is X% reactive and y% project?
• What support do you have from the business?
  • What size company? What size SRE team? What size Product Engineering team(s)?
  • Do you have a Project manager or similar? What non-SRE support is available?
• What tools do you use for project management?
  • e.g. Jira, Github, whatever

I have plenty of ideas and have read and watched plenty of helpful sources, but there are still some gaps for me and I'm interested in real world examples. Especially interested in examples from smaller teams.

I am doing DevSecOps in a place where almost everything is defined as code and goes through CI or CD. We have tried plenty of DevOps security lists and frameworks and most of them felt great in slides but rough in real pipelines. The stuff that stuck was boring but effective: modules and Helm charts with secure defaults baked in instead of one off hardening after the fact. Secrets living in an actual secret store rather than whatever environment variable seemed convenient that week. Static analysis, dependency scans and IaC checks wired into pull requests so they hit developers early instead of as a surprise in staging. And then, once infra was mostly in code, a reality check: drift detection and cloud level scanning to find unmanaged resources and sneaky security group or IAM changes, pushing fixes back through Git before security reviews turned into archaeology. If you are in a similar position, which practices have survived real delivery pressure and which advice sounded good but made everything slower and more fragile?

We had an incident recently that made it clear how "production-blind" our ai coding agents still are. One customer-facing api endpoint started timing out intermittently only under real production traffic, with a specific feature flag state. Staging was clean, local runs were clean and it smelled like one of those classic "only prod is cursed" bugs. We asked our internal ops copilot, hooked into repos, ci, basic metrics and tickets, what was going on. It pointed to a recent refactor and a deploy where p95 latency jumped then suggested optimizing a couple of functions none of that was the real issue. The actual root cause was a subtle combination of a downstream rate limiter, a feature flag rollout and an old retry policy that only triggered under a specific load shape. The ai agent had no idea about live flag state, infra throttling or how those policies interacted at runtime. It was guessing from static code and git history, nothing more. The pattern i'm seeing across ai coding agents and ops copilots: they usually don't get real-time distributed traces and request context, feature flag and config state at the moment of the incident, a timeline of what changed in the last few minutes around the service or past incident and postmortem knowledge about similar production failures. So they act like very smart static analyzers, not like someone who's oncall watching a live system fail. For anyone using ai agents around incident response or production debugging: how are you feeding them real production context like distributed traces, feature flags, and infra events?

I was curious how well today's LLMs understand how systems work. So I gave 11 models the same incident findings, where the real root cause was mixed with its downstream effects

I am responsible for cloud infra in a company that is big to break things but small enough that we do not have a huge SRE team. We needed a way of working that was not perfect world, but did keep us out of obvious trouble. So we settled on a sort of minimum viable discipline. IaC for anything important, with real reviews, not rubber stamps. Tagging and ownership everywhere so we know who to tap when something breaks. Baseline logging and metrics as non negotiable, not aspirational. Simple, documented resilience patterns instead of everyone inventing their own approach. And regular checks where we compare live cloud state to our IaC, fixing whatever drift we find through Git.

It is minimal, but it has kept us out of the worst kinds of trouble. If you are in a similar since you are in a similar situation, what is on your short list, the stuff you insist on so you can sleep at night even if not every ideal best practice is in place?

Is this book still valid after 8 year? I'm interested to read it to expand my theorical knowledge

Stumbled upon the Meta DrP paper from last December. The headline - 50,000 automated root cause analyses (RCA) per day across 300+ teams, five years in production, with MTTR improvements between 20% and 80%.

The design choice that surprised me is that they built investigation as a completely separate platform from their observability stack. The playbooks, the pattern matching, and the hypothesis loop all live in a different system from the dashboards and traces.

We've been treating "better runbooks" as the answer to slow investigations for the past year. Runbooks help when the incident is a shape you've seen before, but they're inert artifacts someone still has to read and follow at 2am. What DrP encodes is the senior engineer's diagnostic sequence as something that runs at alert time automatically.

The part I genuinely don't understand is whether the separation is worth the engineering cost at smaller scale. 300 teams gives you enough incident volume that pattern learning is actually viable. We're running maybe 5 serious incidents a month. I'm not sure if that's enough signal for automated RCA to get smarter over time, or if it just stays static.

Has anyone tried building investigation as a separate concern from the observability layer?

Built a curated list of official DevOps / Cloud / SRE MCP servers and agent skills

Hi folks,

I’ve been collecting and organizing official MCP servers, agent skills, and agent toolkits for DevOps, cloud, platform engineering, SRE, security, IaC, observability, and diagramming workflows.

Repo: https://github.com/DevOpsAIguru123/awesome-agentic-devops

The goal is to make it easier to find trusted sources instead of hunting through random MCP lists. I’m focusing on official or vendor-backed tools where possible, with notes around risk, write-capability, human approval, and operational use cases.

Current areas include:

• AWS, Azure, Google Cloud
• GitHub, GitLab, Azure DevOps, Atlassian
• Terraform, Pulumi
• Grafana, Datadog, Sentry, Splunk, PagerDuty
• SonarQube, Okta
• Databricks, Kubeflow
• Docker, Kubernetes, draw.io
• Agent skills and toolkits

Specialized DevOps/SRE agents and reference workflows are coming soon.

Would love feedback from folks using MCP or AI agents in infrastructure workflows:

• What official tools am I missing?
• Which MCP servers are actually useful in day-to-day DevOps/SRE work?
• What safety/risk fields would make this more useful?

If you find it helpful, a star would be appreciated.

Edit (2026-07-05): Since posting, added New Relic, Elastic, Cloudflare, CircleCI, MongoDB, Vercel, HashiCorp Vault, and Backstage - 55 entries now. Also added a "How entries are scored" section that spells out exactly how action-level, approval gates, evidence/tracing, and maturity are assessed (not just labels), and a weekly CI job that re-audits every link for reachability. Thanks for the early feedback - still hunting for gaps, especially solid official Kubernetes-native and Ansible tooling if anyone knows of one

Edit (2026-07-07): A few more since the last update - added a Vantage FinOps entry under a new FinOps/cloud-cost category, and OWASP MCP Top 10 as MCP security guidance (57 entries now). Also landed the first working reference agent: a read-only Terraform plan reviewer built on Google ADK (flags destroys, IAM wildcards, missing tags; never runs terraform apply), under frameworks/gemini-adk/. Still hunting for gaps - official Kubernetes-native and Ansible tooling especially..

Edit (2026-07-07, later): Leaned into the runnable reference agents - there are two now, both Gemini ADK, both wired to real Terraform Cloud + GitHub Actions and returning structured JSON. (1) Terraform plan reviewer: read-only, runs a speculative plan and returns risk findings with evidence plus an approval recommendation, and never runs terraform apply. (2) Terraform drift detector (new): runs a refresh-only plan on a schedule, flags when live infra has drifted from state (its example catches a bucket's storage class quietly going STANDARD -> NEARLINE), scores severity, and posts a Discord alert only when drift is actually found. Both live under agents/ if you want to lift the pattern. Also added skyhook-io/radar as the first community Kubernetes entry - flagged in the comments here, and it earns it: deep MCP surface (read-only topology/event/RBAC queries plus RBAC-scoped write and GitOps/Helm tools), not just a kubectl proxy.

Edit (2026-07-11): Added a one-command installer for official Agent Skills. It pulls skills from the official skill repos already in the catalog - Google, Microsoft, Azure, Azure DevOps (a few hundred skills total) - and installs them into your coding agent's skills folder: Claude Code (~/.claude/skills/), Codex, Cursor, VS Code Copilot, or Antigravity, or all at once. One curl/PowerShell line per agent, and there's a catalog doc grouping every skill by company and product so you can install just what you want (e.g. --source microsoft/skills --filter azure-sdk-python). Also added an official Redis MCP entry and updated Elastic to its new Agent Builder MCP endpoint

Edit (2026-07-19): Installer fixes since the last update. The one-line install now works on a stock Python — macOS's /usr/bin/python3 ships without PyYAML, which used to make it fail on a clean machine; no pip install needed now. Added --official, --community, and --all flags to grab a whole set at once (361 official skills, 26 community, or both), alongside the existing per-vendor --source/--filter commands. The community skills in the list (Terraform, draw.io, and Addy Osmani's set) are now actually installable, not just cataloged..