So, went to levels.fyi. I've been there before. I'm not putting my salary in though so I don't have full access to the site.

The median US salary for SRE is $205k

Ok that's pretty decent but what the fuck is this >$600k nonsense I'm seeing bandied about in this sub?

Fwiw I'm trying to hire, not seeking a job, so I do want to know if my company is competitive. And it is... to an extent.

Someone please explain wtf is going on at levels.fyi and this sub.

Also, most of you are always going to be median engineers. And that's fine. But just, set your own expectations. Most of you, save for luck, are median skilled median pay band engineers. Myself included. I'm pointing this out specifically because I hear consternation and worry about the salaries of 1% of ICs in this and other career focused subs.

I realize the tone of this post makes me sound like an asshole and it'll probably get deleted by mods. Doing the best I can over here :)

Joined an SRE org from a background that was mostly infrastructure and technical design work. Since joining, almost everything I’ve been assigned has been alert-related in some form. Cleanup, enrichment, TTR, take your pick. It’s been the quarterly assignment three quarters running and there’s another one heading my way.

It also seems to land on me disproportionately compared to others on the team.

When I say more technical, I mean deploying infrastructure, working closer to the OS layer, and designing architecture rather than tuning and cleaning up what already exists.

I’m starting to think I should look elsewhere, but I wanted to sanity check first: is this just what SRE is, or am I getting the short end of the stick?

For my personal experience, AI models are no longer the bottleneck to handle incidents. The pain point is now how to provide right context to the AI to get right answers. I'm specifically working on this area and would like to discuss how do you solve the context problem.

This is what I did:

- Write a rule: what need to do and what should not do for handling an incident. Also what system should be retrieved from to get current production status.

- Put team maintained runbooks, TSGs and other docs in a git repo. Clone the repo locally so AI tools can search them instantly.

- Prepare mcp servers to connect different systems. This helps to get info real time. For example the logs and deployment data needs to be current.

Once we have the above 3 pieces, ask AI to analyze an incident by using them. Different teams may setup differently but very specific to their own needs.

In this way, I can successfully handle incidents by using AI in our team. AI can also give me the citations why it thinks the particular incident should be handled in this way. Then I do a quick manual verification.

I shared this with some of my friends and got positive feedback. I have put a post in our team's blog: https://blog.neatcontext.com/guide/2026/07/22/how-to-build-efficient-context-for-ai-clients/

What are your thoughts? I think the discussion here could benefit the future SRE area for AI leveraging. Thanks!

In my last role, we ended up multi cloud mostly by accident, major workloads on AWS, a big legacy system on Azure, some experiments on GCP, and a handful of SaaS platforms that each came with their own identity and billing surface, another provider in every way that mattered. Nobody planned a unified strategy, it just evolved that way. I have since talked to teams who went multi cloud on purpose, picking providers deliberately for specific managed services, so this is not only an accidental sprawl story, though many of what we learned came from cleaning up the accidental version.

We had to figure out which bits were in the right place for real reasons, like latency or compliance requirements and which were just historical accidents. We tried to get identity and backups into something resembling a common pattern, while accepting that each provider has its own quirks. Terraform helped once we agreed on conventions.

The other key piece was visibility: tooling that showed us what was under IaC across every provider we ran and what wasn't, plus a read on cost and risk for each. Without that, multi cloud felt like we were flying blind.

For anyone who is intentionally or unintentionally multi cloud: have you found a way to make it feel like a strategy instead of entropy or is it one control plane stacked on another?

I have got an offer to join Nvidia on their SRE team which revolves around maintaining or keeping up with the GPU Infra in their compute Infrastructure, and another offer from Palo Alto Networks as a swe, so I just wanted to know like which one to choose, I'm a new grad btw, please let me know your perspective guys

Throwaway for obvious reasons.

SWE/SRE background, ~7 years experience, applying across engineering and adjacent roles. Laid off earlier this year in a company-wide cut.

I tracked everything since the layoff, so here is the actual funnel:

- 240+ applications
- ~50 companies replied with anything human at all (screen or better)
- 33 reached a first round
- 9 reached a second round
- 4 reached a final round
- 1 offer, which stalled in negotiation
- Everything else: rejected at resume screen, req closed, or ghosted

Mostly just wondering if others are in the same boat. The numbers made me question myself a lot until I laid them out like this and realised most of it never even reached a person. Curious how everyone else's search has been going this year, and how you are holding up.

We're getting pushed to use AI for more ops work. At the same time I keep seeing posts here about an agent wiping a prod db or deleting the backups (the Railway/Cursor one especially). Feels backwards that we won't give a mid-level engineer write access to prod, but we'll happily point an agent at it.

How people actually handle the thing that touches prod, human or agent:

• When something needs to change prod (run a runbook, restart a service, rotate a cred, drain a node), how do you control who or what is allowed to do it? Jenkins jobs, break-glass/PIM, Teleport ...?
• Has anyone let an AI agent actually run things in prod, not just read? If so, are you relying on the tool's own guardrails or something you set up yourself?
• The bit I keep getting stuck on: keeping RBAC and the audit trail the same across every tool that can touch prod. Have you got it sorted?

How's everyone handling this?

Hi everyone,

I’ve been observing the AI boom across different sectors, and it feels like SRE isn't getting the same level of hype or rapid integration as Software Engineering (SWE) or Cybersecurity. While AI tools for SRE definitely exist, their progress seems slower, and their impact on the job market feels less disruptive so far.

As a Junior , I’m trying to wrap my head around this. It got me wondering:

1. Is SRE inherently more "AI-proof"? Does the high stakes of infrastructure and the "human-in-the-loop" necessity for critical incidents make it harder for AI to take over?

2. The "Invisible" AI: We see AI tools in the space, but they don't seem to have a clear impact on hiring or daily workflows yet. Am I missing something, or are we genuinely in a more "secure" niche compared to pure coding roles?

I’d love to hear your perspectives—especially from those who have been in the industry for a while. Is our field special, or am I just being overly optimistic?

I'm an SRE at a shop shipping a lot of LLM features and our incident response is basically broken for anything non-deterministic.

Our standard flow is built for the usual suspects like latency, errors, and uptime. If the 5xx rate is low and P99 is fine, our dashboard stays green. But LLMs are giving us soft failures constantly where the API returns a 200 OK and the latency is perfect, but the model is just outputting complete garbage.

Nobody gets paged, but the CX team is losing their minds because users are getting hallucinations. I'm struggling with the severity logic here. Is a 5% drop in eval scores a P3? Does that warrant a 3 AM wake-up call, or do we just treat it as drift for the morning?

I'm trying to wire online eval scores into our alerting pipeline so quality is a first-class metric and not just something we manually hunt down in a trace. We're using Braintrust for the traces, but the bridge between a bad trace and an actionable alert is basically non-existent right now. I need to know how to turn a production hallucination into a high-confidence signal without the noise.

Has anyone actually solved this quality vs. availability mess in prod? How are you deciding what level of dumbness actually deserves a page? I don't want to overengineer the alerts, but I'm tired of CX being the only ones who know the site is broken.

First of all, I hope a post like this is fine here if not I will gladly delete it.

For the past 1,5 years I have been designing Kubernetes clusters in a hub-spoke topology using Cluster API and CRDs. That is exactly what I want to do, but the brand is non existent and pay is mid.

I now got an offer from a better-known company for notably more money, but the role is owning dev tooling that runs on K8s (CI/CD, code scanning etc.), not building the cluster layer itself.
Long-term I want to stay in platform work, I wonder if someone has some opinions on my situation:

• Does ~2 years off cluster-level work hurt your shot at getting back into it later?
• Take the money/brand now, or hold out for the deeper technical role and stay within my lane?

A practical write-up on deadline budgeting, retry timing, clock skew tolerance, and expiration safety.

Pretty much the title. He said "no hand-holding at all" and that he dislikes it, so I'm wondering what he actually meant by that.

I think I'm going to be assigned to him, and I have no idea what he wants from me. Does it mean "shoo, shoo, do it on your own, don't bother me unless the sky is falling apart"? Or is it more along the lines of "okay, yes, I can help, but only after you've exhausted your resources on your own"?

To make matters worse, he seems exhausted as heck and was so busy he could barely find time to set up a meeting with me and had to cancel not two but three times. I've got a really, really bad gut feeling, guys. Maybe I'm overthinking it, but it reads like someone who wants a person who'll train themselves and wouldn't have time to look after a junior.

There is another devops in the team but he barely speaks English and i am seriously considering learning his language since I am a polyglot just so that I can talk to someone less... aloof because that SRE seems lowkey pissy as hell. He is also suspiciously young for a lead and gives a certain tech bro vibe that seems hard to get along with.

I'm fresh out of college, so I'm already anticipating a lot of hardship and some serious studying for hours every day... which I actually enjoy, honestly. How do I survive his style without ending up out on the street?

 Our Trivy scan throws a few hundred findings a week. Almost none of them matter. They're packages baked into the base image that the service never even loads.

Everyone stopped reading the report months ago, obviously. Which means the week a real one lands it slips through with the rest.

Tried severity tuning, a VEX file. The allowlist has just become another thing to maintain. The real problem is the base itself, with hundreds of packages that came with it.

Right now the gate passes everything anyway. Want it fixed before it costs us something.

In SRE book the chapter on “load balancing within datacenter” talks about lame duck state, backend subsetting and load balancing policies. While reading lame duck state I could relate it to pre-stop hooks in Kubernetes and it makes sense for a process to serve remaining requests before termination but stop accepting new requests.

My question is how subsetting and techniques about load balancing policies (weighted round robin etc) are used. I would really appreciate any response from engineers who have used this knowledge in practice.

I use AWS and I feel like there’s always something missing from my monitoring setup. How do you ensure you have everything in place and don’t miss anything critical?

We're working through how to add AI-assisted mitigation to our on-call workflow, while referencing Google SRE's white-paper from May. I noticed it's more concrete and more complicated at the same time.

The architecture has three pieces, AI Operator for autonomous mitigation, Actus as an execution control plane, and IRM Analyzer for continuous readiness evaluation against historical incidents. The Actus piece is just confusing, The mitigation agent can't exceed what Actus allows, even when the agent's own reasoning suggests otherwise. Actus is an architectural constraint, baked into the control plane which is very different from a permission model or a flag you configure per environment.

The IRM Analyzer evaluates readiness nightly against past incidents, so there's an actual record of where the agent failed. This help earn trust through measurement.

The honest question here is what a non-Google version of Actus looks like. We don't have dedicated infrastructure for a separate execution control plane. The constraint we have today is just the on-call engineer reviewing before anything runs. That works until the volume doesn't let it.

Whitepaper: sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!

Every place I've worked has had at least one. Mine right now is a \~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?

In a lot of places I've worked, "cloud infrastructure services" ends up meaning one person is the brain of the whole system. They know which dashboards matter, which Terraform repo is trusted, and how that strange VPC layout came to be. Other people can make changes, but all serious questions drift back to this one person.

It feels convenient at first because decisions are quick and you don't need long discussions to understand the setup. Over time, it turns into a liability. On‑call pressure targets one or two people, docs are never fully current, and the team starts relying more on pings than on their own understanding of the system.

By the time leadership notices there is a single point of failure, it is usually after a close call: that person is away during a bad incident, or hints they might leave. At that stage, spreading knowledge and responsibility is much harder because complexity has already grown around them.

If you've avoided turning one engineer into the "human runbook" for your cloud infra, what did you put in place early, processes, rotations, or ownership models, that worked in real life rather than only in a policy doc?

I am doing DevSecOps in a place where almost everything is defined as code and goes through CI or CD. We have tried plenty of DevOps security lists and frameworks and most of them felt great in slides but rough in real pipelines. The stuff that stuck was boring but effective: modules and Helm charts with secure defaults baked in instead of one off hardening after the fact. Secrets living in an actual secret store rather than whatever environment variable seemed convenient that week. Static analysis, dependency scans and IaC checks wired into pull requests so they hit developers early instead of as a surprise in staging. And then, once infra was mostly in code, a reality check: drift detection and cloud level scanning to find unmanaged resources and sneaky security group or IAM changes, pushing fixes back through Git before security reviews turned into archaeology. If you are in a similar position, which practices have survived real delivery pressure and which advice sounded good but made everything slower and more fragile?

We had an incident recently that made it clear how "production-blind" our ai coding agents still are. One customer-facing api endpoint started timing out intermittently only under real production traffic, with a specific feature flag state. Staging was clean, local runs were clean and it smelled like one of those classic "only prod is cursed" bugs. We asked our internal ops copilot, hooked into repos, ci, basic metrics and tickets, what was going on. It pointed to a recent refactor and a deploy where p95 latency jumped then suggested optimizing a couple of functions none of that was the real issue. The actual root cause was a subtle combination of a downstream rate limiter, a feature flag rollout and an old retry policy that only triggered under a specific load shape. The ai agent had no idea about live flag state, infra throttling or how those policies interacted at runtime. It was guessing from static code and git history, nothing more. The pattern i'm seeing across ai coding agents and ops copilots: they usually don't get real-time distributed traces and request context, feature flag and config state at the moment of the incident, a timeline of what changed in the last few minutes around the service or past incident and postmortem knowledge about similar production failures. So they act like very smart static analyzers, not like someone who's oncall watching a live system fail. For anyone using ai agents around incident response or production debugging: how are you feeding them real production context like distributed traces, feature flags, and infra events?

Preface: I'm trying to grow my team, and that requires a bit more structure. (Was 2 people, is now 5 + me, the manager.) But I struggle with aligning the structure of "Software engineering for Products" to the structure of "Software engineering for Operations". A lot of the misalignment for me comes from the fact that books, articles, and instructional videos are all written about Sprints with Product engineering in mind.

Also, though I've been doing operations work in an SRE-like way since 2008, I've really only ever used kanban. I've been on teams that use a sprint-like organization of the workflow but that was really more like "kanban but we call it sprints".

So, questions:

• How do you organize your project work?
  • 2 week Sprints with grooming/refining, planning, and retros?
    • Who is the "Product manager" for you? Who does the grooming and planning?
  • Kanban?
    • How do you groom/refine?
    • How do you prioritize?
• How do you incorporate reactive work during sprints?
  • on-call person handles most of it?
  • Just leave padding for reactive work? like a sprint is X% reactive and y% project?
• What support do you have from the business?
  • What size company? What size SRE team? What size Product Engineering team(s)?
  • Do you have a Project manager or similar? What non-SRE support is available?
• What tools do you use for project management?
  • e.g. Jira, Github, whatever

I have plenty of ideas and have read and watched plenty of helpful sources, but there are still some gaps for me and I'm interested in real world examples. Especially interested in examples from smaller teams.

I am responsible for cloud infra in a company that is big to break things but small enough that we do not have a huge SRE team. We needed a way of working that was not perfect world, but did keep us out of obvious trouble. So we settled on a sort of minimum viable discipline. IaC for anything important, with real reviews, not rubber stamps. Tagging and ownership everywhere so we know who to tap when something breaks. Baseline logging and metrics as non negotiable, not aspirational. Simple, documented resilience patterns instead of everyone inventing their own approach. And regular checks where we compare live cloud state to our IaC, fixing whatever drift we find through Git.

It is minimal, but it has kept us out of the worst kinds of trouble. If you are in a similar since you are in a similar situation, what is on your short list, the stuff you insist on so you can sleep at night even if not every ideal best practice is in place?

I was curious how well today's LLMs understand how systems work. So I gave 11 models the same incident findings, where the real root cause was mixed with its downstream effects