Written by Aleksandr Krivoshchekov and Walther Lee.

Prometheus has a simple idea at its core: metric_name + labels = time_series. Any two samples with the same metric name and labels map to the same series, but adding even a single extra label initiates a completely new series.

In Kubernetes, that extra label is usually pod. It’s noisy and always changing. It spams our system with more and more series, slowing down queries and draining memory. It’s mostly pointless for our users. But it’s also crucial for Prometheus to evaluate queries correctly.

All these problems with pod are partially solved in Prometheus, but they are still hard to avoid entirely. We could add periodic preaggregations known as recording rules to improve latency. We could sample and keep only a fraction of our metrics to get back some memory. We could even start all over again and try a different database. Each of these options would help for a while, but each of them also create new bottlenecks. What we wanted was an option that kept our existing model with minimal impact for our users.

This post is about that option, which dropped our total number of series used in production queries by 88%, reduced latency by 80%, and achieved 99.85% fidelity over 1M metrics per second in a single instance.

The problem

As the observability team at Reddit, we ran into the challenge of controlling our series count a while ago. If we instrument an application with counter series and send it to Kubernetes, those counters will show up in a pod. And since deployments can have multiple replicas, these counters have to be stored separately to be able to identify them by their own labels. Otherwise, a pod that measures 5 counts in a metric will conflict with another one that counts only 3.

# The label "pod" is necessary to keep track of counts from different targets
http_client_requests_total{pod="pod-a"} 5
http_client_requests_total{pod="pod-b"} 3

In its simplest form, this way of storing counters should scale with our deployments. If we have three replicas, Prometheus should store three counters that we can then add up at query time to get the total number. The reality, however, is not that simple.

Pods have very active lifecycles; they can be restarted after a rollout, shuffled by a node rotation, or killed by Kubernetes because they failed a check. A broad range of infrastructure events can make our pods come and go, and since each will start exposing its own metrics, Prometheus has to keep up with the growth at least until old entries are cleared out. Suddenly, a rollout can double our number of counters just by deploying a whole fleet of new pods and then double them again if we rollback.

Services at Reddit expose billions of metrics at any given time, so these infra events can cause a big impact. A large deployment with thousands of pods often needs millions of counters to monitor values like the total number of errors seen by users. Letting that payload double with every rollout compromises our observability when teams need it more.

This scaling pattern with metrics in Prometheus is unfortunate because applications in Kubernetes are actually designed to be unchanged by these events. A well designed deployment should be able to scale up and down, and to handle unhealthy pods seamlessly, without any impact in user experience. It comes as a surprise to product teams that events that don’t affect their applications do affect their ability to monitor them. 

On top of that, product teams don’t even care for these separation of metrics. They want to see the overall state of their deployments: the total number of requests and errors, the total latency. Single pods are treated as ephemeral and fungible, so the fact that they can influence their observability is an unexpected drawback.

Separation of metrics is an implementation detail that teams don’t care about, so we wanted to find a new solution that could match that expectation. We decided to start with a simpler goal in mind: 

• Find an efficient way of aggregating all counters in real-time to drop only the labels added by our infrastructure layer, like pod.

Aggregate at query time

The trivial solution would be just letting it grow and aggregate at query time. For example, with the expression:

• sum without (pod)(rate(http_client_requests_total[1m]). 

This is basically what we had before doing anything, and the main problem is still the unexpected changes in the number of counters that the query will load. While unexpected, this growth wouldn’t be so bad if it was intuitive. We think it’s reasonable to accept that the number of metrics will go up if we scale up a deployment. But teams don’t expect their query latency to go up just because they rolled out a change.

In practice, aggregations at query time are not consistent in latency or throughput, and the standard way to work around that in Prometheus is using recording rules.

Just drop the labels in recording rules

The single most common request we get in observability from other teams is if we can just drop the pod label. Unfortunately, that takes us back to the original problem: if two pods record different counts for the same metric, there’s no way to know which one we should keep. Probably both.

What teams really mean is to add up the counts without this label that split them in the first place. For that purpose, Prometheus uses recording rules to aggregate common queries periodically in the background. However, this has multiple downsides:

• At a big scale, evaluating a recording rule takes almost as much resources as evaluating the query, so using recording rules helps with latency, but it shifts the heavy payload from query time into a continuous load in the background.
• It generates toil, because teams have to define their rules in advance, and it’s hard to predict what information they will need, especially for incidents.
• And finally, they can still timeout and fail if the number of series spikes after multiple infra events.

In our experience, recording rules were a good option for shifting latency, but they were not a solution for our high volume of aggregations.

Aggregating at scrape time

The main bottleneck that nudged us to move away from recording rules was redundancy. If we use rules to drop labels, those evaluations will run periodically and load data that was scraped just a few seconds ago. We figured that we could do better than that by just aggregating the samples right when we receive them at scrape time.

Another good reason was the fact that scrapes are performed by Prometheus concurrently, as opposed to rule evaluations that compute operations in sequence across the different counters. Concurrent aggregations meant a more efficient use of our resources, so we decided to start by giving that a try.

However, the first problem we found is that counters in Prometheus are cumulative; their values can only go up or reset to 0, so when a query sees a sequence like 10, 14, 5, 8, it assumes that the pod restarted after 14 and then counted up to 5: 

http_client_requests_total    10 14 (reset)  5  8 

# Prometheus assumes the actual deltas were
http_client_requests_total   +10 +4 (reset) +5 +3 = 22 

If we just add up cumulative counters, a single restart will show up as a huge increment right after the reset:

http_client_requests_total{pod="pod-a"}    5 10 15 20    #  +5  +5  +5  +5 = 20
http_client_requests_total{pod="pod-b"}    7 12 17 22    #  +7  +5  +5  +5 = 22
http_client_requests_total{pod="pod-c"}    9 14  2  7    #  +9  +5  +2  +5 = 21

# in a query with increase(...) we would expect the deltas to be
increase(http_client_requests_total[1m])                   +21 +15 +12 +15 = 63

# however if we simply add them up, the series turns into
http_client_requests_total{}              21 36 34 49    
increase(http_client_requests_total[1m])                   +21 +15 +34 +15 = 85

A change in pod-c that was only 2 counts after the restart becomes 34 after the sum. The change becomes even more catastrophic when adding up counters during a rollout, as shown in the panels below:

In the end, we realized that adding up cumulative counters is an inherent but well-known limitation in Prometheus. But it’s fixable. By turning cumulative values into a sequence of increments or deltas we can solve the problem upfront, exactly how recording rules handle it.

Instead of adding up the raw values, recording rules use functions like rate and increase to account for resets and return only the delta over a step interval, and as long as all deltas are calculated over the same window, they can be safely added up.

For example, in the panels below, a pod restart shows up as a gap and a reset in the cumulative counter on the left side, but on the right side, the deltas just keep the gap. Working with deltas, aggregations don’t need special checks to add them up, even between restarts.

Switching to deltas completely solved the problem of aggregating counters at scrape time. We just had to keep track of resets and the last value seen for each counter to compute the deltas after every scrape!

However, this approach introduces a major catch: converting to deltas requires a fixed time interval to calculate them, but our teams query vastly different time windows throughout the day. While a 1-minute delta works perfectly for short few-hours-long graphs, it breaks down on a longer view. Ultimately, we would either have to lock everyone into a single interval or find a way to convert these deltas back into cumulative counters.

This delta-to-cumulative challenge isn’t new. Open-source projects like OpenTelemetry and VictoriaMetrics already offer processors to convert metrics between these formats. However, cumulative counters are stateful. This means that all aggregations must happen inside a single collector. If you split the load across two replicas that started at different times, their cumulative totals won't match, making it impossible for one to seamlessly fill in gaps if the other fails.

So our goal was no longer how we can aggregate our counters at scrape time. The new goal was: 

• How can we turn deltas back into counters in a way so that multiple replicas can agree on the value without requiring consensus? And how can we use the counters in one to deduplicate and fill gaps in another?

The ZigZag

We asked other engineers and read some posts and white papers that gave us an anchor of the error margin that other approaches were achieving. Some were using streaming in external components, probability algorithms, proxies, remote-write and processors. Overall, most of them expected an error margin around 9%.

But before trying any of that, which would require more time and changes and deployments, we decided to try something simpler. From studying how other projects convert cumulative counters to deltas it’s clear that the goal is to shorten the span of time a counter keeps a total count, so what’s the shortest span we can have while still being considered cumulative? The scrape interval! 

If we use the scrape interval as the span window for our deltas, and then we force the counter to reset immediately after, then the counter kind of resembles a delta. It’s still cumulative, but now it looks like a zigzag.

Keeping in mind that we were just trying to get close to the 9% error margin, and since that sounds pretty small, we decided to give it a try. We implemented the aggregation step at scrape time using deltas in Prometheus, and then added them up into a zigzag counter and ran some queries.

last = 0
for every 15s:
delta = delta_since_last_flush()
if delta < last:
last = delta
else:
last = delta + last
flush(last)

And it worked! The panels below show the delta below and its generated zigzag cumulative counter above.

Deduplication for zigzag counters

Zigzag counters let us store the aggregated data back in Prometheus again, but this isn’t too different from what we mentioned before with OpenTelemetry processors. The main difference is that now our data looks kind of the same between replicas.

The zigzags on each replica oscillate at different frequencies and are definitely not identical, but now the scale and values are somehow comparable. We can easily use one to fix gaps in the other, and the rates they return are very accurate! They have an average error margin of 0.15% with just 5 samples, exceeding our expectations by far.

We let this run for a few months in a test instance and it worked great. Aggregating a little over 4M series, driving query load and throughput down to just 3% of our old value for the metrics names with more series, and it didn’t even seem to use more resources. The operations are really light: add up or set a new number. And the new allocations are also minimal. It’s equivalent to adding just one more pod value to the label.

Zigzagging worked well for requests querying recent data, mostly because that data usually comes from a single replica in our system, but after a few months, we noticed an issue with queries spanning multiple hours.

Since Prometheus stores data in 2-hour blocks, long queries connecting multiple blocks had issues jumping between zigzags from different replicas. And we had the same problem deduplicating samples. For example:

25.31 @ 1775649538
50.48 @ 1775649568
25.49 @ 1775649598
25.42 @ 1775649607 // Note how the value goes down again
52.39 @ 1775649628
23.89 @ 1775649658 

The spikes happen because, even though both samples [email protected] and 25.42 @...607 recorded almost the same value at the same time, the second one is slightly lower, triggering the queriers to treat it as a restart. When a rate or increase is calculated, it reads as the following deltas:

50.48 @ 1775649568 
25.49 @ 1775649598 // a change of ~25 in 30s
25.42 @ 1775649607 // a change of ~25 in 9s!
52.39 @ 1775649628 // again, ~25 in only 21s

The extra sample ends up doubling the rate of 25 every 30s, and the inconsistency will last until the rate interval passes that point.

We fixed that by adding an extra step in our function to build zigzagging counters. Technically, counters following the algorithm above don’t have to be zig-zags. A counter whose rate is slowing down would record lower and lower deltas, resulting in a counter that goes continuously down, signaling a reset in every single scrape.

We decided to block that option and force our counters to go down only once in a row, essentially forcing a real zigzag pattern.

With that constraint, now we can walk through our deduplicated samples and explicitly exclude the ones that break the pattern. These values couldn’t have come from the same replica, so we know that a switch was made. The switch loses some accuracy, but this isn’t different from a normal deduplication algorithm that breaks samples’ resolution.

Let's say a change in replicas while deduplicating causes a second drop, resulting in the following two streams:

Replica 1: 22 50 21 
Replica 2:          20 48 19 52 20

Replica 1: 22 50 21 50 
Replica 2:             48 19 52 20

First, we tried discarding the last drop: 20 in the first stream and 19 in the second. This was a small change because samples are streamed for deduplication, so we just had to watch for two consecutive drops and remove the second. However, that doesn’t work for the second stream. After the drop we have:

Samples: 22 50 21 48 19 52 20
Delta:      28 21 27 19 34 20

Samples: 22 50 21 50 48 52 20
Delta:      28 21 29 48  4 20  # 48 and 4 break the steady rate!

Instead, we can fix both inconsistencies by keeping a look forward and discarding the first drop instead:

Samples: 22 50 20 48 19 52 20
Delta:      28 20 28 19 34 20
Samples: 22 50 21 50 19 52 20
Delta:      28 21 29 19 34 20 # Seamless transition

Production results

This algorithm was implemented and tested in a Prometheus fork and deployed to some of our instances in production.

For our largest metric, it shrank the number of series to only 2% its previous value, and the average throughput and latency across all queries went down to 12% and 20%, respectively.

In terms of resources, we didn’t find any significant memory overhead other than the one added by the new zigzag series ingested. CPU usage increased by 6%, but can be easily offset by replacing previous recording rules to aggregate the same data.

Mean error margin between raw and aggregated data was 0.15% with a standard deviation of 0.13%.

This was measured in an instance scraping 1M+ samples per second, using 70Gi and 16 cores before the implementation. It had 17M series before aggregation, and resulted in 2M new zigzag series.

Limits and Tradeoffs

This algorithm doesn’t solve every metric problem. It is limited to counters and counter-like types like Prometheus classic histograms, but gauges, summaries, and native histograms need different aggregations.

It also doesn’t change the existing Prometheus expectation around counter resets. If a pod restarts between scrapes and the new process reaches a value higher than the previous sample, Prometheus cannot know that a reset happened:

# Scrape #1
http_client_requests_total{pod="pod-a"} 90 

# "pod-a" restarts
http_client_requests_total{pod="pod-a"} 0 

# Scrape #2
http_client_requests_total{pod="pod-a"} 140 

# After the second scrape, Prometheus will assume the delta is +50, not +140 

That’s the same behaviour we see today with regular counters.

There is also a replica-local visibility limitation. Our aggregations run independently within each Prometheus replica using only the samples that the replica scraped. If only one of two replicas loses connection to a target, it will miss samples that were successfully aggregated in the other. Resulting in different values that we can’t compare or use to fill gaps between them.

This wasn’t a big tradeoff for us because it is exactly how recording rules work already. So we can think of it as a lightweight implicit recording rule:

# record: cluster:http_client_requests_total:sum
# expr: sum without (pod, instance) (http_client_requests_total)
cluster:http_client_requests_total:sum
==
# An internal series produced by our algorithm.
http_client_requests_total:materialized 

Finally, in our implementation we decided to keep the raw metrics. The algorithm only adds the new series as an optimization path for common aggregation queries, while raw ones remain available for debugging or per-pod queries. In the future, we could drop them after aggregation to reduce memory, which is an option we never had with rules.

What’s next?

Aggregating metrics during scraping is an idea that we had been chasing for a long time. What surprised us most was that we didn't need a massive rewrite or a fancy new storage format to achieve that. We already had all the necessary pieces of the puzzle, we just lacked a small but clever way to put them together.

The production results exceed what we thought was possible without switching databases, or at least without adding proxy-layers. Our algorithm cut query latency by 80% and dropped the total number of series loaded by 88%. And the best part is that it requires no changes for our users: all the same metrics, the same PromQL expression, and the same mental model.

That makes the next step pretty clear. We want to take this feature out of our fork and bring it to open source Prometheus, where our design can be discussed, tested, and improved by the community. By doing so, we hope that these changes can help others and spare them the scaling troubles we faced.

Written by Sarah Thomas

Hi, r/RedditEng!  I’m one of the leads of Reddit’s Women Engineers Employee Resource Group (ERG), WomEng. I’m here to tell you about how Reddit celebrated International Women in Engineering Day this year, and the idea that shaped the programming.

Like many ERGs, WomEng exists to build community among employees with shared identity, and provide a support network for its members. For ERG leaders, the challenge is translating these abstracts into real value and resources for members. 

For a group of women engineers, this can manifest in different ways. Our members share lived experiences, but also technical backgrounds and career interests. So while value can come from bringing members together into physical spaces, it can also be career-oriented and come from technical skill-building and visibility.

This perspective was reinforced in a conversation between WomEng leadership and Sabrina Farmer (former CTO of GitLab and former VP at Google) at Grace Hopper Celebration 2024. One idea from that conversation kept coming up in our planning and discussions:

Shared experiences are important, but having a community where people learn from one another and celebrate each other's accomplishments can be an even more tangible way to build a sense of belonging.

International Women in Engineering Day (June 23rd) provided an opportunity to put this idea into practice. Throughout the day, we hosted a mix of technical and lighthearted programming. Executives participated in segments like “Guess the PR Comment: AI or Human?” and “Can you match the vibe?” where they were challenged to vibe code alternative themes and features for the Reddit feed. We also organized Speed "Dating" networking events in our offices and in-person meetups across multiple cities to help members make new connections.

The centerpiece of the day was a series of mini technical talks delivered by fellow Women Engineer Snoos. Rather than bringing in an external keynote speaker, we turned the spotlight inward.

Members presented on projects and topics they were excited to teach others about. The talks ranged from technical deep dives like “Improving Image Performance on Android” (u/jessekrema) to career advice like “How Giving Back Can Propel Your Career” (u/tbonejones_). Every presentation reflected something the speaker had built or become an expert in.

Choosing our members as the speakers gave attendees the chance to learn something directly applicable to their day-to-day work while also celebrating the breadth of knowledge and experience that already exists within WomEng. 

International Women in Engineering Day gave us a chance to celebrate our WomEng Snoos, but it also reinforced something we hope continues to define WomEng @ Reddit: a community where members connect not only through shared experiences, but also by sharing our expertise, celebrating one another's work, and learning from each other.

~~~~~~~~~~~~~~~~~~~~~~~~

Thanks for reading! Below are bonus clips from the “Guess the PR Comment: AI or Human?” and the “Can You Match the Vibe” segments featuring our CTO, Amit and Senior Director of Engineering, Growth, Shweta!

Can You Match the Vibe

Guess the PR Comment: AI or Human?

Written by Alexey Bykov, Staff Software Engineer at Reddit & Google Developer Expert for Android technology

Reddit serves approximately hundred of millions playbacks a day on Android.
In our last two posts: Improving video playback with ExoPlayer and Taking ExoPlayer Further: Reddit's performance techniques we covered ExoPlayer performance and what you can do to improve startup latency, rebuffering, video quality & stability. (Check them out if you haven't yet, they also show how your production metrics may improve after every optimisation)

But bundling those performance practices into a reusable component with a safe and clear API turned out to be a different challenge. Over time, every new video integration became harder, and we kept finding edge cases that were difficult to support without breaking existing behaviour.

In this article, we'll share how we rewrote our abstraction on top of ExoPlayer from scratch: a better API for teams building video features, and even better performance and stability.

Goal: One obvious way to play video

Our main requirement for the API was simple: engineers integrating video shouldn't need to be video or performance experts. They shouldn't have to think about prefetching, player creation, prewarming, or lifecycle. Video should be fast by default and take a few lines of declarative code to integrate.

ExoPlayer is considered one of the best open-source video players among all platforms. It gives you a dozen valid ways to get to the same playable video, and at our scale, that freedom stops being a feature and starts being a challenge.

Tim Peters put it best in the Zen of Python: "There should be one, and preferably only one, obvious way to do it."

Architecture

Playback engine: ExoKit

To bundle all of the optimisations together, we built an agnostic playback engine called ExoKit.

It abstracts away ExoPlayer, makes the API stricter, owns all player communication, and keeps every playback in a central place with a single app-wide video state and effects handling.

It supports all of the performance features we covered earlier, such as:

• Player pooling and prewarming the pool on the app start, since creating a player can still take up to ~200ms according to our production traces
• Decoder reuse for identical videos, which saves up to ~80ms more
• Warming up videos during first composition (not to be confused with prefetching), so the first keyframes are decoded and rendered before the user scrolls to the video.

You can read more about these optimisations and their impact on our metrics here.

On top of that, it offers two APIs: a declarative one for rendering and an imperative one for managing playback state.

@Composable
fun PostVideo(
    mediaId: String,
    url: String,
    modifier: Modifier = Modifier,
) {
    val key = remember { PlaybackKey(mediaId, "feed") }

    // Imperative API: express playback intent.
    val actions = rememberPlaybackActions()
    PlayButton {
        actions.action(key, PlaybackAction.Play)
    }

    // Declarative API: render from playback truth.
    val state = rememberVideoPlaybackState(mediaId)
    if (state.isBuffering()) {
        LoadingIndicator()
    }

    // Declarative API: describe the video surface.
    Video(
        modifier = modifier,
        props = VideoProps(
            url = url,
            playbackKey = key,
        ),
        surfaceLifecycle = rememberLifecycle(),
    )
}

But even though it looks minimalistic enough, it still leaves a lot of responsibility on engineers. They have to decide where to play the video (which gets tricky with autoplay on), manage the player lifecycle, fire telemetry and coordinate between the declarative and imperative APIs.

To make it easier, we landed on a much more opinionated API. 

Self-hosted UI Components

Our first step was to abstract the imperative API from the feature layer: one less decision means one less way to get it wrong. But that logic had to live somewhere.

Rather than building reusable abstractions and inject them into every screen's ViewModel, we went the opposite way: every UI component became fully independent and self-hosted.

To ensure every composable can access product-related context, such as analytics, every block implements the following contract:

interface Component<Props : Any> {
    u/Composable
    fun Content(props: Props, modifier: Modifier)

    fun key(props: Props): Any = props
}

Implementations live in an impl module and can also access product-related context.
Here is a simplified example of what a lower-level composable might look like inside a component implementation:

u/Composable
// Very simplified version of the code for mute button
fun MuteButton(
    playbackKey: PlaybackKey,
    modifier: Modifier = Modifier,
) {
    // Imperative API
    val playbackActions = rememberPlaybackActions() // manage playback
    val globalActions = rememberGlobalActions() // global state, e.g. settings

    // Declarative API
    val audioSettings by rememberAudioSettings()
    val playbackState by rememberPlaybackState(playbackKey)

    // no sound -> hide, loading -> settings fallback, has sound -> real state
    val muted = when (playbackState.audio) {
        AudioTrackState.HAS_NO_SOUND -> return
        AudioTrackState.UNKNOWN -> !audioSettings.isEnabled(playbackKey.surfaceId)
        AudioTrackState.HAS_SOUND -> playbackState.isMuted
    }

    IconButton(
        modifier = modifier,
        onClick = {
            globalActions.action(
                GlobalAction.SetSurfaceAudioSetting(
                    playbackKey.surfaceId,
                    audioEnabled = muted,
                ),
            )
            playbackActions.action(
                playbackKey,
                PlaybackAction.Mute(!muted),
            )
            // More things, like handling product-related telemetry
        },
    ) {
        Icon(if (muted) Icons.VolumeOff else Icons.VolumeUp)
    }
}

The same pattern applies to every clickable or interactive media component. Every UI component also usually has its own ViewModel.

At the screen level, media becomes a set of declarative components that can be arranged like any other UI:

u/Composable
fun SimpleVideoScreen(
    screenState: SimpleVideoScreenState,
    videoComponent: Component<VideoProps>,
    playComponent: Component<PlayProps>,
    muteComponent: Component<MuteProps>,
    seekbarComponent: Component<SeekbarProps>,
    modifier: Modifier = Modifier,
) {
    val key = screenState.playbackKey

    Box(modifier.fillMaxSize()) {
        // 1. Video
        videoComponent.Content(props = screenState.toMediaProps())

        // 2. Play
        playComponent.Content(props = key.toPlayProps())

        // 3. Mute
        muteComponent.Content(props = key.toMuteProps())

        // 4. Seekbar
        seekbarComponent.Content(props = key.toSeekbarProps())
    }
}

Usually, the Media Foundation team, which focuses full time on media performance and developer experience, implements the components whose internals use both declarative and imperative APIs, whereas feature teams interact with the player through the declarative API.

This new setup gave us three big benefits:

• Product screens stay simple. They only need to arrange the media components and pass in a few props. Nothing more.
• Playback works the same way across the whole app. Every component shares the same playback state wired by playback key, actions, settings and telemetry pipeline under the hood, so users get one consistent experience no matter where they are.
• Performance. Screen-level state does not need to update for every media-related event. Only the affected media component recomposes.

Since introducing a component-based model in our app, AndroidX has made a new addition to their Media3 UI. This new model might serve as a good starting point if you’re thinking of solving similar problems today.  One key difference remains: our media components are tied to global playback state, while Media3 UI’s are tied to one Player. So the challenges still remain the same: player creation and the rest of the lifecycle ownership.

Experimentation Setbacks

Choreographer-based Seekbar

I personally found the seekbar a challenging component to implement smoothly, and there are a few interesting decisions we made.

So what does “smooth” mean in practice?
Basically, don't do more work than the screen can render. A 60Hz display takes roughly 16.7ms per frame; at 120Hz (which is not a rare thing anymore), that reduces to  ~8.3ms to render a frame. And if it goes beyond this limit, the user sees a jank.

This is where Choreographer can help. It runs right before Android draws the next frame, using the display’s real refresh rate. If the previous frame takes too long to draw, it waits for the next frame instead of firing again. That means we do not stack seekbar updates on top of an already-janky UI.

Estimated position or player position?
In our custom seekbar we predict the position by extrapolating from the last update, and only update it if the video is playing.
If your application's use cases are limited to video, ExoPlayer.getCurrentPosition() is a cheap operation, and it already uses an estimated position under the hood.

long elapsedTimeMs = SystemClock.elapsedRealtime() - positionUpdateTimeMs;
long estimatedPositionMs =
    Util.usToMs(positionUs) + (long) (elapsedTimeMs * playbackParameters.speed);

Polling this function will give you a smooth, speed-aware position for free. My recommendation is to still drive the updates using Choreographer and not use an arbitrary delay within a Coroutine or an Handler. 

Reddit's production experience
Moving seekbar updates out of screen-level state and driving them with Choreographer made the full-screen video experience measurably smoother.

On our full video screen, the slow-frame rate dropped by 7.3%.

Playback error 1004 & MediaSource reuse

While experimenting with our newly written ExoKit in production, there was a noticeable jump in playback error 1004) with "Unknown error" message. It affected a small share of devices and was challenging to reproduce locally. Our initial hypothesis was that it was a device-specific playback issue.

The root cause was our MediaSource cache. In some cases, the same cached MediaSource was reused across different player instances. This is something that I wouldn't recommend doing, as every media source is bound to the playback handler of the player it was originally attached to. Otherwise, you risk hitting an error which is thrown here.

Reddit's production experience
We fixed this by keying cached media sources by player id, so a MediaSource can only be reused by the same player instance.

Another possible fix was to move all playbacks onto a single playback thread. That also avoids the handler mismatch, but our production data showed that it made overall startup performance worse:

• % Video started in less than 250 ms: −0.199%
• % Video started in less than 500 ms: −0.132%
• % Video started in more than 1 sec: +1.165%
• % Video started in more than 2 sec: +0.634%

Rebuffering problems

If your app saves and restores playback position, be careful. In our old player we found that at least 36% of plays had a very short stall, under one second.

The main reason came from old choices made years ago. The old player restored the saved position as a check on every play and every move between screens. But the saved position did not match the position the player held at run time. Here is why: if you call player.pause() and then save player.position, and later call player.seek(savedPosition), the two numbers may differ by a few milliseconds. player.pause() does not finish right away. If you want the true position, wait for a playback state change first.

Reddit's production experience
We stopped saving positions this way. Instead we lean on the run time cache kept in the media source, and we reuse the player for videos the user watched before. This removed all of these tiny stalls.

Saving position on your own is still useful when the system kills the process of your app and you need to restore it for a long video.
If you cover this case, watch the order of your calls. Always call seek() first and prepare() after. If you do it the other way, the player loads key frames and chunks you don't plan to play.

Trade-off: Only one playable video at a time

Simultaneous playback is possible on Android, but hardware decoders are finite and shared across the whole device. A high-end phone might decode two or three videos at once, a low-end one just a single video, and an app sitting in Picture-in-Picture can quietly hold a decoder you were counting on. When you run out, you either fall back to software decoding (higher CPU usage, dropped frames) or fail playback with errors like 4001 or 4003.

Reddit's production experience
We could have partly mitigated this by checking the device performance class (a challenge in itself, since not all devices support it), but the development and testing cost wasn't worth the benefit. Instead, ExoKit runs a state machine that picks one active playback by priority (at Reddit it's based on how much of the video unit is visible, but it could also be based on how much of the screen's playable zone it fills).

We expected every surface to start faster after the rewrite. But surfaces that went from playing several videos to playing one improved their startup latency even more than the ones that already played a single video in the control group.

Conclusion

Besides an improved developer experience, the rewrite reduced perceived start latency by 65% at P50 and by 20% at P90. (An additional factor which helped here was the removal of a lot of unnecessary IO work from the startup path. In production traces, a single Main → IO dispatch before playback could add up to 11ms at the P99.)

Kudos to Merve Karaman, Ahmed Nawara, Irene Yeh and Vikram Aravamudhan for making this rewrite possible. We used to talk about this as a dream 2-3 years ago, and now it's our new reality.

Thanks to the following folks for helping me review this article: Iaroslav Khramov, Nicholas Ngorok

Authors:  Sreeprasad Govindankutty, Alex Okolish, Jerry Chu

This is the third post in our P0 media safety series. Previously, we covered Reddit's P0 Media Safety Detection and the Evolution of Reddit's In-House P0 Media Detection.

Introduction: The Blind Spot

Protecting our communities and users from high-risk, P0 content is a foundational priority for Reddit’s Safety team. Reddit's P0 media detection stack has historically relied on hash-matching: perceptual hashing algorithms like PhotoDNA, PDQ and hashsets from NCMEC, StopNCII, and Take It Down that compare uploaded media against databases of known violating content. This approach is precise: if an image has been reported and hashed before, we catch it. 

Our previous blog post identified a key focus area for future work: "Incorporate AI and ML to detect previously unseen media." This post describes how we did that by integrating Google's Content Safety API into our Content Classification Service (CCS), first for images and then for video. Our work has meaningfully improved our ability to identify and action harmful content of CSAM (Child Sexual Abuse Material).

Why Google's Content Safety API

Google's Content Safety API uses an ML model trained to classify raw media and return a priority score. Crucially, it operates on the media itself, independent of whether the content has been reported before. This makes it a natural complement to our existing hash-matching infrastructure. Hash-matching brings precision on known content; ML classification improves recall on unseen content. Together, they cover more ground than either approach alone.

How We Fit a New Signal Into Our Existing Pipeline

Instead of building a new service, we integrated the API directly into our existing media moderation flow. Media upload events are routed into image and video processing paths, where CCS prepares the media for classification. Then it sends the prepared media to Google's Content Safety API. The returned priority score from Google is then emitted into our eventing and analytics systems and used to create review tickets for Trust & Safety operations. The diagram below is a simplified view of the workflow.

After classification, the pipeline emits analytics events to our data warehouse, records telemetry, and creates review tickets based on the returned priority bucket for human review.

Key Technical Decisions

Image preprocessing. We resize images balancing Google's recommendations against preserving enough signal quality for accurate classification.

Rollout gating. A feature-flag-based fractional rollout controls blast radius. Only a configurable fraction of uploads call Google’s API, allowing us to ramp gradually and roll back instantly if needed.

The Image Integration: Design and Results

We ran a time-boxed evaluation over two weeks in Q4 2025. Reddit production image traffic was sent through the API, with priority scores logged to our data warehouse. Sampled review tickets were created across all priority score tiers, then reviewed and labeled by our Trust & Safety operations team using standard enforcement guidelines.

Because our sampling rates were deliberately biased toward higher-priority scores, we applied Inverse Probability Weighting (Horvitz–Thompson estimator) to correct for the bias and project true violation volumes in each bucket.

Image results

Meaningful CSAM recall lift. The API prioritized confirmed CSAM images that were not flagged by hash-matching. These were net-new detections: content that would have remained on the platform without this signal. The recall improvement over our hash-matching baseline was significant.

Priority scores sort risk effectively. The highest priority scores concentrated on CSAM with high precision. And the lower scores carried near-zero signals. This clean separation made it straightforward to route tickets to different Ops queues with appropriate review priorities.  

Higher actionability than hash-matching. When we compared the overall actionable rate of Google Content Safety tickets against PhotoDNA tickets (aggregating all violation types), the ML-based signal was substantially more actionable. After full rollout, the CSAM review queue maintained strong actionability.

Extending to Video

With the image results validated, video was the next frontier. Tens of thousands of video messages per day were flowing through CCS with no ML scoring. We built an event-driven callback for video, mirroring the image path with the addition of ffmpeg re-encoding and Google's video-specific constraints.

We rolled out at 100% for all priority levels over a multi-week evaluation window, creating review tickets for every video except the ones with the lowest scores. This contributed to the net-new CSAM coverage on video. The API surfaced confirmed CSAM videos that our existing video hash-match systems would have missed. The recall lift was smaller than the image evaluation (video upload volume is much lower), but directionally the same: the API catches what hashing cannot.

Impact and Takeaways

Hashing finds what is known; ML classification finds what is new. The combination is more effective than either approach alone. On images, the API delivered a meaningful CSAM recall lift. On video, it also added net-new CSAM coverage.

Stratified pilot design pays off. Deliberately varying sampling rates across priority scores let us measure both precision and projected volume before committing to full rollout, while keeping operational costs manageable. 

Future Work

1. Exploring the API's embeddings endpoint for additional signal types beyond the current priority classification.
2. Continuous monitoring of model drift via lower-score sentinel sampling to catch degradation early.
3. Evaluating Google's AI image detection API for additional classification capabilities beyond the Content Safety API.

If you're interested in working on problems like these, check out our careers page. Reddit Safety org is always looking for engineers who care about making the internet safer.

Hi friends - The job market is rough, and a lot of good people are looking for work. Good news is, Reddit is hiring across a wide range of functions, not just Engineering.

If you're looking, or know someone who is, take a look: redditinc.com/careers

Feel free to share with your network, group chats, Discord servers, fantasy leagues, family text threads, and other highly sophisticated professional recruiting channels. And not just with your engineering friends. Let's look out for one another!

Good luck out there. We're rooting for you.

Written by Shadi Altarsha, Transport team

When I started my career in infrastructure engineering, I wasn't sure how my work connected to the people actually using the product. I imagined myself deep in low-level systems that nobody knew existed, the kind of work that only surfaces when something breaks at 3 AM. 

Four years later, I've come to understand what I believe is one of the most important responsibilities of an infrastructure engineer: Keeping infrastructure concerns off application and platform engineers’ plates, so they can focus on their domain instead of paying the tax of context-switching into ours.

The way you deliver that experience is by doing the hard work of managing complexity, and building simple interfaces and abstractions so other teams don't have to think about it.

This post is about one such effort: how we removed an entire Envoy proxy from the serving path of Reddit's Home feed, Search rankings, and Notifications, and in doing so, improved availability, cut hundreds of CPU cores, and made onboarding new services trivial. The technical story involves gRPC service mesh, cross-namespace routing, CPU-aware load balancing, and a migration pattern for safely moving thousands of requests. But the underlying theme is simpler: sometimes the best thing infrastructure can do is disappear.

The Problem

When you open Reddit, your personalized experiences are powered by Ranking Platform, which is a set of gRPC services handling tens of thousands of requests per second across multiple Kubernetes namespaces and clusters. Traffic comes from our primary API gateway (GraphQL), Notification Platform, Answers, and other clients.

All of this traffic used to flow through an Envoy Gateway, a reverse proxy deployed and maintained by the Ranking Platform team, routing requests to the right service based on gRPC method and custom headers.

Here is the thing with great tools like Envoy: adopting one without deep experience is like giving a team a Formula 1 car when they have only ever driven regular sedans. A sedan is forgiving. You can change your own oil, and when something breaks it stalls on the side of the road. An F1 car has hidden sharp edges. It takes a pit crew to keep it on the track, and when it does fail, the failure is loud.
At first it feels like a huge upgrade. Then the complexity sneaks in. Configuration explodes, operational overhead grows, debugging becomes non-trivial, and before you know it, the team is not just the driver anymore. They need to be the mechanic and the pit crew too, on-call for a big dependency they did not originally sign up to own.

Why doesn't the Transport Infrastructure team just own these gateways? 
For one, we are a small Transport team supporting a large fleet of services across the company. More critically, the value of this kind of infrastructure comes from consistency across the fleet, which is hard to achieve when each team manages it independently.
We love Envoy and our Ingress deployment serves us well, but for gRPC service-to-service communication, we chose simplicity. So about a year ago, we adopted a proxyless service mesh instead.

Portal Transporter

Portal Transporter is Reddit's xDS-based control plane. It is a Kubernetes controller that watches GRPCRoute resources, EndpointSlices, and services, and then builds an xDS snapshot that it advertises directly to gRPC clients. The client dials an address like xds:///ranking.ranking-platform and gets back everything it needs, including routing rules, endpoint lists, load balancing configuration,  without any proxy sitting in the middle.

Instead of Client -> Proxy -> Backend, the architecture becomes Client -> Backend. The gRPC client itself handles routing and load balancing, programmed remotely by the control plane. This means you don’t need a sidecar, or a gateway.

If you want to understand the system in depth, my colleague Sotiris Nanopoulos gave a talk at gRPConf last year about it: Building a gRPC Proxyless World: How Reddit Scaled Resilience with xDS. It covers the control plane architecture and how we handle client-side observability as well. I would really recommend watching it if this topic interests you.

For this post though, all you need to know is this: Portal Transporter lets a service owner define their routing table once, and every client in the fleet gets it automatically. No client code changes needed. That is the foundation everything else in this post builds on.

The Goal

Here is the thing, to migrate one of the most important customers in the company to our infrastructure golden path, you cannot just show up at their door and say "hey folks, please use our toys." 

I approached this migration by asking myself a series of questions:

• In an ideal world where engineering cost is irrelevant, what is my perfect end goal?
• How much effort do my team and the team I am migrating have to pay to reach that end goal?
• Okay, that is very expensive. How can I find a common ground that achieves 90% of the dream goal without burning everyone out?
• And most importantly: how can I make the migration safe for both my team and my customer, and build trust in the process?

After working through these questions, here is what I landed on:

• Migrate the Ranking Platform to Portal Transporter without changing any of their existing architecture. I did not want to walk in and tell the Ranking team to restructure their services. The migration should be invisible to them as much as possible and the clients should only dial one address similar to how they used to do with Envoy.
• Cross-namespace routing was our biggest blocker. When I sat down and studied the Envoy Gateway configuration in detail, I realized we could not do this yet. The Ranking Platform team runs a cross-namespace setup, with services spread across multiple Kubernetes namespaces but unified behind a single Envoy routing table. Portal Transporter did not support that at the time.
• There were other challenges too. Envoy was a major source of SLO metrics and alerts for the Ranking team. They had even written a custom WASM filter to add tailored metrics to Envoy, which made replacing the observability layer harder than just swapping the data path. We needed to bring equivalent or better observability to the table before we could ask anyone to move.
• And above all, I had to do this safely. This is not some low-traffic internal tool, it serves traffic that powers some of Reddit's most important user experiences. A bad migration here would be very visible.

On the positive side, I was not coming empty-handed. I personally believe standardization is valuable, even if not everyone shares that belief as strongly, so I knew I needed something more concrete to bring to the table: CPU-aware load balancing (ORCA), which lets gRPC clients route requests based on how busy each server actually is. In practice that means better pod balance under load and real CPU savings, which are the things teams actually optimize for.

In the upcoming sections, I will walk through how we tackled each of these blockers and how we arrived at the architecture below.

The missing piece: cross-namespace routing

Before we could migrate anything, we had to solve a fundamental gap in Portal Transporter. The Ranking Platform's services all live in separate Kubernetes namespaces but are served through a single unified routing table. Clients call one address and the routing layer figures out which namespace to send the request to based on the gRPC method and headers. Portal Transporter had no concept of this. It assumed every GRPCRoute lived in the same namespace as the service it was routed to.

To make things harder, we already offer a public API to service owners across Reddit that supports exactly this kind of parent-child relationship across namespaces for the Ingress layer. Our ingress infrastructure relies on it, and our cross-cluster routing is also using it. Whatever we built for Portal had to fit into that same API. I would not introduce a new, ambiguous API or force changes on teams already using it. 
One API surface, two backends, that was the constraint.

This took a lot of iteration to get right. My first instinct was to look at what the Gateway API spec already offered. The Gateway resource has a model for cross-namespace routing, and it seemed like a natural fit. But we realized it was built on different assumptions than our infrastructure. So we stepped back and looked for something simpler.

What we ended up with is actually quite clever, and I say that knowing it took me three attempts and a lot of help from my colleagues to get there. We decided to use the GRPCRoute's backendRef field to reference other GRPCRoutes from different namespaces. Here is a simplified example:

The root GRPCRoute, let's say in the reddit-service-streaming-pipedream namespace, defines the main address that clients dial. But instead of routing directly to a service, some of its rules point to GRPCRoutes in other namespaces as backend references:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: ranking-platform
  namespace: reddit-service-streaming-pipedream
spec:
  rules:
- matches:
- method:
service: reddit.rankingplatform.ranking.v1.Ranking
backendRefs:
- kind: GRPCRoute
name: ranking-comments
namespace: reddit-service-ranking-comments
- kind: GRPCRoute
name: ranking-channels
namespace: reddit-service-ranking-channels
- kind: GRPCRoute
name: ranking-onefeed
namespace: reddit-service-ranking-homefeed

Each GRPCRoute in its own namespace then defines its own specific routing rules and points to its actual service. When Portal Transporter's control plane encounters these backend references, it expands them by pulling in the child routes and building a combined routing table that it serves to clients as a single xDS snapshot.

This design maps cleanly to the parent-child model our public API already supports for ingress. Service owners define their setup once, and it generates the right resources for both our ingress layer and Portal Transporter. No breaking changes or any special cases.

I want to be honest about the process though. This was not a clean straight path. The first design went through code review and was ultimately closed. The second attempt at combining GRPCRoutes had the right idea but the wrong execution. It was only on the third iteration, with significant input from my teammates, that we landed on the backendRef approach. It took several months from the initial proposal to the merged PR. 
I think there is something worth saying about that: good infrastructure design is not about getting it right the first time. It is about iterating and maybe discovering the real requirements as part of the process and it is about having colleagues who will tell you when an approach is not working, and being willing to throw away code that you spent weeks on (it is not easy when your “ego” is involved, by the way) and I am grateful my team gave me that.

Metrics and Alerts

Envoy was a major source of SLO metrics and alerts for the Ranking team. This was not a small detail but a real blocker.

At Reddit, we offer a standardized set of gRPC metrics and recording rules that teams can use to build dashboards and alerts. The Ranking Platform has a specific challenge: they use the same gRPC service and method across all their components, and they rely on custom headers like x-route to direct traffic to separate deployments. To get the breakdowns they actually needed, we had to add a custom layer on top of the standard metrics.

There was also the client-side SLO question. Client-based SLOs are an area we are still investing in, so there was no off-the-shelf path I could lean on for this migration. Aggregating metrics across multiple client sources is also not trivial, which meant I had to be pragmatic about scope.

This is where the "ideal world vs reality vs common ground" framework came back. A perfect, like-for-like replication of the existing observability layer would have taken months. But I could deliver a solution that gave the Ranking team most of what they actually needed:

• We built custom server and client metrics using grpc-go's StatsHandler, with the required headers as labels.
• We injected these metrics into the Ranking Platform components on the server side and into the clients.
• We used the server-side metrics to build alerts for each ranking tenant.
• We use the largest client as a stand-in for client-side SLO measurement.

Is it a one-to-one replacement for Envoy's observability? No. Does it produce a reliable outcome that the Ranking team can actually use? Yes. And honestly, I think that is the right tradeoff because perfect parity would have taken months of additional work and delayed the entire migration. 

Good enough, delivered now, was the better call.

De-risking the rollout

Here is a secret on how you make your managers and users happy at the same time: reliability. If there is one north star we operate under at Reddit Infrastructure, it is reliability. Everything else flows from that.

How do you deliver reliability when you are shuffling many RPCs?

Before migrating any traffic, I wanted to assure correctness to the best of my knowledge. And I want to be honest here, I really do not understand everything going on with the Ranking Platform and its clients. I do not know exactly what Notification Platform is calling or how Answers gives you all the information you need about the best daily shoe recommendations (spoiler: it is the Adidas Evo SL, thank me later). So what could I do to build confidence without pretending I understood every edge case?

I broke it down into three requirements:

1. Whatever development environment the engineers who work on Ranking components use must support Portal Transporter, not Envoy.
2. The Ranking Platform CI needs to run tests against my new setup and Envoy at the same time, and I should expect 1:1 correctness if my routing table is right.
3. My migration knobs should be able to turn traffic up and down between Envoy and the proxyless mesh in under two minutes.

For number one, we designed Portal to be development-friendly from day one, so there was no fundamental issue here. I just had to do some tweaks to make cross-namespace routing work in our development environment, but nothing major.

For number two, thanks to the Ranking Platform team, they already had very well thought-out smoke tests in their CI running against Envoy. All I needed to do was hook up the same calls via Portal alongside the existing tests and hope for the best. To my relief, it worked, the routing table was correct and the smoke tests passed on both paths.

Number three is where it gets interesting. We built a gRPC sampling client that has been part of Portal Transporter's migration toolkit since the early days. The idea is simple: we spin up two gRPC clients at the same time, one for the existing path (Envoy) and one for the new path (Portal Transporter). We use our dynamic configuration framework to provide a deterministic sampling knob that allows us to shift traffic between the two on the fly, straight from a UI. If something looks wrong at 10%, flip it back to 0% in seconds.
Under the hood, the implementation is actually quite neat. One of our engineers had the idea to override the ClientConnInterface interface in grpc-go, so that when Invoke is called, it checks against our rate sampler to decide which client handles the RPC. This means the sampling is completely transparent to the application code and the service owner does not need to change anything.

Teaching Servers to Tell Clients How Busy They Are

Remember when I said standardization itself is valuable, but I'm bringing something extra to the table? This is it.

With Envoy out of the picture, your gRPC clients now connect directly to server pods. The default load balancing strategy is round robin where every pod gets roughly the same number of requests. Sounds fair, right? The problem is that "fair" and "efficient" aren't the same thing as not all pods are equal. Some run on busier nodes, some share CPU with noisy neighbors, some just got unlucky with garbage collection timing. Round robin doesn't care, it sends traffic to an overloaded pod just as happily as it sends traffic to an idle one.
We implemented ORCA (Open Request Cost Aggregation) which is a standard that lets servers whisper back to clients: "hey, here's how busy I actually am." Each server pod reports its CPU utilization in gRPC response trailers, and the client uses these signals to compute weights. 
More loaded? Fewer requests. Less loaded? More requests. 
This is Client-Side Weighted Round Robin, and it's beautiful in its simplicity.

For Ranking Platform, the impact was significant: tighter CPU distribution across pods, improved availability, and a meaningful reduction in the number of CPU cores needed (we're talking about hundreds of cores saved).

My colleague wrote an excellent deep dive on ORCA at Reddit: Cheaper, Safer Scaling of CPU-Bound Workloads. If you're curious about the weight formula, the observability setup, and the operational lessons, go read that. It's worth your time.

After the migration

• We secured another nine on Ranking Onefeed.

• Hundreds of cores reclaimed across Ranking workloads, thanks to ORCA.

• Latency stayed flat
• Envoy Gateway fully decommissioned
• Simplified architecture
• Standardization: Ranking now uses the same infra golden path as the rest of Reddit

Conclusion

I started this post by saying that sometimes the best thing infrastructure can do is disappear. I think we did that here.

The Ranking team didn't have to change their architecture. The Transport team moved one of the most important deployments in the company to the golden path. The migration was seamless and honestly, the best compliment we got was that nobody realized it happened. For an infrastructure team, that silence is very important because it means trust, and in infrastructure, trust is your currency.

We improved availability for our users, saved a lot of CPU cores, and deleted a system that no team should have had to own in the first place. 

This work was also recognized internally, and that recognition mattered to me and the team. Migrations of this scale are never the work of one person. They happen because engineers across Transport, Ranking Platform, and the surrounding teams chose to invest in each other, and I am grateful to all of them.

A very happy ending 😄

Written by Will Johnson

Hello, my name is Will Johnson, and I’m a web engineer on Reddit’s Design System team. My team is responsible for Reddit's Design System, RPL, its corresponding component libraries, and helping other teams develop front-end experiences that adhere to our design system principles on all of Reddit's platforms.

Recently, I took on a project to resolve a persistent platform issue regarding web component dependencies that frustrated both Redditors and our engineering team. After responding to several internal bug reports, my team and I decided it was time to take a deeper look into the issue.  In this post, I share how I tackled that problem and what I learned along the way.

Missing Components

Reddit.com runs on a web component-based framework called LitElement, so we have quite a few template functions that render markup that contains a mix of native and custom elements. Since web components require JavaScript to render, developers were responsible for ensuring each page included the necessary component imports. Below is an example of what the markup and component imports would look like.

Example of server HTML template

<header>
 <reddit-header-small is-logged-in hide-on-scroll></reddit-header-small>
</header>
<main>
 <article>
   <h1 class="sr-only">Today's top posts</h1>
   <shreddit-post permalink="/r/cats/comments/abc123/fluffy/">
     <div slot="title">My cat learned to open doors</div>
   </shreddit-post>
   <shreddit-post permalink="/r/programming/comments/xyz789/rust/">
     <div slot="title">Why I rewrote everything in Rust</div>
   </shreddit-post>
 </article>
 <aside>
   <comment-forest-empty-state post-id="abc123">
     <p>No comments yet. Be the first!</p>
   </comment-forest-empty-state>
 </aside>
</main> 

// Register custom elements by importing their modules.
import 'reddit-header-small';
import 'shreddit-post'; 
import 'comment-forest-empty-state';

Historically, an engineer would write the page's markup, review the result in their development environment, and add any necessary dependencies to the import list if the page didn't display correctly. However, whenever an engineer removed a dependency or decided to load it asynchronously, the application could break. While "feature" may seem like a generic term, it has a specific meaning in our web codebase. Our web application is mostly server-rendered, where each page has a root feature and several ancestor feature nodes. Each feature has HTML markup and includes the dependencies necessary for that page to render. To visualize what happens when a dependency is missing, let's look at the example below. In that example, removing the modal’s import results in a darkened screen appearing without any additional context.

Having to keep track of your dependencies by hand created two problems: the first was a series of bugs in which various components appeared not to render, and the second was the need for increased internal support to debug these issues. To solve this problem, our team had the following idea: what if all client imports were automatically managed, so developers wouldn’t have to track them manually? By using import attributes, we could create a system that treats them as signals for client dependencies.

The big refactor: Adding Support For Import Attributes

To get things started, I needed to add the missing client imports to every template file that existed in our repository. I decided to enable our no-missing-imports rule from Lit Analyzer and create a script to parse the warnings and convert them into actual import statements. These imports would use a type attribute to indicate that they were client files, where the import would look something like this:  `import u/reddit with {type: ‘client’}`.After adding these imports across the codebase, I launched the development server to evaluate the system's response.

Unfortunately, the new client imports initially broke development and rollup builds. Node only supported type: json, and the rollup server builds picked up client imports, causing various issues. I chose to use plugins to strip these imports rather than more complex alternatives like dynamic imports or converting them to another format.

While the esbuild plugin for Node was simple, adding support for Rollup was a bit tricky. The TypeScript plugin performs type checking during the buildStart phase, which is too early for standard transformation hooks. I bypassed this by using type stubs in the monolith build to maintain type safety across specific builds while removing incompatible typings. With the server stable, I moved on to the parser.

Constructing the Shreddit AST Parser

Before coding, I aligned the team around a design document.

The node plugin approach was relatively straightforward: I used esbuild to omit client import attributes, but the rollup parser introduced an interesting problem. Whether I wrote my own transformer or used an off-the-shelf plugin, the imports remained unchanged. Although we have a rollup plugin to strip client imports, the TypeScript plugin sets up a file watcher and performs type checking during the buildStart phase of the plugin lifecycle. Using `buildStart`meant that block of code would always run before any plugin that uses a transform hook to load. To get around this limitation, I ended up using type stubs to remove incompatible typings in our monolith rollup build, which meant it did not affect the type safety of our client- and server-specific builds. Once the development server was operating correctly, I shifted my focus to building the parser.

Constructing the Shreddit AST Parser

I initially used the Recast AST parser to visit file dependency trees recursively, but later refactored to oxc-parser to cut runtime from 40 to 10 seconds. Before that, I relied on a few heuristics to reduce processing time, such as excluding child features, since Shreddit’s Orchestrator automatically handles their assets, and excluding components from a common client file that’s loaded on every page.

As I added more functionality to the parser, I discovered that each file contained metadata that could be used to parse the tree more efficiently. I decided to turn that into a class that would act as a cache, preserving visited files and telling me everything I needed to know about a file without revisiting it.

Metadata captured for visited files

export type Filemetadata = {
 features: Set<string>;
 dependencies: Set<string>;
 clientImports: Set<string>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 exports?: any[];
 imports: Map<string, Set<string>>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 ast?: any; // Store parsed AST for later access
};

type Entrypointmetadata = {
 skippedFiles: Set<string>;
 eligbleExports?: Set<string>;
 assets: Set<string>;
 clientImports: Set<string>;
}; 

With the basics in place, I decided to run the script and determine how the system was working so far. The image below captures the initial run and some of the work that the parser was able to do. As I looked more closely, I realized that some unexpected dependencies were sneaking into various features. These new dependencies caused the bundle size to increase dramatically. Were the exclusion heuristics working as expected? Was the parser incorrectly parsing the dependency tree? Or maybe these dependencies were correct after all. Either way, we needed a way to reduce import bloat.

Tree-Shaking Barrel Files

After completing my investigation, I realized that we had multiple barrel files in a few of our shared libraries that were not getting parsed accurately. There had been some discussion about this being a pattern we’d like to move away from, so my initial plan was to conduct a large refactor across the codebase. I decided to leverage an LLM to perform the refactor, so the work was done pretty quickly. But I soon realized that this change would require many reviews to get merged, a lint rule to curb its usage, and would ultimately still lack proper support in the parser.

To combat this problem, I decided to implement tree shaking to determine whether a module was actually requested before capturing its dependencies. The design of the algorithm involved peeking at the barrel file’s imports and temporarily holding them while I determined whether any modules had been imported by upstream files. While this incurs the overhead of parsing the barrel file rather than a direct import, it provides a sufficient mechanism to reduce the number of imports the parser detects. You can check the diagram below for a more detailed breakdown of how this workflow behaves.

Experimentation & Mitigations

With the final known roadblock resolved, I set out to begin experimentation by exposing a small subset of users to our newly generated imports. Unfortunately, the initial report showed several performance regressions. We were shipping more JavaScript, effectively slowing the rendering of various pages.

To unblock the project, I investigated which new dependencies were absent from production, using a staging utility to identify unused components. While I was considering a platform-level asset-exclusion API, another engineer used my work to address performance bloat by converting shared templates into features, powered by our feature-inclusion API, to exclude unnecessary assets from specific pages.

After finishing the migration, I created a document outlining the performance benchmarks we needed to meet. The goal was to prove that our changes were effective, keep partner teams informed about systemic impacts, and reach agreement on acceptable performance levels. The improvements were substantial, and we were finally in a position to roll out the feature to all users.

Conclusion

Overall, this project establishes a highly accurate dependency tree across our web codebase. It included improved code hygiene by systematically adding 4,000 previously missing client imports and removing 600 unused ones. Not only did this project eliminate a class of bugs in which UI components failed to appear due to incomplete dependency listings, but it also provided a foundational framework for future performance improvements.

Having an accurate dependency tree immediately enabled critical performance optimizations, including unused asset pruning, an optimized CSS delivery project leveraging the newly created AST parser, and a significant component-reduction initiative on a core surface. Adoption was made easy for engineers by including automated developer safety checks during commit and Continuous Integration (CI).

Additionally, we improved performance metrics across multiple areas, including Time to First Byte, First Contentful Paint, Total Page Load, Largest Contentful Paint, and Interaction to Next Paint, with strong statistical significance.

We’re hopeful that this platform will allow us to make additional improvements around serving more user-specific asset bundles in the future.

written by Nomi Khedawala, Technical Program Manager

Intro

I’m Nomi (Know-Me).

I joined Reddit as a Technical Program Manager in October 2024. I came from a background in product operations and technical program management, and what drew me to Reddit was the pace and the scale of the problems (and being a longtime lurker). I’ve had the opportunity to work on so many different areas of the business in my tenure here so far: Games on Reddit, Age Verification, Data Foundations, and even Search and Answers!

I'm part of the Tech PMO team, Reddit's centralized team for Technical Program Managers. Each of us manages programs for a specific org, set of teams, or one-off high priority initiatives. I recently wrapped up a company-wide Code Yellow focused on our consumer data foundations, and I'd like to share with you what a typical Thursday has been like for me during this Code Yellow.

If you're not familiar with Code Yellows at Reddit: they're our mechanism for escalating specific operational issues. When a problem is too big or too cross-cutting to fix through normal prioritization, we declare a Code Yellow. Think 4-6 week sprints with clear exit criteria and a temporary but significant shift in engineering priorities. They're designed to converge fast on a set of goals.

The Code Yellow I led alongside Paul Raff, our Data Foundations Lead, spanned the full consumer data supply chain across 10 surface areas. We defined how events should be instrumented, stood up real-time data quality monitoring, built curated data tables from raw events through aggregates and cubes, migrated metrics, and established a steady state operating model so the work would persist beyond the Code Yellow. It's the kind of program where the scope keeps wanting to expand and the TPM's job is to hold the line on what "done" actually means.

What follows is a snapshot of a typical Thursday during the Code Yellow. Thursdays were my heaviest day. Here's what one looked like.

Morning: 8:00 AM - 12:00 PM

8:00 AM - I start the day by facilitating a session of our Collaborative Problem Solving forum. This is something I built and run for our TPM team. The intent is that we get together every other week to work through real challenges we're each facing on our programs and look for opportunities to standardize how we operate. It's part peer support, part playbook development. Today I'm running a quarterly check-in to make sure the forum is still hitting the mark for the team.

9:00 AM - CPS wraps and I shift into Code Yellow mode. I'm reading through Slack messages that came in overnight and earlier this morning,. answering Qquestions from engineers on the Code Yellow and a thread with Paul about one of our exit criteria. I'm also reviewing every tracker, timeline doc, and milestone artifact to make sure nothing material has changed since we sent the Steer Co agenda out 48 hours ago. The Steer Co is a meeting with our executive sponsors and stakeholders, it’s later this morning and I want to be sure we walk in with a complete and current picture.

10:00 AM - Paul and I co-host office hours for any Code Yellow participant withquestions about scope, timelines, goals, or non-goals. These sessions are some of the most useful meetings on the calendar because the questions get genuinely technical. Today we're working through how to structure the data supply chain from raw events through fact tables, aggregates, and cubes. Someone raises a question about backfill strategies and the cost tradeoffs for different look-back periods. Another engineer wants to talk through whether we should have separate fact tables for web vs. mobile or consolidate by platform group. Paul and I don't always have the answer on the spot, but we talk it through with our counterparts and agree on a path forward whether that's deferment, fast-follow, or recommendation to senior leadership.

10:30 AM - More prep for the Steer Co. I'm pressure-testing each section of the agenda we drafted earlier this week. Is the status accurate as of this morning? Are the risks framed clearly enough for a room of senior leaders and our execs to act on? Do we need to add more context about a risk?

11:30 AM - I lead the Steer Co. This is our steering committee meeting with our CTO, CPO, VPs, and Directors. We share the program's current health, walk through progress against our five exit criteria, flag risks and blockers, and lay out what they should expect between now and our next check-in. Steer Cos are not the place for surprises. The prep I did all morning is so that every question gets a clear answer.

Afternoon: 12:00 PM - 4:00 PM

12:00 PM - Straight from the Steer Co into the cross-functional engineering sync. This is where I gather status updates from all of the eng teams contributing to the Code Yellow. We talk through their progress, timeline changes, risks, and I share relevant outputs from the Steer Co we just finished. Information has to flow quickly upwards and outwards during a Code Yellow. The engineers executing the work need to know what leadership is thinking, and leadership needs to know what progress is being made week over week.

12:30 PM - I meet 1:1 with a teammate to talk about how we can make the CPS forum even better. This week's session was my quarterly assessment, so we're comparing notes on what's working, what's not, and how to make the outputs more actionable. My goal is to take these findings, write up a retro, and present it to my manager and Sr. Director later to establish a baseline and start folding the best ideas into our team's standard operating procedures.

1:00 PM - Lunch. I close the laptop, make some coffee, and spend time catching up with my wife. One of the genuine perks of working from home is being able to take an actual break with the people you live with. Thumper, our cat, hates to see me coming, but I have to get a good arm wrestling match in with him so he doesn't sleep all day.

2:00 PM - Back at my desk. I go through my raw notes from the Steer Co and start synthesizing them into a stakeholder summary: key takeaways, discussion points, action items, and owners. Before I send it out, I reach out to each action item owner individually via Slack to confirm they're aligned on ownership and timelines. I learned a long time ago that assigning someone an action item in a meeting recap they didn't agree to is a fast way to create confusion and lose trust.

2:30 PM - I start drafting the weekly update that goes out to our Code Yellow participants and leadership. Milestones, health of each exit criteria, timelines, what moved forward this week, what risks or blockers surfaced, and how we're addressing them. This is one of those artifacts where the quality of the writing matters as much as the substance. If leaders can't parse your update in under 3 minutes, they stop reading it.

3:00 PM - I switch gears to the CPS retro. I'm taking the notes from my 1:1 earlier and incorporating them into the retrospective document I'm drafting for my manager and Sr. Director. The goal is to establish a clear baseline for the forum's performance and propose specific improvements that could be codified into how we run the CPS forum. This is the kind of work that doesn't show up on a sprint board but compounds over time.

3:30 PM - I'm starting to wrap up. I respond to Slack messages I couldn't get to earlier, schedule a few messages for teammates in different time zones, and share my draft weekly update with my program leads so they can add their content. I'll revise and edit tomorrow morning and get it out before the end of day Friday.

What a TPM Actually Does

A TPM's job is to make sure that the right things are happening at the right time and that everyone has what they need to focus on what they're good at. I handle the coordination, the communication, the risk identification, the stakeholder management, and the organizational scaffolding that lets a program with dozens of contributors across multiple teams actually converge on a set of milestones and objectives.

I love being a TPM because I love to be a part of other people's successes. Being a TPM means that you wear multiple hats so that the people around you can have peace of mind that all of the peripheral stuff is being handled while they can focus on their craft. TPMs are enforcers, enablers, coaches, communicators, therapists, and most importantly... friends not food! 🐟

If you're interested in becoming a Snoo, check out our open roles: https://redditinc.com/careers

Written by Cláudio Ribeiro u/EmeraldMacaw

TL;DR: Considering localization (L10n) at the inception of an online product isn't just a “nice to have,” it helps beyond translations by keeping the code cleaner, improving the UI's flexibility, and making sure the text content is top-notch.

Oftentimes, when a new online product is released, translation is treated like a future problem. It seems logical to say “I'll come back and fix it once we've scaled.” This happens often with software created by companies focused on a local market. But, including localization in the beginning is helpful beyond reaching more users: it makes the code more readable and guarantees text will display as intended everywhere.

Localizing after a product is out can be compared to making a fuel car electric, or trying to restyle a subreddit after millions of users have already gotten used to it. The effort required to retroactively localize is the most compelling reason to not leave it as an afterthought. Take Reddit, for example: our first attempt at localizing Old Reddit was crowdsourced and loosely supervised, which created an inconsistent experience and incomplete translations. Contributors also lacked the necessary context and visual aids to get the work done. In the end, few people used the localized versions and Reddit remained an English-first platform. (Though I must recognize the Pirate English version was pure gold.)

Once it became more noticeable that more and more people from different backgrounds and origins were browsing, contributing and creating, Reddit began working on a localized, globalized heart of the internet. Our first attempts were timid (volunteer translators commenting their suggestions in threads that contained the source strings), but we’ve matured our approach. We’ve implemented a translation management system (TMS) and are developing code in ways that keep localization in mind. Reddit now offers translations into 35 languages from 33 countries and supports 7 different alphabets that are used by millions of users.

Not surprisingly, we faced some setbacks before we got to where we are today: alphabets that wouldn't render, translations that weren't 100% adequate (as reviewers couldn't edit them), truncated text where the UI lacked room, untranslatable content (try translating the Tragedy of Darth Plagueis the Wise…), a mess with genders, plurals, and syntax, etc. These were difficult challenges to overcome, and we learned lessons along the way.

On that note, I’d like to share some of them with you. Below we’ll focus on some key aspects that illustrate the pros of pre-planning and how to get the house in order: accessibility, design, content review, time-to-market, data analysis, quality assurance, and code maintenance.

Accessibility

One of the first places where localization proves its worth is adding descriptions to images, buttons, and options (they even have their own writing style to be most useful to the end-user) to make a platform more accessible. Localizing the website is still relevant even when there are no ambitions to expand the brand abroad, as it's essentially “localizing” for users with impaired vision.

By making sure accessibility is implemented, a company can access a market within their own domain, and it becomes easier to localize into other languages. Accessibility is a way of “localizing” for users who need it; it includes different alphabets, such as  sign language and braille.

At Reddit, focusing on accessibility was a gamechanger. We improved our apps to include those with impaired vision, which allowed us to better serve our existing users–and to remain inclusive when we entered new markets with new languages.

When it comes to a product's design, localization can also help in less obvious ways. English is a “short” language, meaning it doesn't take much space and can express a lot of information without a lot of characters. This makes it easy to fit into tiny spaces, but other languages can take up way more space (up to 40% more than English in some cases) and that can often break the UI for users of longer languages.

This is where pseudo-localization comes in: it can run in design tools (Figma, Sketch, Penpot) and in the code, artificially expanding each word in English to 20~40% its size in a random distribution, allowing for designers to account for the most expansive languages without compromising the original content. It's like using “banana for scale” for buttons. Using pseudo-localization to design products improves the overall experience by preemptively ensuring the UI is comfortable to use in any resolution and language.

Cross-Functional Collaboration

When localization is introduced into a product's development lifecycle, it could mean potentially dozens, if not hundreds of extra specialized eyes will carefully inspect each word being written by different teams, for different projects, at different times. In order to do their best work translating, they interpret the context and the intention, so they can preserve the tone in their language. Their reading is way more focused than the general audience's, especially on a  platform like Reddit, which has a truly unique personality and tone.
Translators are professional linguists who need to read, interpret, and fully understand each string we publish, be it in our product itself, in the Reddit Help Center, or even in marketing material. This greatly amplifies Reddit's capacity to fix typos, outdated content, inconsistent experiences, and so on, as translators need to pay more attention to the source and often find errors a casual reader might miss.

Linguist eyes bring an extra level of polish to what we write and make our content even more slick and “together,” which translates into trust with our users. A translated product name can sometimes even serve as inspiration for a company's naming conventions, and since each culture has its unique way to express itself, different perspectives can make what we say more universal and human, which is kind of our thing.

Linguists will navigate through the entire UI/UX to see the localized product in practice. This allows them to help engineering teams by finding issues that might have been overlooked, or that the regular user wouldn't bother to report. Any new feature release gets extra pairs of hands play testing the content. This, allied with the content review component, adds an extra layer of polish and  results in a better overall user experience. 

Localization Infrastructure

Caring for localization infrastructure helps in content homogeneity, but also makes us more nimble when it comes to market expansion. Even if expanding into new markets seems like a distant dream, getting the structure ready from the get-go gives a company much more speed when deciding to launch in a new region. 

Properly implementing plurals is important because many languages have more than “one, other” plural options.

It ensures that code is ready to connect to translation management tools when needed, and dramatically reduces the cost and time spent to get things in place for translators and engineers. The effort required to go back and finish an incomplete structure and remove redundant code when more challenging markets come aboard (e.g. multiple plurals, different characters, right-to-left orientation, etc.) will inevitably delay your go-to-market timeline in those markets. When Reddit  introduced Arabic, addressing these concerns was critical to how we shaped our approach and launch strategy.

By creating strings with localization in mind, the code also becomes cleaner and string drift is avoided (i.e. we don't have the same word being spelled in three different ways in three different files). Centralizing all the product's strings means normalizing the storage of site content, which is a core tenet of good database and software design. We decoupled the management of translations from logic and reduced complexity and overhead in our code.
Engineering with L10n in mind helps make the code cleaner, more readable, and robustly documented. It's easier to understand where any string gets inserted, makes changes simpler and safer (ensuring no hardcoded strings ever reach production), and paves the way for automated tests that can enforce  best practices.

After an online product is localized, bugs are squashed, and continuous rounds of testing are carried out to ensure nothing is broken and that translations follow the context, are easy to understand, and don't conflict with the UI. That’s a huge win for users and development teams.

Descriptions can also be helpful for engineers who might need to update a piece of code related to a specific string.

L10n should be woven into every aspect

Localization ties together linguistics with development, influences marketing strategy, provides data for a coordinated expansion, encourages best practices, and is intimately intertwined with product development, whether it has been activated or not. That is why it can't be seen as a “plug-in” you can add at a later stage, but as a foundational layer that must be taken into account at the ideation stage. It will most certainly save you a headache in the future.

I'm new to L10n. Where should I start?

Check out the Unicode CLDR Project and find out how implementing a repository that takes care of dates, currencies, patterns, and measurements can also help in preventing bugs related to date, time, and locale.

Read about the ICU Message Format to learn how your strings can contain logic, plurals, and gender variants (this can be used even in English to personalize the user experience with “Mr.” and “Mrs.,” for example).

See the first steps to create localization-ready code in Python and Go.

Written by Chris Schenk (u/The-Real-Zucchini)

TL;DR

The GraphQL team at Reddit is expanding our observability to include build-time static code analysis to detect availability and latency regressions during the dev cycle before code goes out to production. This post covers how we detect changes to backend services calls in GraphQL changes that may fundamentally shift internal traffic to services within Reddit - before those changes go live in production. We wrote two versions, with the second version running in ~5 minutes in CI, a 36x speed increase compared to the first version.

Background

Reddit uses many standard tools to serve all the best Star Wars memes and arguments of when to celebrate May the Fourth (it’s May 25th, if you’re wondering). We’ve standardized on some critical technologies:

• Go, as the successor to our Python services
• gRPC, as the successor to Thrift
• Prometheus and Grafana to monitor our services in real-time
• OpenTelemetry tracing across our services to debug slow requests

GraphQL as our client API for the rest of Reddit to build features

GraphQL Primer

We’ve written recently about GraphQL in posts The Five Unsolved Problems in GraphQL and Protecting your GraphQL. This is a continuation of our ongoing efforts to make running our GraphQL service less of a burden on our team and create more stability and availability.

GraphQL promises an “implement once, use anywhere you need” system. GraphQL is a type system with named types which can either be scalars (Int, String, etc) or fields on object types. Anyone can extend the schema of your data model of types such as Post or Comment in the case of Reddit. Once a new field is added to any of these types and the backend is implemented to fetch and return that data, you can re-use it any time you like.

The Setup

In reality, not all data is created equal, and neither are the backend systems that hydrate it. As described in our Protecting your GraphQL post, the data backing your home feed, post, and comments have different criticality than loading your settings. To frame our problems, we have the following:

1. Our Go-based GraphQL service has millions of lines of code
   1. With hundreds of contributors to the codebase.
2. Our schema defines thousands of types
   1. With many 3.5x times more fields for those types.
   2. Supports very diverse feature sets that require different handling, scaling, and criticality classification.
   3. This number grows daily.
3. Our service talks to hundreds of backend services to fetch and mutate data for all of our users.

The Problems

GraphQL Makes it Easy To Break Scaling

GraphQL allows you to add any data to any query at any time. Take this example of an outage that has occurred periodically:

1. An engineer adds a field to a high requests-per-second (RPS) operation.
2. A low-scale backend service sees a significant increase in RPS.
3. The service runs out of memory, fails to scale, and causes the operation to return an error.

Our team is able to identify and implement workarounds, yet we remain vulnerable to these kinds of changes.

Running Multi-Region Requires More Care

Reddit is a global service, and in order to serve our users with a good experience, we also deploy GraphQL to multiple regions. But not every backend service is deployed to every region. While we can fall back to dependencies across regions, this can add latency.

In a multi-region world, we must take great care to not cause users to have a worse experience, so we must know in advance if modified GraphQL dependencies might cause problems before we deploy.

GraphQL Makes Debugging More Difficult

As GraphQL has grown into a monolithic API for our clients, the team receives many questions, including asks to perform root-cause analysis. While these problems are aren’t incident-worthy, it creates a non-trivial burden on the team:

• I added field `Post.foo` to my query, and it took down production. Why?
• Why does query `GetFoo` have 300ms increased latency since date X?
• If I remove field X from my query, will it reduce latency?
• Is the field `MyType.foo` used in any operation?
• What backend services are called for field `MyType.foo`?
• Why did RPS to upstream Q increase 30x?

Many of these questions arise because we already have good runtime observability and can see when we have latency or availability regressions after we deploy code. The follow-up challenges lie in:

1. Manually reading through our extensive codebases for the cause.
2. Analyzing pull request (PR) diffs to identify why a change was made.
3. Cross-referencing with changes and deployments from other repositories and teams if it wasn’t our code.

The above is common for any central platform team, but our time is limited and we need more self-service solutions for our contributors.

Our Motivating Questions

For some of our incidents, we began to wonder how we can improve our observability prior to some of these changes going out. We began asking the following questions after documenting a lot of the above issues:

• Can we detect runtime regressions at build time or development time?
• What problems specifically can we detect at build time?

Limits of Current Observability

To motivate the decisions for our dev-time solution, we will go into the limitations of our current observability. We can always add more telemetry, but solving for the above use-cases is not straightforward.

High Cardinality Breaks Prometheus

Today our telemetry includes two key metrics as shown in the diagram below:

1. Unique GraphQL operation identifiers requested by clients.
2. RPS and latency for each backend service endpoint call made.

One might think that we could simply put these values as labels in a single metric, and in fact we already tried! Each unique label value has a multiplicative factor on the total number of values saved in Prometheus. When we attempted to track both of these on a single metric, we went beyond the capability of our Prometheus instances due to:

1. Tens of thousands of operation identifiers live at any given time.
2. Hundreds of backend service endpoints.
3. Unique Kubernetes identifiers for each running pod.

These constraints are well-known and normal in any production metrics system, and we have to work within them.

Additional Abstractions Missing in the Metrics

Additionally, we have layers of abstraction between these two metrics that we still would not know even if we had one metric. Specifically we need to know the links between our GraphQL operations (teal) and fields (purple) to resolver functions (blue) and backend service calls (green+orange).

AI tools do help with this problem and we’ve started to use them for this purpose, and yet they can still run out of context and miss details with the size of our codebase in particular.

What We Get if We Have The Data

Let’s revisit the example mentioned above of an engineer adding a GraphQL field to an existing, high-RPS, critical operation. Let’s call this operation `Colors`. During development time, we would be able to do the following with this data:

1. Engineer modifies the `Colors` operation by adding field `transparency`.
2. Our CI systems detect critical GraphQL operation `Colors` has a field added.
3. Look up resolver function for the `transparency` field.
4. Find endpoint `GetTransparency` on `rgb` service is called to resolve the field.
5. Check if `rgb` service is fully deployed to all regions.
6. See that the service is not yet deployed everywhere.
7. Warn engineer and others that they will cause a latency regression if the change is made live.

This is only one example, and there are a lot of areas of prevention to explore once we are able to fill these data gaps.

A Static Code Analysis Prototype Is Born

As part of a Snoosweek (Reddit's semiannual hackathon) in the Fall of 2025, I explored static analysis to fill in the missing data. Could we effectively find endpoint calls within the resolver functions in order to tie fields to dependencies when GraphQL resolves in an operation?

Our first prototype showed us that it’s possible to do, so we made time on our roadmap to create what we call Dependency Mapper, specifically for our Go subgraph.

Goals

We wrote down the following goals for our design:

1. Summarize code changes involving endpoint calls.
   1. Link resolver functions to endpoint calls.
   2. Link GraphQL fields to resolver functions.
2. Detect negative impacts to production availability, performance and efficiency.
3. Do it at development time.

Building this nuanced analysis as requests flow from frontend to backend is what we mean by adding more “end-to-end” (e2e) observability.

Go Dependency Mapper V1 with gopls

The gopls tool implements the language server protocol (LSP). This is what your IDE uses to show you type information, allow you to click-through function calls, or find implementations of an interface in Go.

Since gopls already has logic to traverse codebases, we opted to use it in our first implementation of mapping our endpoint service dependencies for our GraphQL field resolvers. While gopls was a good first choice to prove the concept, this approach had multiple problems and missed endpoint calls.

Go Interfaces are a Hard Stop

This is speaking specifically to interfaces in the Go programming language, not GraphQL. If the mapper ran into a Go interface type that had more than one struct implementation, the program would not be able to traverse into the implementation without knowing which one to use. Normally we humans select which implementation to visit in our IDE, but a robot can’t do that without context.

This causes the mapper to miss entire sections of the code base, as interfaces are used by contributor teams for their own internal patterns, particularly for complex entity types like feeds and posts.

gopls is Memory Inefficient

gopls is fine for IDE use, as the number of requests issued during code authoring is quite low. But at a higher scale, mapping the entirety of the GraphQL resolver codebase, it uses up all available memory it can. This caused pods to be OOMKilled in Kubernetes as gopls would use more memory than available to the pod. Analyzing our codebase took ~3 hours, and we’d have to periodically stop and restart gopls to prevent this from happening.

Generally, gopls is also not meant to receive tens of thousands of requests per second; this isn’t the right use-case for it. Even running a single unit test would take minutes to complete, so we could not effectively iterate to improve the algorithm, and required us to do a second implementation.

Version 1 Is Still A Win

Even with these problems, our first run across the entire codebase proved to be illuminating, seeing how many endpoint calls each resolver function was making even when the dataset was incomplete. We also knew that in order to rely on these GraphQL field-to-endpoint mappings to make effective decisions to protect our infrastructure, we had to make it as complete as possible.

Go Dependency Mapper V2 using Go AST Traversal

In order to replace gopls, we had to reimplement what it is doing internally, and it is parsing and traversing the codebase by analyzing the Go Abstract Syntax Tree (AST) for our code.

Walking the AST is a Recursive Problem

An AST node represents a logical piece of syntax in the language. These nodes can reference each other in a recursive manner. Some examples from Go include:

• `ast.IfStmt` - represents an `if` statement and all of its constituent parts.
• `ast.CallExpr` - represents a function call expression and its arguments.
• `ast.Ident` - a single-name identifier, such as a variable name “foo” used in `foo := 12`.
• `ast.SelectorExpr` - a selector from a variable or package, such as accessing a struct member like `myStruct.SomeVal = …` or `mypackage.OtherFunc(...)`.

Once we have the code parsed into the abstract syntax tree, we can then walk each of the nodes and inspect information about them in order to detect the endpoint calls we’re interested in. We end up having switch statements that make for a regular recursive algorithm. You can read up on all the statement and expression types defined for the Go language AST at the Go language specification:

// walkStmt walks a single statement
func (dm *DepMapper) walkStmt(stmt ast.Stmt, pkg *packages.Package, ctx *WalkContext) {
  pos := pkg.Fset.Position(stmt.Pos())
  dm.logTrace("walking %T statement at %s %v\n", stmt, pos, stmt)
  switch s := stmt.(type) {
  case *ast.AssignStmt:
    dm.walkAssignStmt(s, pkg, ctx)
  case *ast.ReturnStmt:
    dm.walkReturnStmt(s, pkg, ctx)
  ...
  }
}
// walkExpr walks an expression
func (dm *DepMapper) walkExpr(expr ast.Expr, pkg *packages.Package, ctx *WalkContext) ExprType {
  pos := pkg.Fset.Position(expr.Pos())
  dm.logTrace("walking %T expression at %s %v\n", expr, pos, expr)
  switch e := expr.(type) {
  case *ast.CallExpr:
    return dm.walkCallExpr(e, pkg, ctx)
  case *ast.FuncLit:
    return dm.walkFuncLit(e, pkg, ctx)
  ...
  }
}

Parsing the Code

We use a combination of the following libraries to replace gopls:

• go/ast - In-memory representation of Go code.
• golang.org/x/tools/go/packages - Library for parsing and loading type information in the code.
• go/types - Representation of types loaded from `packages` above.

The packages.Package package (yep, that’s right) gives you options when loading and parsing the packages in `packages.Load`. By far the most important thing is to load all your packages together, as this library will reach out to nearby packages and parse them anyway. In our codebase, parsing our codebase with `packages.Load` takes ~2 ½ minutes, and we are expecting a significantly faster runtime compared to V1 and gopls when traversing the AST directly.

patterns := []string{
  "github.com/reddit/graphql/packageone...",
  "github.com/reddit/graphql/packagetwo...",
}
cfg := &packages.Config{
  Mode: packages.NeedName |
    packages.NeedFiles |
    packages.NeedSyntax |
    packages.NeedImports |
    packages.NeedTypes |
    packages.NeedTypesInfo,
  Dir: "path/to/go/mod",
}
pkgs, err := packages.Load(cfg, ...patterns) 

Fast Iterate-And-Test Loop

The slow speed of V1 makes it impossible to find and fix bugs, as traversing a single resolver function with gopls can take 3 minutes. Here in V2, `packages.Load` also caches its parsed files, so subsequent loads of code across executions take less than 10 seconds, which is impressive for a codebase of our size. This enabled us to write unit tests for various edge cases in the algorithm with a much faster time to completeness of our dataset.

The Go Interface Traversal Problem

The most important problem to solve replacing gopls is traversing through Go interface types encountered in the codebase. Even though `packages.Load` above gives us type information, it doesn’t give us runtime type information. Let’s illustrate with an example.

In this code, we have two service clients that both have the `GetPost` endpoint, `ProfileHydrator` and `SubredditHydrator`:

package services

import postpb "reddit.com/subreddit/api"
import profilepb "reddit.com/profile/api"

type SubredditHydrator struct {
  // Our gRPC client for posts
  postClient postpb.SubredditClient
}

func (s *SubredditHydrator) GetPost(id string) (Post, error) {
  return s.postClient.GetPost(id)
}

type ProfileHydrator struct {
  // our Thrift client for profiles
  profileClient profilepb.ProfileClient
}

func (p *ProfileHydrator) GetPost(id string) (Post, error) {
  return s.profileClient.GetPost(id)
}

We have a Clients struct that is initialized with each service client at startup:

package clients

type Clients struct {
  Subreddit services.SubredditHydrator
  Profile   services.ProfileHydrator
}

func New(cfg Config) *Clients {
  c := &Clients{}
  c.Subreddit = services.NewNewPostHydrator(cfg.Subreddit)
  c.Profile = services.NewProfileHydrator(cfg.Profile)
  return c
}

Now we have a helper function that loads a post for anything that has `GetPost`, specifically accepting the `PostHydrator` interface type as a parameter:

// PostHydrator allows for loading anything that looks like a post
type PostHydrator interface {
  GetPost(id string) (*model.Post, error)
}
// DoPostHydration takes any type of post hydrator
func DoPostHydration(id string, hydrator PostHydrator) (*model.Post, error) {
  return hydrator.GetPost(id)
}

In our GraphQL Go service, we use gqlgen as our execution engine. Our field resolver functions all have a receiver struct that is auto-generated, such as `queryResolver` or `mutationResolver`. These receivers have access to the Clients struct initialized with the service so the resolvers can make service calls to hydrate data:

package resolver 

import "reddit.com/graphql/clients"
import "reddit.com/graphql/services" 

type Resolver struct {
Clients                *clients.Clients
...
} 

type queryResolver struct{ *Resolver } 

// SubredditPost is the resolver for the subredditPost field.
func (r *queryResolver) SubredditPost(
  ctx context.Context,
  postID model.ID,
) (*model.Post, error) {
  // Load post using the helper function
  result := services.DoPostHydration(postID, r.Clients.SubredditHydrator)
  return result
} 

Notice the above where we pass in `r.ClientsSubredditHydrator`. When we traverse into the `services.DoPostHydration` call without any additional context, we don’t know what concrete type was sent into the function by looking only at the type signature of the `DoPostHydration` function. This is the same limitation as gopls. While gopls can find all implementations of an interface, it leaves the choice of which one to follow to the user. Since this is a program, we won’t have a human available to make that choice.

We conclude three things about solving this problem:

1. As an invariant, the runtime execution context has all the concrete implementations available,
2. We need a way to find concrete implementations during static analysis, and
3. We must track variables and their types as they’re used throughout the code in order to collect the implementations we need for continued traversal in the code.

Tracking variables and types as a pattern enables us to do the following:

1. Detect and track the concrete types for our services returned in our `clients.New` startup function.
2. Bind those concrete types to the GraphQL resolver function variables `r.Clients` above.
3. Pass variables as parameters to function calls throughout the call tree.

With this logic implemented, we can traverse our codebase and detect each endpoint call and their locations, linked with our GraphQL fields.

Detecting Thrift and gRPC Calls

A simple way to have implemented this detection would be to hard-code the Client APIs in a list and look for those in the resolver functions. Since the systems architecture at Reddit continues to evolve and we add new services, we needed a generalized approach to handle any new clients that are added to our codebase.

With the recursive nature of the algorithm, we are able to traverse into all code and make decisions based on where we are. Since Thrift and gRPC generate code, we can rely on patterns in the generated code to detect if a function call resides within that code. After analysis, we found the following statements in the generated code to use as our detection heuristic.

For Thrift:

var _ = thrift.ZERO

For gRPC:

const _ = grpc.SupportPackageIsVersion9

For HTTP, we pass in a specific package and struct name combination for the struct used for all HTTP service clients.

The Performance of V2

The full runtime of the V2 Dependency Mapper after the initial AST parsing depends on how many GraphQL resolver functions we have to traverse, including how complex the call trees are underneath them.

GraphQL Resolver Functions

There are a number of interesting implications when using gqlgen for GraphQL execution:

• Each top-level GraphQL field automatically gets a new resolver function.
• Additional per-field resolver functions can be added to gqlgen.conf.

For our many thousands of fields in our schema, only about 10% have resolver functions defined for them. This means each resolver function is responsible for resolving a lot of data that may be requested underneath it. We can correlate resolver complexity with how much time it takes to traverse the function call chain.

The Numbers

The amount of time to traverse our GraphQL resolvers is ~2 ½ minutes for a total of ~5 minutes of runtime combined with the `packages.Load` parsing of the code. This is a 36x speed increase in our runtime.

WIth a runtime at ~5 minutes, this is fast enough for us to move away from a Kubernetes cron job to standard validation during our build process in CI. Every time we push code to our mainline branch and release it to production, we are guaranteed to have a static analysis dataset of what’s latest in production just minutes after landing.

This is also single-threaded and we could get further runtime gains if needed by adding Goroutines to process a queue of resolver functions. 

The Result

We finally arrive at our destination: a mapping of GraphQL Operation’s fields (purple) to resolver functions (blue) to service endpoint calls (green+orange). We store the dataset identified by the Git commit SHA of what code was analyzed which enables us to link it to a release in production.

The example JSON below demonstrates the data:

1. The associated GraphQL field in `graphQLField` object: `Query.postById`.
2. Information about the resolver function, in this case `func (r *mutationResolver) GetPostById(...)`.
3. Service endpoint calls in the `endpointCalls` object.

​

{
  "git": {
    "sha": "d3f43d80f68583cfff85aca3869d011498134107",
  },
  "createdAt": "2026-04-08T21:22:42Z",
  "durationNanos": 194648368750,
  "service": {
    "serviceName": "graphql",
    "language": "go"
  },
  "targets": [
    {
      "serviceType": "graphql",
      "durationNanos": 194648367792,
      "graphQLData": {
        "configFilePath": "gqlgen.yml",
        "resolverFunctions": [
          {
            "package": "reddit.com/graphql/internal/resolvers",
            "filename": "resolver.go",
            "line": 283,
            "column": 1,
            "functionName": "GetPostById",
            "functionReceiver": "mutationResolver",
            "graphqlField": {
              "parentType": "Query",
              "fieldName": "postById",
              "isDeprecated": false
            },
            "endpointCalls": {
              "total": 1,
              "countsByUpstream": {
                "reddit.com/graphql/internal/backend.Clients.Post": {
                  "name": "reddit.com/graphql/internal/backend.Clients.Post",
                  "total": 1,
                  "countsByEndpoint": {
                    "GetPostsByIds": 1
                  }
                }
              },
              "calls": [
                {
                  "package": "reddit.com/graphql/internal/backend/posts",
                  "filename": "posts.go",
                  "line": 408,
                  "column": 42,
                  "clientID": "reddit.com/graphql/internal/backend.Clients.Post",
                  "clientLocation": [
                    "reddit.com/graphql/internal/backend.Clients",
                    "Post"
                  ],
                  "endpointName": "GetPostsByIds",
                  "protocol": "grpc"
                  "callStack": [
                    ...
                  ]
                }
              ]
            },
            "durationNanos": 424958
          },
        ]
      }
    }
  ]
}

Limitations of Static Analysis

Static Analysis Tells You What Might Happen

It does not tell you what actually happens. This is an important distinction when making sense of the detected endpoint call counts. The static analysis essentially gives you the worst-case call counts if every single branch in every portion of traversed code was executed, including all conditional branches, which is never the case in reality.

For Loops are a Problem

Since we’re acting as an interpreter, “for” loops become a problem:

orderedIdx := make([]int, 0, limit)
for i := 0; i < limit; i++ {
  orderedIdx = append(orderedIdx, i)
}

Since we are an interpreter, we don’t actually know the value of `limit` during our analysis, so we currently are unable to properly assign variables and process the block with the correct variables. We have cases in our codebase where function literals that contain backend service calls are added to a slice and then iterated and handed off to Go routines, and we have yet to come up with a solution for this.

Range statements are similar, but are more approachable.

ptrEvents := make([]*model.Event, 0)
for _, event := range events {
  ptrEvents = append(ptrEvents, event)
}

Here `events` is a slice that we may have been able to internally track through built-in `append` calls. If so, we would be able to iterate the values we could interpret and run the block with the correct variable assignment. However, if the `events slice was assigned through indexing (e.g. `events[i] = myValue`) we would not have the data.

Ultimately, we may be able to solve this problem with detecting index references inside of loops and implement a back-tracking algorithm to iterate when we see a slice indexed by an integer. This is future work for us to explore as it would require a decent amount of work to implement roll-back functionality, especially if the slice reference happens further down the stack through another function call (which is possible).

How We’ll Use This Data

Reducing Data Over-Fetching at the Operation Level

We are already underway with client projects to reduce data over-fetching and make the app more efficient and performant. With this data set, we can now parse a full GraphQL operation and look up the field mappings while we’re traversing the operation and summarize all the possible work that an operation might perform during execution.

Our client teams have also generated data sets through static and runtime analysis of what data is fetched but not referenced within client code. The next step is to analyze the unused fields and group them by resolver function, so client teams can prioritize removing groups that result in entire backend endpoint calls being removed from the runtime execution, resulting in faster page loads for everyone.

Regional Service Readiness

As Reddit continues to expand its global infrastructure footprint, we want to know what GraphQL operations are fully-servable within a region. We aren’t yet able to roll out all our services at once when serving in a new region, so we want to use this data set alongside our Achilles SDK which we use to manage our Kubernetes workloads, to detect if an operation can be fully served out of a region or not. This way, we can perform intelligent routing to keep your Reddit experience quick and responsive, no matter where you’re coming from in the world.

Analysis for Backend Go Services

Since the Dependency Mapper fundamentally operates on analyzing a function and all of its dependencies, we can adapt it to also work on our backend services and continue to build out a static analysis graph across service calls at the company.

Detection of Database and Experiments Calls

The logic for detecting “edge” calls that exit the system could be easily extended beyond endpoints to support systems such as:

• Redis, Memcache
• Postgres, MySQL or No-SQL databases
• Sqlc queries and extraction
• Experiments systems calls
• And more!

We can add these as a configuration parameter to enable/disable at analysis time. We can detect uses of any of the associated libraries and track those to be reported in the final data set as well.

Tracing Data Sources for Fields

Today, the Dependency Mapper tracks what backend calls are made during execution. The algorithm and data structures could be extended to tell you exactly where a piece of data comes from when it is returned in the GraphQL API, even if that data is derived from multiple sources. This is helpful as we continue to migrate data to dedicated services and need to know where data is used so we can update references in our code.

And Finally

We reached our goal to connect our two runtime datasets together with a static analysis dataset, and have a strong roadmap for adding more functionality for detecting more regressions before they go to production.

Special thanks go to our teammate Brendon Kofink for his V1 implementation of the Dependency Mapper.

We’re always looking to improve our infra here at Reddit, and this is an observability gap we are excited to fill. Let us know how you’re continuing to improve your observability, too.

Written by Spencer Koch (u/securimancer), Nathan Handler (u/nhandlerOfThings) and Pratik Lotia (u/wind_lectric)

When you are running dozens of AWS accounts, each with its own legacy OAuth proxy that you can barely track down on GitHub, along with bastions that have not been touched in years, a painfully slow VPN, and the aftermath of a security incident, it becomes clear that something has to change.

This is the story of how we rethought and rebuilt Reddit’s internal access model from the ground up by moving to a Zero Trust architecture aligned with modern best practices, and the unexpected challenges we encountered along the way.

The Legacy Mess

We had duct-taped solutions that worked just well enough for long enough that nobody wanted to touch them. It was a classic "if it ain't broke" situation, except it definitely was:

• Legacy intranet proxies running ancient OAuth2 implementation with policy defined in Puppet that very few actually understood.
• Bastions that were deployed once and then forgotten using Puppet deployed SSH keys that were a pain to manage.
• An in-house VPN that only covered part of what we needed and was set up by an engineer who had moved on years ago.
• All of this duplicated across dozens of AWS accounts.

The worst part was session management. Every service had its own session identifier. When someone got phished and we needed to hit the big red button, there was no big red button. Just a collection of smaller ones spread across different systems, all with their own session lifecycles and bad Ansible playbooks that broke easily.

Then we had a public breach. That got everyone's attention. It finally pushed us to invest in fixing the problem for real.

What We Actually Needed

We knew that collapsing all those ingresses (NGINX, SSH, random VPNs) into a single access layer would cut down a lot of operational toil. We wanted real zero trust with device trust and strong, consistent policies. Not SSH public keys pushed around with Puppet or manually adding people to NGINX auth groups.

But what really sold the project internally was developer experience.

Our developers were spending more time waiting on Docker image pulls than writing code. The root cause was pretty clear. Everything was routed through AWS us-east-1, and those links were saturated because all traffic went through our in-house VPN. We needed a provider with a real global network backbone and points of presence close to our engineers. And our international Snoos experience? Abysmal. 

Logging was another major gap. Bastion logs looked nothing like web traffic logs, and in some cases we were not capturing access logs at all because disks would fill up. There was no consistent way to answer a simple question like “who accessed what, and when?”

We needed unified, reliable logging across everything.

Why Cloudflare?

We ended up choosing Cloudflare for a few key reasons. That said, you could get to a similar place with other vendors.

• A true global network backbone. This ruled out solutions that rely on third-party infrastructure. We needed consistent, high-performance access for engineers regardless of location.
• Built for infrastructure as code. Strong API coverage and Terraform support made it straightforward to integrate with our existing workflows.
• First-class Kubernetes support. We run bare metal Kubernetes across AWS and GCP, so we needed something that fits cleanly into that model without awkward workarounds.
• No phone-home dependency. We did not want access to depend on EC2 nodes establishing outbound connectivity before anything could work.
• Flexible DNS integration. Partial CNAME support lets us keep using Route 53 and avoid a disruptive migration.

The Migration: What Actually Worked

DNS: Partial CNAMEs and External-DNS

We used partial CNAME resolution by appending .cdn.cloudflare.net to our Route 53 records and letting Cloudflare handle the backend plumbing. For example, example.snooguts.net becomes a CNAME to example.snooguts.net.cdn.cloudflare.net. This meant we could keep managing DNS in AWS without migrating everything.

The tradeoff was moving from private to public DNS zones. Security through obscurity took a hit, but we decided we did not care if people knew what kind of infrastructure we hosted. The DNS can be public if the policies are solid.

What actually made this work was external-dns. Instead of requiring separate Terraform PRs for Cloudflare and Route 53, developers just add an annotation to their Kubernetes manifest and DNS entries show up automatically. That enabled a much better developer experience.

Wildcards, Certificates, and Depth

Early in the migration, we leaned heavily on wildcard DNS and wildcard certificates to move quickly.

We created records like:

• *.snooguts.net → *.snooguts.net.cdn.cloudflare.net

and paired them with wildcard TLS certificates so that anything under that domain would resolve and terminate TLS without extra setup.

This worked great for bootstrapping. Teams could stand up services without waiting on DNS or cert provisioning, which helped us migrate quickly.

But it also introduced a problem. People started relying on the wildcard instead of defining services explicitly. That made routing harder to reason about and policies harder to enforce.

We eventually shifted how we used wildcards. Instead of being the default path, they became a safety net.

If a service was not explicitly defined, the wildcard would still catch the request and route it, but in a controlled way. That gave us visibility into misconfigurations and missing definitions instead of silently routing traffic forever. From there, we could fix the service properly and move it onto an explicit hostname and policy.

Another constraint we ran into was hostname depth. A single wildcard only covers one level. So *.snooguts.net works for service.snooguts.net, but not api.service.snooguts.net.

Rather than trying to support arbitrary depth, we standardized our hostname patterns. Keeping things predictable made DNS, certificates, and policy matching much simpler.

Resolver Policies for the Win

We set up resolver policies using regular expressions to route traffic to specific AWS VPCs. Across dozens of legacy clusters, this lets us say “if you’re hitting dev.snooguts.net, resolve it using this VPC’s reserved DNS at 10.X.X.2.” This allowed us to also have flexibility on that public zone publishing, we could leverage AWS’ internal DNS endpoint to query private zone as well, depending on the circumstance.

This ended up being one of the most useful tools we had.

We leaned on it to separate human traffic from service traffic without touching everything upstream.

Kubernetes API servers stayed static. No DNS tricks there.

But CI was different. For anything user-facing, we routed humans through a Cloudflare Access app so we could enforce policy and get proper logging. For services running inside the cluster, we kept things simple and let them talk directly to the 10-dot (RFC1918) address.

That meant we didn’t have to lift and shift the entire CI stack just to get better access controls. We could layer security in for humans and leave service-to-service traffic alone.

Cloudflare Tunnels: Surprisingly Easy

Tunnels were honestly the easiest part of the whole thing. We ran a couple replicas in each account, pulled secrets from Vault, and shipped them like any other Kubernetes service.

The tricky bit was CIDR mapping.

We had years of overlap between dev and test that we never cleaned up. It finally caught up to us. Different clusters using the same 10.x ranges meant the tunnel had no idea where to send traffic.

Virtual networks (vnets) got us out of that. Instead of relying on raw IP space, we could define logical networks and explicitly map each tunnel to the right one. That let us disambiguate overlapping CIDRs without having to re-IP entire clusters, which would have been a much bigger project.

It worked, but it’s not something you want every developer touching. The abstraction is powerful, but the UX still feels pretty infrastructure-heavy.

Scale was the other surprise.

On paper, 30,000 connections per tunnel sounds like plenty. In reality, our dev environment has 1,500 to 2,000 engineers, each working in their own Kubernetes namespace. That’s a lot of traffic funneled through a relatively small number of tunnels.

Then autoscaling made things worse. Our clusters scale aggressively, and tunnel pods were getting recycled every 10 to 15 minutes. That’s fine for stateless workloads, but not great when you’re holding long-lived developer sessions.

We ended up scaling tunnels vertically instead of horizontally. Fewer, larger pods were much more stable and stopped killing sessions.

The Work Breakdown

This wasn’t a quick project. It stretched across multiple quarters and became one of the first real, sustained partnerships between Security and IT at Reddit. The work didn’t fit neatly into team boundaries, and nearly every change affected both orgs, so we ended up working side by side on almost everything. Rethinking access meant changing how every engineer, service, and workflow interacted with infrastructure. The technical work was important, but the harder part was keeping the company running smoothly while we were actively reshaping the ground underneath it.

We had to:

• Migrate hundreds of applications without breaking access
• Rework DNS and routing in a way that didn’t surprise people
• Introduce new clients and access patterns on developer machines
• Replace long-standing behaviors (VPNs, bastions, NGINX auth) that people were used to

None of that works without tight coordination. Security and IT were constantly in the loop together, often pairing on changes that crossed boundaries.

Along the way, we also had to rethink how we handled access logs and telemetry at scale. What started as “just get logs somewhere” eventually turned into a broader effort to rebuild our pipeline entirely, which we wrote about separately in our custom SIEM work.

But the thing that made or broke the migration was communication.

We treated this like a product rollout:

• Regular all-hands presentations to explain what was changing and why  
• Clear documentation and runbooks for both service owners and end users  
• FAQs that evolved as we learned what confused people  
• A steady stream of updates so nobody woke up to a broken workflow without context

Even with that, we still hit rough edges. The difference was that people knew what was happening and where to go when something broke. For a migration that touches every human and every service, that matters more than any individual technical decision.

Automation: Building Operators

Terraform: The Foundation

We’re a pretty typical engineering org in one important way: nobody is clicking around in the console making changes directly.

If you needed a new tunnel, an application, or a policy, you made the change in code, opened a PR, got a review, and let Atlantis handle the plan and apply.

Cloudflare had Terraform support, which got us most of the way there. What it didn’t have was a clear opinion on how to structure things. There wasn’t a reference layout for tunnels, apps, or policies, so we had to figure that out ourselves.

We tried a few approaches and eventually settled on separate Terraform root modules per AWS account. That gave us enough isolation to iterate on one environment without accidentally breaking another, which mattered a lot during the migration.

We also hit a fun surprise halfway through. Cloudflare moved from app-specific policies to reusable policies. It’s a better model, but the timing meant we had to adjust our Terraform while everything was already in motion.

Nothing catastrophic, but definitely one of those “learn it the hard way” moments.

The Operator Inflection Point

At the same time as the migration, we were rethinking how we run Kubernetes. In the new model, clusters are disposable. We create them when we need them and tear them down when we don’t. That falls apart fast if every new cluster depends on a human to wire up tunnels, routes, and configuration by hand, so we needed something that could keep up. We moved that logic into Kubernetes operators. Terraform gets us to an initial state, but operators handle continuous reconciliation. If something drifts or a new cluster shows up, it gets fixed automatically. We built two operators using our open-source Achilles SDK.

CF Tunnel Operator
Handles the plumbing for new clusters end to end. It deploys tunnel pods, wires up routes, pulls secrets, and configures the Cloudflare side automatically. Our older setup had around 15 clusters that were created manually with some Terraform help. In the new stack we’re at 25+ clusters, and all of them come up with working tunnel configuration without anyone touching it.

CF Application Operator
Lets developers define what they want directly in their service repo instead of going through Terraform. Our generators turn that into a Kubernetes manifest, and the operator takes care of creating the Cloudflare application, policies, and routing. The big win is feedback and ownership. Developers can look at the App object in Kubernetes, immediately see if something is broken or misconfigured, and fix it right there instead of waiting on us.

Split Tunnel vs. Full Tunnel

We chose a split tunnel model.

Yes, we know how that sounds. If you work in security, you’re probably already side-eyeing this decision. But at the time, we were already changing how people accessed internal services and asking teams to migrate critical systems. Layering endpoint changes on top of that would have been a breaking point for the org.

The reasons were pretty practical:

• Full tunnel broke things people rely on day to day, like AirDrop and various Bluetooth workflows at home.
• Reddit runs on IPv4 10-dot CIDRs (for now), which collided with a lot of home networks and caused routing issues.
• We didn’t yet have a clean way to correlate Okta identity signals with Cloudflare logs, so we couldn’t answer basic questions like “where did this login come from?” with confidence.

So we made the call to keep the blast radius smaller and go split tunnel.

That said, this wasn’t meant to be permanent. Now that we have a proper security data lake and can join identity and network telemetry the way we originally wanted, we’re actively moving toward full tunnel.

Looking back, split tunnel bought us time to get the foundations right without overwhelming everyone at once.

Nuking Legacy Infrastructure

One of the most satisfying parts of this whole effort was deleting things.

We were pretty firm on this from day one: if the old stack was still running, the migration wasn’t done. Keeping both around would just double the complexity and confuse everyone.

VPN & Auth
We moved access control into Cloudflare using Okta groups, but we kept the policies intentionally broad. Most engineers can reach Cloudflare apps at the edge, and each application handles its own authorization. That split between edge access and app-level auth was deliberate.

That let us avoid creating and maintaining a huge number of fine-grained IdP groups, and it gave teams flexibility to build auth that fits their service.

And yes, we shut down our in-house VPN that was running on Pritunl.

SSH & Bastions
We removed almost all bastions across our AWS accounts. Instead of hopping through a jump host, engineers connect through Cloudflare tunnels directly to nodes. It’s simpler to reason about, and session termination is much more straightforward when someone leaves the company or loses access. Today we keep one deployment of SSH Bastions as a breakglass in case Cloudflare WARP or Access goes down, and we can SOCKS proxy traffic through those bastions in a pinch.

Intranet Proxies
We migrated proxies incrementally to keep the blast radius small and let teams work in parallel.

We turned on NGINX access logs and used them as a signal. Once traffic for a service dropped to zero, we knew it was safe to cut it over. Along the way we cleaned up DNS that had drifted into the wrong zones, wired everything into external-dns, and unwound years of wildcard shortcuts.

After the first handful of services, the same issues kept showing up. Once we recognized the patterns, migrations sped up a lot.

Webhooks and Service Auth

Webhooks were one of the more annoying edge cases.

GitHub webhooks can’t add custom headers, which meant we couldn’t use our normal service auth pattern. We ended up allowing bypass auth on a small set of endpoints. Not ideal from a visibility standpoint, but GitHub signatures still gave us a reasonable level of validation. We validated that receiving services for those webhooks were using the webhook signing secrets to confirm identity.

For service-to-service traffic, we went the other direction and tightened things up. We added custom JWT middleware and used Cloudflare service tokens.

That gave us proper attribution, consistent logging, and a much better story than wide-open bypass rules.

Incidents and Breakglass

The Great Tunnel Wipeout

One Friday morning, all our tunnel routes disappeared. Every single one.

Thankfully Terraform saved us. We ran terraform apply across all root modules and everything came back. Then it disappeared again. And again. Apply, recover, delete, repeat.

The issue traced back to a change we shipped the night before. We added cleanup logic to the CF Tunnel Operator so it would delete tunnels when clusters were torn down. Sounds reasonable. The problem was a bug in the Cloudflare API. When a specific route lookup failed, the API returned all routes instead of none.

Our operator saw that response and did exactly what we told it to do: clean up.

On the next reconciliation loop, it had the full list of routes and deleted them all.

We rolled back quickly. Terraform restored the legacy clusters, and the previous operator version brought the newer clusters back within minutes.

Lesson learned: be very careful with cleanup logic in a continuously reconciling system. Also, invest in a real test environment early. We added unit tests and much better safeguards after this.

Breakglass

We kept a single, heavily locked-down bastion around for breakglass scenarios.

Nobody wants to hear “Cloudflare is down, so production is inaccessible.” Even if the provider is reliable, you still need an escape hatch.

In an outage, engineers can request temporary access via SSH certificates and short-lived group membership. From there, they proxy into internal services.

It’s not meant to be convenient. Access is tight, everything is audited, and you only use it when things are actually broken.

When Cloudflare has issues, we handle it like any other external dependency. Page the right people, communicate clearly, and work it like a normal incident.

The Sharp Edges

Not everything was smooth. A few things still hurt.

• gRPC doesn’t work with Cloudflare Access apps. This is probably our biggest gap. Most of our newer CLIs speak gRPC and protobuf, not REST. That mismatch shows up quickly, and it’s only becoming more common across our tooling.
• No load balancing across backends. If your service runs in multiple regions, you have to point the Cloudflare app at a single one. That’s not great for HA or active-active setups. We’ve had to be deliberate about which region we pick, and it’s a real limitation.
  • We’re working on adding load balancing support here.
• Caching behavior surprised people. We saw reports of stale responses or traffic not behaving the way developers expected. In practice, we had to standardize on clean, RFC-compliant cache headers and in some cases just turn caching off to avoid confusion.
• Wildcards caused their own problems. Early on, people leaned on them to avoid doing the proper Terraform or operator setup. That worked until it didn’t. During the migration, we shifted to treating wildcards as a safety net. Now if something hits the wildcard, it’s basically a signal that the service wasn’t configured correctly.

What We'd Do Differently

A few things we’d change if we were starting over.

• Pay down tech debt first. We knew about the rough edges: inconsistent access patterns, wildcard overuse, overlapping CIDRs. We hoped to clean it up along the way. That didn’t really work. It just showed up later in more painful ways.
• Test breakglass earlier and more often. During the June 12th Cloudflare outage, parts of our breakglass path didn’t behave the way we expected. We fixed it quickly, but it was a good reminder that “it should work” isn’t the same as “we’ve actually exercised this end to end.”
• Push harder on consistency across environments. Different generations of infrastructure meant different assumptions baked into each environment. That led to a lot of one-off surprises during migration. Standardizing earlier would’ve saved time.
• Plan for incidents. Even with careful rollout and testing, things will break. That’s part of a migration at this scale. Getting leadership aligned on that upfront made it easier to respond when it happened.

The Wins

This was a long migration with plenty of sharp edges, but the end state is a lot better.

• Unified logging across everything. Bastions, proxies, apps all land in one place with a consistent format, which makes debugging and investigations much easier.
• Real device trust and posture checks. Not just “did you authenticate,” but “is this a managed, healthy device.”
• Much better developer experience, especially for teams outside the US. No more routing everything through us-east-1 just to pull images or hit internal services.
• Simpler operations. One ingress layer instead of a pile of proxies, VPNs, and one-off access paths.
• Centralized session control. When something goes wrong, we can actually revoke access in one place instead of chasing sessions across a dozen systems.

What's Next

We’re rolling out full tunnel now, starting with Security, IT, and a few early adopter engineering teams. Two years ago it would’ve been too disruptive. Today the foundations are there and it’s the right next step.

The harder part isn’t the rollout, it’s support. WARP breaking on hotel WiFi is a real thing, and telling people “file a ticket” isn’t a solution. We’re building better self-service flows so people can temporarily disable and recover without getting stuck.

We’re also working through device certificate deployment on macOS and Linux, which is required for stronger device identity. That’s been more operationally complex than expected.

Finally, SaaS apps. Moving internal apps was one thing. Moving vendors like Figma or BrowserStack behind Cloudflare is another. Every provider handles OIDC differently, and it’s still a lot of manual setup and testing. The payoff is worth it, but it’s slow, steady work.

The Bottom Line

Moving 200+ apps across several dozen AWS accounts to a zero trust model, without slowing down ~2,000 developers, is not something you knock out in a sprint. This was a multi-quarter effort across Security and IT, with a lot of iteration along the way.

We broke things. We fixed them. We built operators so we never have to do this by hand again. And we ended up with something that actually looks like zero trust, not a pile of proxies, bastions, and crossed fingers.

If you’re thinking about doing this:

• Pay down tech debt early. You’ll deal with it either way.
• Put everything in Terraform from day one.
• Test your breakglass flows before you need them.
• Invest in operators once you start scaling.
• And seriously, fix your CIDR overlaps first.

It’s a lot of work, but once you’re on the other side, there’s no way you’d go back.

If you are interested in learning more about our Zero Trust Odyssey, we presented on the topic at Cloudflare Connect and SREcon26 Americas.

Ten years ago, I came back to Reddit. Twenty years and change ago (October 2005, if we're being exact), I got a call from u/spez while I was grabbing coffee with a labmate. Paul Graham had suggested he hire me as Reddit's first engineer. I said yes before I hung up the phone.

This week, I’ve decided to step down as CTO and take on a new role as Reddit's first Senior Technical Fellow.

The last decade has been the honor of my career. We took a company a lot of people had written off and rebuilt it: the stack, the org, the culture, all while it was still in flight. Along the way, we built something the world uses—something millions of people rely on every day to learn, to connect, and to understand what’s happening around them.

We went public. We found our footing in an AI-native internet without chasing every trend, and I’m even more optimistic now than I was then. The opportunity ahead for Reddit (and for the kind of internet we believe in) is bigger than ever. The right bet is still the one we made: be the source worth surfacing.

Amit Puntambekar (u/zerowaitstate55) is stepping into Reddit’s CTO role, and I couldn't be more confident about where engineering is headed under their leadership.

None of it was mine alone. Reddit has one of the best engineering teams in tech, and being in the room with them has been a gift. To everyone who built this with me over the last ten years and in the early days before all of it: thank you.

More to come.

Written by Roman Levitas and Tim Zhu.

TL;DR

The operational pitfalls of Kubernetes sidecars are well-documented: resource limits that quietly throttle your app, scaling constraints that force wasteful over-provisioning, and cascading failures that are maddeningly hard to diagnose. At Reddit, we ran headfirst into all of them—with our experimentation infrastructure, of all things.

Reddit's Decider SDK is the system that powers experiment bucketing (deciding which users see which variant), exposure event emission, and experiment configuration retrieval across Reddit's services. For Go services specifically, the SDK's architecture relied on three sidecar containers to handle those responsibilities, and under Reddit-scale traffic, that architecture started to crack.

This is the story of how we built Native-Go Decider (NGD): a pure-Go reimplementation that moved core logic in-process, dropped two of those three sidecars (with the last one on the chopping block), and delivered dramatic improvements in latency, efficiency, and cost.

A Quick Primer: What Are Kubernetes Sidecars?

For those less familiar, a Kubernetes sidecar is a secondary container that runs alongside your main application container in the same Pod. Both containers share the same network namespace and storage volumes. It's a common pattern for offloading auxiliary concerns like logging, secret management, service mesh proxies from your business logic.

Sidecars have real benefits: modularity (decouple operational concerns from app logic), reusability (one image, many services), maintainability (version and update independently), and consistency (platform teams can inject them automatically).
At Reddit, sidecars are a first-class pattern: our service framework baseplate.py defines several that handle concerns like secret management, live configuration retrieval, and event emission. The Decider sidecar, responsible for experiment bucketing in Go services, is the one this post is about.

How We Got Here: Sidecars and the Go Decider SDK

Reddit's Experimentation SDK was designed around a compelling idea: centralize bucketing logic in Rust, then create language bindings for Python, JavaScript, and others. One Rust implementation, consistent behavior everywhere—fewer bugs, easier feature rollouts, a single source of truth.

The problem? Golang doesn't support C interop in the way we needed. Cgo exists, but it's not used at Reddit for good reasons (garbage collection headaches, performance overhead, frequent context switches). So we introduced the Decider sidecar. It is composed of the Rust Decider SDK wrapped in a gRPC server, called by Go services via a gRPC client when bucketing experiments.

At the time, adding another sidecar to our Decider SDK that already had 2 sidecar dependencies, Live-Data (pulling experiment configs from ZooKeeper/S3) and Event-Publisher (emitting exposure events), didn't seem so bad. But as traffic grew, we discovered this architecture doesn't scale well.

So… What's Wrong With Sidecars?

Even the official Kubernetes blog comes with a warning: "Proceed with caution if resource efficiency is your primary concern, minimal network latency is critical, simpler alternatives exist, or you want to minimize troubleshooting complexity."

We learned these lessons the hard way.

Resource Management Nightmares

Sidecars require their own CPU and memory limits, and when those limits are wrong (or traffic patterns change over time/spike) things get ugly fast. We hit both failure modes:

• OOM CrashLoopBackOff: The sidecar exceeds its memory limit (e.g. as the total number of experiments at Reddit grew over time), the Pod crashes, alerts fire, and you're scrambling.
• CPU throttling: The sidecar hits its CPU request ceiling, contexts get cancelled, error rates spike—but intermittently, which makes it incredibly difficult to diagnose unless you specifically know to check sidecar resource utilization.

Figure 1 below shows a CrashLoopBackOff we encountered, and Figure 2 shows the Decider sidecar's CPU utilization spiking well beyond its allocated resources.

state:
  waiting:
    message: back-off 5m0s restarting failed container=...
    reason: CrashLoopBackOff
...
terminated:
  reason: OOMKilled

Figure 1: A wild CrashLoopBackOff appeared! Caused by a Decider sidecar OOM when exceeding its memory limit.

Figure 2: Grafana showing Decider sidecar CPU utilization at 376%—well beyond any reasonable request ceiling.

The Scaling Wall

Sidecars are fundamentally 1-to-1 with their Pod. They can't scale independently of the main container. If your application adds more Decider callsites and traffic increases, the sidecar can't just spin up more replicas to accommodate the load—it's stuck sharing the same Pod resource envelope.

This became a real problem. Our Decider sidecar experienced gRPC traffic saturation at 35K RPS, which was the root cause of a major incident. The only mitigation was to manually scale the entire service by increasing the minimum number of pods just to dilute traffic to each sidecar. Extremely wasteful.

Figure 3: An accurate depiction of Josh Quack, our Duck Duck Goose (DDG) experimentation platform mascot, debugging sidecar resource exhaustion.

Native-Go Decider (NGD): How We Got Out

This left us with one choice: keep patching a brittle architecture or rewrite the bucketing logic natively in Go. We chose the latter.

Native-Go Decider (NGD) is a ground-up reimplementation of the Decider SDK in pure Go. This was a carved-out exception to our "centralize in Rust" philosophy, and it required extensive parity testing to ensure correctness with other Rust-based Decider SDKs:

• TapCompare running NGD in shadow-mode alongside the legacy sidecar to compare bucketing results in production traffic via metrics
• A Playground environment for manual validation
• Randomly-generated experiment configs in CI to stress-test edge cases

The key architectural differences:

Figure 4: Go Decider Sidecar evolved into NGD!

By moving bucketing in-process, we eliminated the gRPC hop entirely. No more network serialization overhead, no more sidecar resource management, no more traffic saturation at scale.
A new gRPC Event Collector allowed us to emit Expose events in batches without relying on the Event Publisher sidecar to do so via HTTP. Experiment configurations are also pulled down via the same logic as in the Live-Data sidecar, but rewritten as part of NGD natively.

The Impact: Numbers That Speak for Themselves

Our primary GQL backend was the first major service to fully migrate to NGD, and the results were substantial.

Performance:

98% p50 bucketing latency reduction!

• Figure 5: Bucketing latency for NGD (99 μs) vs legacy Decider sidecar (40.8 ms) 

Efficiency:

• >90% cache hit rate via a Request-Level Deduper that prevents redundant bucketing within the same request 
  • >90% reduction in unnecessary CPU usage
  • >90% reduction in Expose event emission

Cost:

• 30% in projected annual cost savings across GQL from dropping the Decider and Event-Publisher sidecars

Figure 6: NGD was super effective! Josh Quack after seeing the migration results.

What's Next: Dropping the Last Sidecar

Two sidecars down, one to go. The Live-Data sidecar, which currently pulls experiment configs (the Manifest) from ZooKeeper/s3 and onto the pod's filesystem, is next on the chopping block.

The logic from Live-Data sidecar has been re-implemented in Go as part of NGD and will:

• Add resilience with an S3 fallback in case ZooKeeper becomes unavailable
• Resolve a startup race condition that currently causes pod restarts
• Remove the last sidecar dependency, simplifying the Decider architecture and onboarding process

This functionality is currently being tested as an opt-in option in the NGD config, with careful monitoring to compare behavior against the existing sidecar prior to full cutover.

Figure 7: Our roadmap for the remaining sidecars, visualized.

In Closing

Reddit's Go experimentation SDK relied on three Kubernetes sidecars that introduced resource headaches, scaling limitations, and troubleshooting complexity at Reddit-scale traffic. We replaced that architecture with Native-Go Decider (NGD) moving bucketing in-process and event emission to direct gRPC. This eliminated two of three sidecars, with work underway to drop the last. The result is a dramatically faster, leaner, and more reliable experimentation stack, now powering over 70 Go services at Reddit.

Sometimes the best sidecar is no sidecar at all.

Written by Chris Slowe and Lisa O'Keefe

Happy 5th birthday, r/RedditEng!

Huge thanks to the Reddit engineering team for building, scaling, and sharing so much great work here over the years. This subreddit has become a rare corner of the internet where people can go deep on real systems, real tradeoffs, and the messy, interesting work behind running Reddit.

From infrastructure and reliability to developer productivity, ML, security, and all the delightfully weird engineering problems in between, the posts here have been consistently thoughtful, generous, and fun to read.

Thanks for five years of awesome posts, hard-won lessons, and excellent engineering. Here’s to the next five.

And of course, for the readers and commenters, as we plan out those next 5 years: what do you want to see more of? 

Written by Nazareno Lorenzo

As an engineer you learn a lot from building, but I believe you learn exponentially more from breaking things. We have a saying in the country where I grew up: Those who burnt themselves with milk, see a cow and cry. 

If you can expand this and learn not just from your own mistakes, but from the mistakes of your entire company, you multiply your learning opportunities and unlock a path to becoming a stronger engineer, faster.

This is where incident reviews come in as a powerful tool. In a company with hundreds of engineers, without a shared learning process, we would be making the same mistake hundreds of times.

What exactly is an Incident Review?

At its core, an incident review is a structured conversation that happens after an outage, a system failure, or a major bug. It's a dedicated time for the team to get together and dissect what went wrong. This is also called an Incident Postmortem.

This can take a few different forms:

1. An informal meeting with the members of the team responsible for the system to discuss it.
2. A document prepared by one or more contributors, often following a template. 
3. A company wide process, where the affected system owners and reliability experts at the company discuss the incident.

At Reddit, we use a combination of all of the above. It is important to find the right balance: If we invest the same amount of time on every issue we notice, the process quickly becomes tedious. 

For an incident where the impact was small and we have already identified enough actions to take to avoid it repeating, we may not need an intensive review process. For larger or more unclear outages, we follow our more structured processes to help us get all the answers we want. And, when we believe it’s of value to the industry broadly, we share our insights publicly. For example, this post, the Unseen Catalyst: A Simple Rollout Caused a Kubernetes Outage.

What do we want to answer?

The goal of this process isn't to point fingers or assign blame. Instead, it's an objective look at the timeline of events before, during and after the incident in order to prevent, detect and mitigate similar issues in the future.

Caption: Incident Postmortem Template Document

During the Incident Review process, we want to get answers to a few questions:

1. What happened? (The factual timeline of the incident).

In order to enable a good discussion about what happened, it is critical to understand and document clearly the timeline and facts.

A good way to do so is by building a detailed timeline of what happened. Some suggestions of things to include:

• Relevant charts showing the impact.
• Steps taken to resolve it.
• Automated alerts triggered.
• Links to all related pull requests.
• How responders got engaged

2. Why did it happen? (The root causes and contributing factors).

Generally, by the time we resolve an incident we have an identified root cause (what actually tipped the system over): a bug in code was released, some system got overloaded, etc. If not, we should investigate it in detail; try to reproduce it in some non-production environment; or even use our fault injection framework to artificially recreate failures and delays between any two services.

After finding that, we should try to identify all contributing factors. A frequently used technique is Five Whys. I prefer to think of a few areas separately, using questions similar to these:

• Testing and CI/CD:
  • Did we detect this issue while developing?
  • Did the contributor have good tools available to test this easily?
  • Do we have any way to detect this automatically when a PR is created? 
• Release:
  • Was this detected during deployment before reaching most users? (e.g. using canary deployments, progressive rollouts, experiment flags, etc)
  • Was it reverted automatically? 
• Alerting:
  • Did we learn about this through automated alerts or through a user report?
  • Could we have learned about this faster?
  • Did the right owner get notified?
• Graceful Degradation:
  • Did other systems handle the outage in the best way possible?
  • Can we add fallback mechanisms so we serve a better degraded experience?
• Incident Behaviors:
  • Were we able to bring in the right people to help with the incident quickly?
  • Did we identify the root cause easily through our monitoring?
  • Did we have good visibility of what changed at the time the incident started? (e.g. deploys, experiments, etc)
  • Did the responders know how to run the necessary remediation steps or where to find runbooks/documentation for it? Were they blocked at any point?

Each of these questions is interesting enough to write a separate post about, and the exact list will need to be calibrated to the shape and maturity of the platform you are working on. A two-person startup won't have or need the same infrastructure as a tech giant, but you can always find the best next step to improve.

3. How do we stop it from happening again? (The actionable steps to improve the system).

We can now put all the investigation we did above to use and define specific Action Items: follow up tasks that will help us avoid repeating this incident. They should be focused on short-term mitigations. If an incident sparks a six-month architectural redesign, that's a great discussion to have, but it belongs on a roadmap, not as a quick incident action item. 

A team of Google SREs wrote in a ;login: magazine article, "Postmortem Action Items: Plan the Work and Work the Plan," the properties that a good action item should have:

Actionable: Phrase each action item as a sentence starting with a verb. The action should result in a useful outcome.

Specific: Define each action item's scope as narrowly as possible, making clear what is in and out of scope.

Bounded: Word each action item to indicate how to tell when it is finished, as opposed to open-ended or ongoing tasks.

Following those, some examples could be:

What did I learn from this?  

Mistakes are unavoidable, and I don’t feel bad making them

I have been coding (and sometimes breaking things) for around 20 years, at a variety of companies with widely different approaches to this. At Reddit, I have participated in at least 40 incident review meetings and contributed to many more documents.

I worked at a company where all engineers were afraid of making mistakes, as we knew the reaction from our leadership would be harsh. If I shipped a bug to production, I would have tried to quietly fix it before someone else noticed. 

That, clearly, didn’t help me make fewer mistakes. I patched my bugs, maybe even trying to sneak the fix as part of another change; but without adding any guardrails to stop me or others from making that same mistake. Sometimes my rushed fix attempt made things even worse.

I’m convinced that no process should ever depend on humans not making mistakes. Even more so, no system should ever depend on a single component not failing. 

Following that idea changed how I feel when I break something. It made it easier to shift the scenario in my mind from “Oops, I f#@’d up” to “Oops, this is fragile. Let’s improve it”. And that has made a massive difference to my psychological safety.

Unsurprisingly, fear doesn’t do much for systems reliability. I did not break things more frequently once I stopped moving with fear, but the opposite. 

Not all technical debt is the same

You can find a lot of posts online (for example, in the r/programming or r/ExperiencedDevs communities) from people complaining about technical debt growing as other things get pushed on top of the development team’s priorities.

Incident reviews can help you prioritize migrations or refactors. In the last couple years, I helped drive a large effort to rebuild a part of Reddit’s posts and comments creation backend. Tracking and referencing incidents in this area was very useful to help defend the importance of this work, and to later measure the results.

There will always be some tech debt (I will always look at code I wrote a year ago and think “Who wrote this?”). But there’s a big difference between code that I dislike and code that has contributed multiple times to incidents that impacted our users.

It’s not unique to software engineering

Outside of engineering, my biggest passion is skydiving; and the approach we take to safety is surprisingly similar. Safety in skydiving is built by redundant layers of tooling and processes.

The sport is growing, with new competitive disciplines and people pushing the limits of bodyflight. But the statistics show something very clearly: skydiving keeps getting safer through the years. Humans are as likely to make mistakes as before, but our training, processes, and equipment keeps adapting, learning from the mistakes of the past.

We still make mistakes all the time, but we plan ahead to reduce the impact of those. When we identify a dangerous situation, we talk about the multiple contributing factors that led us there, review our videos, and often write and share incident reports. 

Focusing on response helps fix things faster

When a service is down and alarms are firing, someone may be tempted to jump and ask: "Wait, why wasn't this caught in our tests?" or "Who approved this change?". 

An advantage of having an established incident review culture is being able to table those discussions until after the pressing issues are resolved. I have more than once said: "That is a good question. Let's note it down for the incident review and focus on getting the system back online now." 

What can you do?

The answer will depend on your company.

Dedicate part of your time to learn from other people’s incidents

If your company already has a strong culture of incident review, try to learn as much as possible from it. I think it is one of the most valuable uses of time for any engineer. This is one of your best tools to learn what patterns work well, what common sources of mistakes are, what has been missed in the past.

If there’s shared incident reports somewhere, try to read them. If there are incident review meetings, ask to attend them, even as a silent spectator. 

"But my company is small and we don't do these things!"

If you're a new engineer at a startup with 15 people, you might find that there aren’t any processes for that, or any documents to consume. 

The good news is that anyone can start this. Next time you break something, you can flip it into a good opportunity to show engineering leadership. You don't need a heavy framework. 

• Write a small doc, email or even chat message. Reframe the narrative from, "Oops, I broke the database," to "Oops, our system experienced an issue, here is what happened, and here is what I learned about how we can avoid it." 
• If people are interested enough, set a meeting to discuss it; or ask your manager for 15 minutes during the next weekly team meeting.

P.S. Our process wasn’t always what it is today. Take a look at r/shittychangelog for a look at how we used to document some of our incidents and how far we’ve come.

If you’ve been following this blog for a bit, you’ve almost certainly heard us mention Snoosweek before. Last year, we shared a judge’s perspective on the festivities. Today, we’re pulling back the curtain to share what went down at our most recent Snoosweek and give you a look at it all comes together.

I'm new here - what is a Snoosweek!? 

First off, welcome!

Snoosweek is Reddit’s internal hackathon week. We encourage employees to step away from their day-to-day responsibilities and pursue any project that sparks their interest. It’s a dedicated window for creativity, innovation, and cross-functional collaboration. We host two Snoosweeks each year, one in Q1 and one in Q3.

Whether it’s addressing long-standing technical debt, building dream features, or brainstorming future Reddit, Snoosweek empowers employees to explore their boldest ideas. The best part? It brings people together from all over the company - technical or not, everyone is encouraged to participate. This fosters amazing creativity and leads to many ideas that actually make it into production! 

Lights 💡

A team of volunteers (including the crew that runs this blog!) begins planning for Snoosweek months in advance. From marketing the event, recruiting judges, preparing the demo deck the night before - a massive amount of coordination happens behind the scenes to ensure the week, and the final Demo Day, runs smoothly.

Camera 📸

Snoosweek is here! Teams work Monday through Thursday prototyping, testing, and making demos of their wildest ideas. This year, we set a record for the number of projects submitted prior to the week. Even more impressive? Our "completion rate" hit an all-time high, with 88% of those projects reaching the demo stage!

Action 🎬

Snoosweek wraps up with a high-energy, multi-hour session hosted by our Chief Technology Officer (CTO), Chris Slowe.. Everyone watches the 60-second demos together (either virtually or in one of our global offices). And, in true Reddit fashion, high-velocity shitposting in the internal chat during the demos is not only allowed but is encouraged! 

Wrap 🏆

Following Demo Day, our wonderful group of judges evaluates the demos and selects winners for eight distinct awards. 

Snoosweek is one of our most beloved traditions and a cornerstone of our company culture. Beyond the tangible benefits we've highlighted, it’s an incredible opportunity for our Snoos to connect and collaborate with colleagues beyond their usual teams. It is amazing to see such a tradition be able to continue and thrive while scaling a company. It is a true testament to what the week means to our culture! 

Written by Spencer Koch

The Problem Space

Dependency management is hell. Internal dependencies are hell. Knowing when to upgrade, what to upgrade, how to upgrade - it's all hell. We won’t argue about the value of bumping dependency versions, that’s been done plenty of places on the internet. Instead we’ll focus on the overall dependency management process and how that proverbial sausage is made. So how can we, as the Security and Developer Experience teams, help make it a little easier? Well let's take a journey...

What Didn't Work...

Dependabot

So Reddit's codebase is largely in a Github Enterprise Server instance that we run in our AWS account. This is the first inflection point, because we miss out on some of the 'niceties' of Github Cloud. We don't have Github Advanced Security (GHAS) licensed. Which means we're a bit limited in terms of functionality that we have access to. When we started our dependency management journey four years ago, we had access to a rudimentary Dependabot version. It had to manage it through the Github console, and didn't have a lot of options for customization / opinions. Heck, auto PR creation didn't even exist back then. We were largely turned off on Dependabot back then, and Renovate was a more familiar tool in the open source community that our developers were aware of and had been exposed to. 

Looking back, we'd likely have a similar opinion. GHAS makes some decisions that don't work for Reddit and our workflows (we could argue about their secret management decisions here). Having the ability to customize and tailor the experience to our needs continues to be a strong requirement for us, especially now in a world of AI assistants and dev velocity. We want to be able to make decisions about what to upgrade, when to upgrade, and how to upgrade. We want to be able to customize the experience to our needs. We want to be able to prioritize internal dependencies over external dependencies. We want developers to own and control their dependency destiny, with a Security team that provides the tooling and looks at the overall governance model. We're just really picky - so a self-hosted option it is.

And obligatory reference to Renovate's comparison.

Snyk

We had a brief foray into Snyk several years back which worked for a while when our team was still relatively small (1.5 appsec engineers), before the Renovate hop and introduction of OSV. Our largest complaint there was the poor API interface and the flakiness of the service. It didn’t have the knobs like Renovate either, but we were paying for support / compute. Unfortunately, the developer experience just didn’t pan out and the complaints from appsec and engineering grew too great. Looking back, we’d likely replace it with Renovate anyway because of the configuration requirements and customization. 

Requirements

A major challenge, regardless of tool, is the fact we have several internal dependency registries that need to be considered for network line of sight and access management. We've got an Athens Goproxy and Artifactory playing host to all our other registries (pypi, npm, docker, Scala, Maven, others). So having a tool deployed in a specific AWS VPC and ability to inject credentials is a must. 

On top of that, we want to provide knobs and levers for the behavior of the dependency management system. Because we have teams of various sizes, languages, and workflows, allowing a decentralized configuration for the behavior of a dependency management system is a must. We also want the ability to provide high level requirements (like how prioritization of security related or internal library dependency versioning is done, or how to safely group internal dependencies). So a config that can allow the union between a global config and a decentralized config is a must.

And lastly, we needed the ability to execute custom post upgrade actions on dependency bumps. Reddit's internal DevOps library called 'infrared' (which has APIs for definition of Kubernetes manifests, ownership and service composition info, Drone CI and Dockerfile config generation, and more), requires a re-generation step after version bumps of that library and its subcomponents to catch any underlying APIs change, and so being able to securely and accurately execute that so that dependency bumped PRs aren't broken when a developer gets to them is a must. 

So much like a normal Redditor, we had opinions and wanted things to be done our way. 

How We Do It

So enter Renovate. We actually use a combination of Renovate OSS CLI and Mend Renovate Community Edition, so we'll talk about both. Both run in our CI AWS account and Kubernetes cluster. We have a global Github app across our handful of Github organizations. We'll dive into the various configurations for each component below.

It's worthwhile to briefly mention how we rolled this out as well over the past 4 years. This started as an experiment by one security wizard with an explicit opt-in, meaning the Github app installation was the envelope that controlled what repos were in scope for execution, with Renovate running on all the repos that it could "see" - if the repo wasn't onboarded, go thru the onboarding Issue creation and use a default standard config. This allowed us to control the scope of the experiment and roll it out to a small number of repos before we were ready to go full blast, experimenting with configurations and schedules and general developer experience. This also meant it was high toil to add repos in. At the point that we had teams ASKING for this capability, we inverted the logic - we added the entire org into the Github app installation. But, we changed up how we did onboarding - we wouldn't process the repo unless a Renovate config file was present. This was intentional to limit unnecessary work by Renovate, and coupled with our 'infrared' DevOps library being opinionated about how to generate the Renovate config. We'll get more into that, but this change in approach was key in our ability to launch this to the entire org.

Configurations

First we should talk about our configuration approach. We have a global config that is always inherited by the local repo's config. This utilizes the config preset functionality, so in our primary Github organization we have a renovate-config repo (with secondary Github orgs pointing to this primary org's config, and exposing any org specific configs we might have). This repo contains a few things:

• CI to check config validation provided by Renovate via renovate-config-validator with the --strict flag set on each config.json file.
• A default.json that is our global config entrypoint. This contains globally true behavior like Issue construction, PR behavior, npmrc config, custom regex managers that are globally true (like our Artifactory docker pullthrough cache), or our infrared postUpgrade tasks.
• A slew of other repeatable configs that a repo might opt into using via the extends functionality - groupings, custom datasources and managers, language specific configs. 

Then in our repos, we have a code-generated renovate.jsonfile that contains multiple extends directives and ignores paths based on how the repo is configured. Here’s an example:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "description": "Code generated by infragen. DO NOT EDIT.",
  "extends": [
    "local>reddit/renovate-config",
    "local>reddit/renovate-config//infrared_v2",
    "local>reddit/iam-renovate-config"
  ],
  "ignorePaths": [
    "**/.reddit/**",
    "**/infrared/**/*.tf",
    "**/node_modules/**",
    "**/npm-offline-cache/**",
    "**/vendor/**",
    ".drone.yml",
    ".github/renovate.json",
    "AGENTS.md",
    "Dockerfile.all",
    "Dockerfile.consumer",
    "Dockerfile.grpc",
    "Dockerfile.serviceauth",
    "Dockerfile.session",
    "Makefile"
  ]
}

I should also mention some config options that we've found real interesting/helpful:

• osvVulnerabilityAlerts - we use OSV in our Code Scanner as well, so being able to align those detections with Renovate has been great. It's still tagged as "experimental" by Renovate and we're really loving it, so hopefully it's here to stay. 
• dependencyDashboardOSVVulnerabilitySummary - expose those CVEs to devs since they're already working with the Renovate Issue? Sure, yes please.

• packageRules.prPriority - we defined internal dependencies via matchPackageNames that need to be escalated in priority to be opened since we have a limit on how many Renovate PRs are opened at any point in time for that repo, to not DoS our developers.

  {   "$schema": "https://docs.renovatebot.com/renovate-schema.json",   "description": "Reddit: Prioritize PR creation for Baseplate related things",   "packageRules": [     { "prPriority": 11, "matchPackageNames": ["/^{.infrared./"]} },     { "prPriority": 10, "matchPackageNames": ["/^{.baseplate./"]} },     { "prPriority": 9, "matchPackageNames": ["/^{.drone-plugin-./"]} },     { "prPriority": 9, "matchPackageNames": ["/reddit-go/"] }   ] }

Scheduled Execution Cronjob

This is the meat of the operation. We need to run the Renovate CLI over the ~2700 repos that are in scope today. When we had 150 repos, we were able to JUST use the webhook job and didn't need this. As we evolved our scope and size, the webhook couldn't scale vertically (and the horizontal scaling is locked behind their Enterprise offering). We're good at Kubernetes, so we can solve this with some fancy cronjobs and parallelism. 

Our k8s cronjobs then take this rough shape:

• One cronjob per organization that discovers repositories with Renovate enabled and writes them to a file (using https://docs.renovatebot.com/self-hosted-configuration/#writediscoveredrepos) that we stuff into S3 for consistent retrieval via repo.json
• One cronjob per organization with completions and parallelism that break up the repo.json into distinct chunks
• Bespoke cronjobs for some of our snowflake monorepos that "take too long" to execute in the workload above

We drew inspiration from https://github.com/renovatebot/renovate/discussions/13172#discussioncomment-2341331 discussion by hooking the JS functionality of Renovate to dump a JSON file that we toss up into an S3 bucket. We expanded on that concept by also manipulating the JSON file based on the Kubernetes JOB_COMPLETION_INDEX that comes from the k8s CronJob completions and parallelism concepts. So we have a Docker image that has this customization applied to the config.js that Renovate starts with:

const fs = require('fs');
if (fs.existsSync('/home/ubuntu/repos.json')) {
  // Load all repositories from the file
  allRepositories = JSON.parse(fs.readFileSync('/home/ubuntu/repos.json'));
  allSize = allRepositories.length;
  // Check if we're running in a parallel job (using JOB_COMPLETION_INDEX and JOB_COMPLETIONS)
  // or in a dedicated repo job (without these variables)
  if ('JOB_COMPLETION_INDEX' in process.env && 'JOB_COMPLETIONS' in process.env) {
    // Standard parallel job processing
    const segmentNumber = Number(process.env.JOB_COMPLETION_INDEX); // JOB_COMPLETION_INDEX is 0 indexed
    const segmentTotal = Number(process.env.JOB_COMPLETIONS);
    chunkSize = parseInt(allSize / segmentTotal);
    chunkStartIndex = chunkSize * segmentNumber;
    chunkEndIndex = chunkSize * (segmentNumber + 1);
    if (chunkEndIndex > allSize) {
      chunkEndIndex = allSize;
    }
    const repositories = allRepositories.filter((_, i) => segmentNumber === i % segmentTotal);
    module.exports.repositories = repositories;
    module.exports.autodiscover = false;
    console.log(
      `/home/ubuntu/repos.json contains ${
        allRepositories.length
      } repositories. This is chunk number ${
        segmentNumber + 1
      } of ${segmentTotal} total chunks. Processing ${repositories.length} repositories.`,
    );
  } else {
    // Support for dedicated repository jobs that don't use JOB_COMPLETION_INDEX/JOB_COMPLETIONS
    // For these jobs, filtering is done by the run-renovate script
    // using SPECIFIC_REPOS or EXCLUDE_REPOS environment variables
    module.exports.repositories = allRepositories;
    module.exports.autodiscover = false;
    console.log(
      `/home/ubuntu/repos.json contains ${allRepositories.length} repositories. ` +
        `Running in dedicated repo mode. Processing all repos (filtering handled by run-renovate script).`,
    );
  }
} else {
  module.exports.autodiscover = true;
}

We also have a custom Docker entrypoint that handles the availability of this repo.json file, either signaling we need to write to S3 or to process and use jq to parse out the chunk of work to be run. 

#!/bin/bash
set -eo pipefail
DISCOVERED_REPOS_FILE=/home/ubuntu/repos.json
REPOS_DIR=/tmp/renovate/repos
if ! [ -f config.js ]; then
    echo "Error: config.js is missing from $PWD"
    exit 1
fi
if [[ -n "$GHE_INSTALLATION_ID" && -n "$GHE_ORG" ]]; then
    echo "GHE_INSTALLATION_ID: $GHE_INSTALLATION_ID ($GHE_ORG)"
else
    echo "Error: GHE_INSTALLATION_ID or GHE_ORG is not set"
    exit 1
fi
if [[ -n "$JOB_COMPLETION_INDEX" && -n "$JOB_COMPLETIONS" ]]; then
    echo "JOB_COMPLETION_INDEX: $JOB_COMPLETION_INDEX"
    echo "JOB_COMPLETIONS: $JOB_COMPLETIONS"
    if [ "$JOB_COMPLETION_INDEX" -gt "$JOB_COMPLETIONS" ]; then
        echo "Error: JOB_COMPLETION_INDEX is greater than or equal to JOB_COMPLETIONS"
        exit 1
    fi
fi
# Only download the repos file if:
# 1. We're not writing discovered repos (RENOVATE_WRITE_DISCOVERED_REPOS is not set)
# 2. We're not specifying specific repos (SPECIFIC_REPOS is not set)
# This way, we avoid downloading a potentially large file if we're just going to override it
if [ -z "$RENOVATE_WRITE_DISCOVERED_REPOS" ] && [ -z "$SPECIFIC_REPOS" ]; then
    aws s3 cp s3://${AWS_S3_BUCKET}/repo.${GHE_INSTALLATION_ID}.json $DISCOVERED_REPOS_FILE
    echo Processing "$(jq '. | length' $DISCOVERED_REPOS_FILE)" repos...
fi
# Filter specific repositories if SPECIFIC_REPOS is set
if [[ -n "$SPECIFIC_REPOS" ]]; then
    echo "Creating new repos.json with only these repositories: $SPECIFIC_REPOS"
    # Convert comma-separated repos to JSON array with org prefix
    JSON_ARRAY="["
    IFS=',' read -ra REPOS <<< "$SPECIFIC_REPOS"
    for i in "${!REPOS[@]}"; do
        # Add quotes and org prefix to each repo
        JSON_ARRAY+="\"${GHE_ORG}/${REPOS[$i]}\""
        # Add comma if not the last element
        if [ $i -lt $((${#REPOS[@]} - 1)) ]; then
            JSON_ARRAY+=","
        fi
    done
    JSON_ARRAY+="]"
    # Write the JSON array directly to the repos file
    echo "$JSON_ARRAY" > $DISCOVERED_REPOS_FILE
    echo "Created file with $(jq '. | length' $DISCOVERED_REPOS_FILE) repos"
fi
# Exclude specific repositories if EXCLUDE_REPOS is set
if [[ -n "$EXCLUDE_REPOS" && -f "$DISCOVERED_REPOS_FILE" ]]; then
    echo "Excluding specific repositories: $EXCLUDE_REPOS"
    # Convert comma-separated exclude repos to an array
    IFS=',' read -ra EXCLUDE_REPOS_ARRAY <<< "$EXCLUDE_REPOS"
    # Create a simple jq filter that filters out the excluded repos
    JQ_FILTER="[.[] | select("
    for i in "${!EXCLUDE_REPOS_ARRAY[@]}"; do
        if [ $i -gt 0 ]; then
            JQ_FILTER+=" and "
        fi
        JQ_FILTER+="(. | endswith(\"/${EXCLUDE_REPOS_ARRAY[$i]}\") | not)"
    done
    JQ_FILTER+=")]"
    # Apply the filter
    TEMP_FILE=$(mktemp)
    jq "$JQ_FILTER" $DISCOVERED_REPOS_FILE > $TEMP_FILE
    mv $TEMP_FILE $DISCOVERED_REPOS_FILE
    echo "After exclusion, processing $(jq '. | length' $DISCOVERED_REPOS_FILE) repos"
fi
# add loglines so we know when the renovate binary exited cleanly
echo "Starting renovate script processing..."
renovate
if [ -n "$RENOVATE_WRITE_DISCOVERED_REPOS" ]; then
    echo Discovered "$(jq '. | length ' $DISCOVERED_REPOS_FILE)" repos, writing to S3...
    aws s3 cp $DISCOVERED_REPOS_FILE s3://"${AWS_S3_BUCKET}"/repo."${GHE_INSTALLATION_ID}".json
    curl --data-binary @- "${RENOVATE_METRICS_PUSH_GATEWAY}/metrics/job/renovate_repos/instance/${GHE_ORG}" <<EOF
# HELP renovate_enabled_repos Number of repos enabled for Renovate
# TYPE renovate_enabled_repos gauge
renovate_enabled_repos{org="$GHE_ORG"} $(jq '. | length' $DISCOVERED_REPOS_FILE)
EOF

Webhook from Github Interactions

In addition to cronjob runs, we have a webhook listener based on https://github.com/mend/renovate-ce-ee that listens for Github events, largely around humans interacting with Renovate Issue or PR body to cause Renovate to take action. We take the upstream Docker image from ghcr.io/mend/renovate-ce and add customization on top of it for our own purposes:

• Add tooling requirements like AWS CLI, helm and helm-s3 plugin (where our internal helm charts are published to), and gRPC compiler dependencies for our internal use cases
• Explicitly pin the renovate CLI version (so we can be ahead of the webhook image's version and keep in sync with the cronjob worker version for consistent behavior)
• Add additional scripts that are used in post processing so we can allowlist those scripts via explicit bash entrypoint (like removing golang's toolchain, or running our CI tooling, etc) which improves security and eliminates holes in the regex allowlist that might be introduced

A lot of the magic here happens via these Dockerfile steps:

...
# renovate: datasource=github-releases depName=renovatebot/renovate
ENV RENOVATEBOT_VERSION=43.83.0
# onprem docker image doesn't contain the latest renovatebot version, so let's force it
WORKDIR /usr/src/mend
RUN sed -i -e "s|\"renovate\": \"*.*.*\",|\"renovate\": \"$RENOVATEBOT_VERSION\",|" package.json && \
    npm install --production --ignore-engine && \
    npm rebuild
WORKDIR /usr/src/app
# overwrite the broken renovate binary in path to the one we just installed
RUN ln -sf /usr/src/mend/node_modules/renovate/dist/renovate.js /home/ubuntu/.local/bin/renovate
...
ENTRYPOINT [ "docker-entrypoint.sh" ]
CMD ["node", "/usr/src/mend/src/community.js"]
EXPOSE 8080

Today, we utilize a local SQLite file for the job queueing which has its limitations that we'll discuss more below. But it works "fine" and our 4 hour cronjob is a decent enough fallback that our developers haven't complained about it. 

Metrics and Observability

As part of our maturation story around how we use Renovate, we joined forces between Security and Developer Experience, and they brought a desire for metrics and observability that tied together Github and Drone CI metrics so we could tell an integrated story about Renovate. 

We have data around CI job execution and build statuses in BigQuery that can be targeted by the Github user `renovate[bot]` that can be queried for a variety of interesting metrics:

• Percentage of PRs and CI jobs that pass/fail the build
• Duration of build jobs caused by Renovate PRs
• Burden of PRs by repo and human

We also experimented with Prometheus metrics following https://github.com/raffis/renovate-metrics which required a bit of adjustment due to the cardinality of our usage and some duplicate metrics from back in the day. This provided some interesting high level metrics, but nothing that really drove us to change our behavior. Most interesting, as a security person, was renovate_dependency_update{vulnerabilityFix="true"} metric and label that would let us understand if there were outstanding security related deps to go solve for. 

In addition, we’ve integrated Renovate checks and health into our internal tool called Reticle (our take on Chime’s Monocle) to provide developers a check on what they should be doing (we want teams to use Renovate and to ship “security” related PRs that come up). Below is an example of two Renovate checks we have for developers to ensure they’re doing the right thing. We should write a blog post about that at some point…

Scaling Challenges

Over the past year we've had some scaling challenges, which we've addressed (or ignored) in various ways:

Filesystem / Caching / GHE load

Since we run our own GHE server, we have to be kind to it in terms of API usage and abuse. When we run our periodic health checks for GHE, our Renovate user is at the top of the resource consumption (understandably so). So we attempt to cache these calls wherever possible, which means utilizing Renovate's caching functionality by specifying a filesystem to store a Github cache. Since this is running in Kubernetes, we need a mechanism to expose this to multiple pods. We currently utilize EBS for our default StorageClass which doesn't allow cross node attachment, so we pivoted to using AWS EFS as a shared storage mechanism, with bounded throughput. That bit was important as we found we were burning money using burst throughput. Today we have a provisioned throughput of 800 Mi/s which serves us well on cost vs. access.

As we found in our recent quarter's worth of optimization, EFS and Renovate's caching using cacert doesn't play well together. Renovate has default file system caching behavior defined here. This is used for all filesystem caching (repo, registry, etc.) by default with no external knobs to change behavior. This wouldn't be a problem except for lots of small files, the network roundtrip incurred by EFS really adds up. In troubleshooting where the bottleneck was in the Renovate processing, we saw we were spending ~90 minutes in cache cleanup attempting to process through ~27k files on EFS. Each cacache.get() call during cleanup requires multiple NFS round-trips (stat + open + read + close), each costing 1-10ms. On top of our parallelization, we easily started to hit our IOPS max throughput, all to evict a handful of files for the entire session. 

We're going to be re-attempting to use clustered Redis next for this optimization, which honestly we tried in the past but didn’t get around to troubleshooting the NOAUTH Authentication required. or Connection timeout errors we were getting, and it was an additional component in the infrastructure that we didn't prioritize at the beginning. So those 4 hour cronjobs will drastically shorten (they often don’t “finish” currently) and our resource utilization will look much better.

Self-Inflicted Failure from Goproxy

We also had a challenge we encountered with our go module proxy. Before increasing the proxy’s k8s resources, the proxy would get overloaded by the Renovate volume that resulted in an HTTP error to Renovate, and Renovate would think that there was no longer an update available, causing it to close the PR. It would then cycle/churn re-creating/closing PRs. We ended up setting hostRules.abortOnError to cleanly handle this and prevent the PR flapping.

Webhook Worker SQLite Database Contention

The other problem we had was with the webhook workload. Realistically, you'd scale off the number of waiting jobs you have. We actually have a KEDA ScaledObject with a trigger on the reported queue size: 

  triggers:
    - metadata:
        ignoreNullValues: "false"
        query: ceil(avg_over_time(mend_renovate_queue_size[5m]))
        serverAddress: http://thanos-query.monitoring.svc.cluster.local:10902
        threshold: "10"
      metricType: AverageValue
      name: mend_renovate_queue_size
      type: prometheus

But this doesn't mean much when you're still using a SQLite database across multiple pods, because you run into DB lock contention issues past 3 or 4 pods. So we're going to be migrating this over to a proper PostgreSQL database in the near future, but we managed to run without it for quite some time. 

Troubleshooting with $AI_AGENT

Not to be an LLM fanboy, but did want to shout out at how this is a great place for an AI CLI agent to shine. Parsing the Renovate logs at this volume and sprawl is rather difficult for a human, but unleashing our $AI_AGENT_CLI of choice on our Kubernetes logs combined with our Grafana MCP got us detailed information around resource utilization optimization, timings for various parts of the Renovate execution process, and made detailing the location and types of bottlenecks a lot easier than doing it by hand and fit in between an afternoon of incidents and architecture reviews. As usual, we validate the recommendations coming back, but I love not having to parse JSON logs with my human eyeballs.

What's Next On This Journey

So what started as an experiment is now an understood part of our dev environment, and we're trying to optimize this for how Reddit operates. We want to do a better job between how we handle internal libraries vs services as the cadence of releases, threat models, and behaviors are different between the two. The sensitivity to third party dependencies vs internal dependencies are also wildly different. 

LLM Enhanced Merge Confidence

Today, we have Renovate's PR that packages things up nicely in terms of documentation: release notes (when available), what's changing, CI checks. All of that is an improvement over the yesteryear of yeeting version bumps. But we can do better - that context in the PR is ripe for an AI assistant to take a look and provide even MORE value on "how dangerous is this version bump"? I don't want to review all the release notes and run it against a mental model I have for how I should think about that version change, I want an LLM to aggregate all of that and give me the distilled down version for me to make a judgment about. This is taking the existing Merge Confidence capability and enhancing it. 

Evolving past this would also be analyzing what actually did change. You see this already with reachability tooling (to various degrees), but LLMs are now REAL good at analyzing what's changed and tracing code paths. There's a world, in the next few months, where we have a coding LLM evaluate the differences between the two versions, figure out if the change actually impacts any calls we're using. The holy grail of reachability determination, at the expense of some tokens. 

LLM Enhancements to CI Passing

The other interesting thing would be to have an LLM loop through fixing Renovate PRs where there are CI issues or "harder" problems than the deterministic Renovate postUpgrade tasks can account for. We see this today with some of our more aggressive grouping of dependencies together into a single PR. This also has the potential money incinerator where Renovate tries to update the repo before the PR has been approved/merged and the AI bot "fixes" the PR and when it resets Renovate (as Renovate gives up if the PR has been modified unless you rebase it). 

Another interesting use case is where a Renovate PR has become stale, which may be addressed by the above improvement. If a developer adds a commit to a Renovate PR, then Renovate will stop processing that PR until someone decides to signal to Renovate to rebase and overwrite the previous commits. This results in the Renovate PR potentially drifting and then getting lost under future PRs. Improving the likelihood of merges when the Renovate PR is opened will eliminate this failure mode we currently have.

Automerge

Then the next logical step would be improving our automerge. Renovate has some automerge limitations that are well-documented. In addition, we’re currently utilizing Github's CODEOWNERS to power our approval flows, but as we add more bots then the simplistic CODEOWNERS type flow doesn't work well. We'll likely end up having to deploy a policy enforcing Github app, and then have an LLM handle the more complicated workflows of CI passing, rules for what should be automerged (ex. patches/minor semver only), safety of changes from merge confidence signal, and possibly other business logic.

And finally, a painful interaction point is when multiple dependency changes end up in merge conflicts that require constant rebasing. Renovate can handle this, but a human has to poke this to quickly address this which (depending on the volume of update PRs) can be high toil. An LLM skill or automation that loops to take care of these is a great automation that reduces the pain that these PRs can cause based on how lockfile conflicts are resolved. Coupled with automerge, and it becomes a seamless process.

In Conclusion

From the initial state of "Dependency Hell," our journey with Renovate has transformed dependency management at Reddit from a source of high toil into a core, scalable part of our developer environment. By prioritizing a self-hosted, highly customized solution over off-the-shelf tools, we have successfully managed over 2,700 repositories with decentralized configuration, robust cronjob parallelism in Kubernetes, and bespoke integration with our internal tooling like 'infrared'. While we continue to optimize our infrastructure—addressing caching bottlenecks with Redis and database contention with PostgresQL—our future is focused on leveraging AI. We are now positioning LLMs to enhance merge confidence, automatically fix CI issues, and enable sophisticated automerge policies, completing the journey to where we can finally stop worrying and truly love version bumps, moving closer to the 'holy grail' of dependency management where every version bump is safe, automated, and provides immediate value to our developers.

Author: René Treffer

At Reddit we care a lot about your cat memes (see e.g. SLOs @ Reddit).
In mid 2025 we started to see 1-2 incidents a week where tail latencies and errors would sharply rise, breaching our SLOs for a fraction of Reddit's traffic and functionality. Each incident was narrowed down to multiple services on a single Kubernetes node having issues. The nodes were quickly removed from the cluster and returned to our cloud provider to mitigate the issue.

After grouping the incidents and looking at our telemetry a pattern emerged

• Each incident was caused by a single Kubernetes node
• the machine would use excessive CPU compared to other machines
• workloads would be slow while overdrawing their cpu requests
• network packet processing would take excessive amounts of CPU time

Most of the incidents happened on newly provisioned machines, but around 5%-10% happened after machines were running for hours or days, excluding provisioning related issues.

It looked like machines collapsing under load, except for the network processing part.
There was no increase in network packets or connection tracking work.

Something in kernel space or in the hardware was breaking the throughput of the machine. We knew that it might take a while to find the root cause. Restoring consistent performance at the service level was top of mind.

Our priorities were

1. Restore consistent performance by mitigate the issue systematically and automatically
2. Escalate the cases to our cloud provider to attempt to find the root cause

Outlier detection to the rescue! (mid 2025)

In our quest to minimize production impact, we needed to identify and quarantine these machines from production.  We had the idea to track outliers – and create automation to remove them from the serving fleet.

Standard scores everywhere

The observed usage pattern was consistent for all workloads on a degraded machine but unique within each workload.

With this observation we build a standard score (z-score) based by the book outlier detection

• Group pods into workloads (through owner refs)
• Compute per workload average usage and standard deviation of the usage
• Compute per pod a standard score  z-score := (pod usage - average usage) / standard deviation
• Use Stouffer's Z-score method to compute a weighted per-node z-score

SA small service called k8s-zscore is responsible for querying our in-cluster thanos setup to produce the required metrics.

Kubernetes makes it easy to remove machines, meaning the cost and impact of a false positive is low. But a false negative (degraded, not detected) can be an incident. The low cost of false positives makes an outlier detection approach acceptable.

Based on our data from the next incidents we established a threshold of 7 for 10 minutes as the signal for a degraded machine.  Ten minutes seemed a reasonable compromise by being faster than a human could debug the issue while being long enough to eliminate most transient false positives.

We use the node conditions field in Kubernetes to communicate any node degradations. In this case we set a NodeZScoreUnhealthy condition via k8s-zscore.

Automatic mitigations

We use a system called node-health-manager to automatically mitigate machines based on node conditions.

Our current action plan for NodeZScoreUnhealthy is:

• After 1 minute taint the node (no new workloads)
• After 10 minutes drain the node and mark it for rotation (return to the cloud provider)
• If the node recovers then untaint after 5 minutes

This mitigation is still active today and we are regularly mitigating machines. Our goal was to clean up the fleet by removing problematic machines.

Not a full solution

The outlier detection caught and mitigated some cases. This was a big step forward but it wasn’t sufficient to solve the issue. The list of problems:

1. Detection was too slow - 10 minutes for the outlier detection signal alone
2. Not all slow machines triggered the outlier detection as it depends heavily on workload characteristics, e.g.
   • cpu limits cap the signal
   • workloads that do not reach the ready state or flap readiness skew the signal
   • some workloads are noisy in nature (e.g. workers getting variable size tasks from kafka)
3. Services got better at shifting traffic away

Point (3) was interesting to observe: as our services got better at routing around single slow nodes we would lose the cpu overuse signal.

Incident, no incident, incident, no incident, …

While incident severity and duration dropped, frequency remained constant. Late August was another low point: any decommissioned machines would result in a new incident with the exact same symptoms in less than 24 hours! We were in for a weekend long game of whack-a-mole! 
We were seeing unique kernel messages that we had never seen before. We suspected that we were getting the same machine over and over again as the kernel behavior was unique throughout the fleet, yet consistent between and incidents. And the incidents weren’t overlapping in time.  With no ability to uniquely identify hardware instances to avoid them if they return to our fleet as a new instance on restart, we needed to sideline these bad machines so they would not come back.  

We invented a new process on the spot: Instead of returning the machine to our cloud provider, we isolated it by marking it as unschedulable as we had no other way to block machines from reappearing in load bearing clusters. Unfortunately, this meant that we were effectively paying for a machine we couldn’t use and didn’t want.  We continued to raise the situation to our cloud provider.

The new runbook was

• Isolate the machine, eating the cost
• Open a support ticket and escalate the situation to the cloud provider
• Wait for action on the ticket before returning the machine (Eventually, once validated they would remove the machine from service).

This was very toilsome but helped to resolve the incidents in a more lasting way. We were also able to work more closely with our cloud provider to track down the issue as we accumulated more data points for them to debug.

The isolation also provided valuable time to investigate the underlying machine. Our benchmark efforts quickly yielded a root cause: the mbw memory bandwidth test consistently reported throughput numbers below 100MB/s. CPU heavy benchmarks (like openssl speed tests) would be close to normal. Healthy machines rarely dropped to 2GB/s per core and never below 1GB/s. An order of magnitude in degradation.

We had found the root cause: memory bandwidth was collapsing!

How about a direct measurement?

We initially wanted to get a passive reading of the issue. The recommended metric to detect memory controller congestion is instructions per cycle (see e.g. CPU Utilization is wrong): “how many cpu instructions get executed per cpu per cpu cycle?”. This number should drop way below 1 if we are waiting for memory all the time and it should be around or above 1 for any normal operation.

This approach would be free of any cost as we are running node_exporter already.
However it was not feasible:

1. We would not be able to get the metric from all instances we operate
2. and we hit a kernel bug for the required permissions (fixed upstream)

How about benchmarking, everything, all the time?

We could not get a passive reading, but we were able to find the issue with benchmarking. What would it take to benchmark the fleet all the time?

There are a few interesting constraints when benchmarking a fleet:

1. Benchmarks must not interfere with other workloads
2. We need high resolution to mitigate issues quickly
3. We should use as little resources as possible as we will run everywhere
4. We expect the benchmark to run 10x slower when the machine is broken

How small can we get?

We want to hit the memory controller, not any CPU caches.

We use 2 buffers, filled with random data initially. We then read/write data:

1. Read Buffer 1 for cache busting
2. Copy Buffer 2 to Buffer 1

We settled on 2x 256MB buffers. We read 512MB per run and write 256MB. This is larger than the largest L3 caches giving us a guarantee that we will read main memory.

At 100MB/s we expected the benchmark to approach ~5s for the copy and another up to ~2.5s for the initial cache busting. Healthy machines should see less than 250ms of benchmarking every 15s or ~2% of the time. This is still acceptable as the memory controller is a shared resource that we can’t saturate.

Our impact on other workloads is roughly 1/100th of the controller and cpu ~2% of the time. The memory resource usage is significant but less of a concern as our fleet is usually cpu constrained.

Does it work?

We tied this detection into our custom load shedding daemon, halon. It will set a DegradedMemoryPerformance node condition and depending on the node group start a slow drain of the machine.

Our node-health-manager will take the same steps we did manually:

• Cordon & drain the node (isolate)
• Freeze it for 24h (keep it)
• Force rotate after 24h (return it after escalation)

This has been running since December 2025 and it worked nicely. Detection and mitigation takes roughly 10 minutes. There was only one issue: sometimes the problem would go away as the system drained, leading to an oscillation between healthy and degraded. This is solvable with a flip/flop detection, any node that repeatedly joins the mitigation will get deprovisioned.

We still needed to report each case through support tickets as we needed the machines fully removed.

The last mile

Managing support  tickets for every single machine became a major pain point. We set out in 2026 to automate this part of the process with an achilles based support-case-controller.

If a node shows degraded performance for 1h then we will go ahead and create an external support case with the metadata of the machine. This filters out any potential false positives and cases where the machine completely failed within 1h.

The state of each ticket is reflected in Kubernetes. We export the status as prometheus metrics so that we can visualize the state in Grafana.

Fin

At scale, we increasingly need work-arounds that can be implemented faster than the overall support case speed. In large cloud environments the “Birthday Problem” means that while something seems relatively rare, for sufficiently large populations of machines many workloads experience these problems daily, or more.  In this case, triangulating the problem often takes many data points, and close partnership with our cloud provider. In this case, our early hunch was bad hardware – our support cases proved invaluable here to aid in collaborative debugging, but it took months. Finally, more than six months later, enough data was gathered to root cause the problem and discover their detections were insufficient. After adding their detections, we saw a marked reduction in the performance cases we had to triage with our own automation. 

Today, this incident is resolved (we’re back to expected baseline failure rates). Given the law of large numbers and a complex heterogeneous serving fleet, we still see “anomalies” with performance of the long tail of our cloud operators machine fleet.  We continue to work with our cloud partners to find a generalized formula for how to address and debug these machines in a timely manner. Fortunately, we now have our own automated detection, quarantine, and ticket escalation workflow that should make this faster for us to return the platform to healthy serving quickly.

Written by Neven Miculinic

TL;DR

At Reddit, we send millions of log events per second and compress terabytes of data each day, keeping fourteen days of retention. That’s a lot! Our third-party logging SaaS provider was no longer able to meet our needs. We were facing operational and reliability concerns, scaling demands, and we lacked an integration with Grafana, our central observability hub. 

To meet those demands we developed Snoolog, our in-house, self-hosted logging platform. It gives us complete control over our logging infrastructure, eliminates vendor lock-in, and better integrates with our other internal tools. 

To minimise operational overhead, we built it on top of Clickhouse, a generic OLAP system that’s already used across other Observability Team products (including tracing and error tracking). To continue using Grafana as our central observability hub, we built a custom datasource and exposed a Lucene-like query language to end-users. This let us reuse our existing OLAP expertise while keeping a familiar, search‑style interface for querying logs.

Problem

Unstructured Logging is the core component of observability. Customer processes can write arbitrary information, and developers can later inspect it to understand what’s happening with their services and make educated decisions on how to respond. Unstructured Logging differs from Structured Logging (e.g. security audit logging) in that the log lines are arbitrary text, albeit commonly structured in key-value pairs. Unstructured Logging also has fewer guarantees on completeness, a shorter retention window, and offers lower comparability over time. Logs are fundamental observability tooling, and we need reliable and performant support for them.

Our previous solution didn’t scale with Reddit’s logging volume, leading to frequent outages, and ingestion delays. Further, it lacked integration with Grafana, Okta, and other internal tools. 

We needed a logging system that prioritized reliability, guaranteeing continuity of service and stability even when noisy services spiked traffic. It had to support efficient structured and full-text search, integrate seamlessly into Grafana alongside our metrics and traces, and cover security essentials like PII scrubbing and proper identity management. Crucially, it needed to scale with Reddit's growth without costs scaling linearly alongside it.

Why OLAP for Logs?

If you squint at the workload characteristics of observability data (logs, traces, metrics…), they all look remarkably similar: write-heavy, read-recent, with queries that filter and aggregate large volumes of semi-structured data.

For years, the industry relied on search-engine-derived technology (Elasticsearch, Solr) built for full-text search with heavy indexing. The industry is shifting toward OLAP databases like ClickHouse for observability workloads which have been used successfully for petabyte-scale logging

The appeal for us was concrete. We already ran ClickHouse for tracing and error tracking, and moving logs to ClickHouse meant we could further consolidate our storage layer. We’d already solved for tiered storage, query federation, disaster recovery, and access control, and the efficiencies allowed us to deepen our operational expertise on a single system. Additionally, since observability and monitoring is a critical function requiring redundancy, we run separate ClickHouse clusters per product.

Architecture

The pipeline is straightforward, and deliberately so:

Log events flow from application containers through vector.dev agents deployed per cluster, which read the logs and apply client-side rate limits to protect the system. These agents ship logs to a central ingestion layer that handles metadata enrichment before storing the payloads into Kafka. From Kafka, a dedicated ClickHouse loader process consumes the events and writes them into ClickHouse for long-term storage. Finally, Grafana serves as the query frontend through our custom datasource plugin.

At ingestion time, we parse JSON log lines and separate system attributes (namespace, pod, cluster, log level) from service-specific attributes (anything in the application logs). This separation lets us optimize primary keys and skip indices for the most common query patterns. Storage is tiered: recent "hot" data lives on EBS SSDs for fast queries, while older "cold" data moves to S3.
 

Building the UX Layer

Making logging available in the same place meant engineers didn't context-switch between tools during an incident, because service dashboards could display all observability signals together. We leveraged existing Grafana log panels, and only built a datasource adapter for the new system.

OLAP alone doesn't make a user-friendly interface. SQL is powerful, but it assumes you know table schemas, column names, function names, and how to express time ranges, filters, and text search correctly. While that’s fine for analysts during office hours, it’s a terrible fit for an engineer at 3 am responding to an incident. This is why we built a Lucene-like query language UX with Grafana datasource, translating the key:value AND "error" syntax into optimized ClickHouse SQL under the hood. Because we fully own the UI, any potential migration from ClickHouse to a different OLAP won’t involve any client-facing migration needs. 

The query editor also includes autocomplete for attribute keys and values, visual attribute filtering, URL sharing for specific log views, and Grafana variable substitution for reusable dashboards. 

Challenges and Lessons learned

Technical Realities of OSS ClickHouse

ClickHouse has an amazing query engine. However, compute-storage separation (SharedMergeTree) is kept proprietary, making OSS (auto)scaling operationally hard. 

ClickHouse OSS offering has a shared-nothing architecture: every node handles ingestion, background merges, and queries. While great for simplicity, it creates operational realities we had to accept: there is no automatic scaling, no read/write separation, and each replica maintains its own redundant copy of data on cold S3 storage. Adding a replica is an expensive operation. So, we need to carefully plan our capacity and manual sharing in advance of.

We also learned a hard lesson about potential over-engineering. ClickHouse isn't (at the time) a search engine, but to support arbitrary substring search across log messages, we used ngram bloom filter indices. The problem: these filters have a significant false-positive rate, making broad text searches unexpectedly slow as the engine scanned too many granules (which we later tuned). In hindsight, we should have asked if engineers truly needed full substring search, or if token-based search (matching whole words) was sufficient. Sometimes the simpler approach is the right one. Clickhouse’s capabilities improved over time. With lazy materialization, streaming skip-indexes, and full-text inverted index ClickHouse has all primitives to build & tune your own search engine for observability use cases.  

UX Pain Points

While using upstream Grafana log panels sped up development, we are beholden to its quirks and limitations:

• JSON Noise: We parse and flatten arbitrary JSON log attributes into key-value pairs. For deeply nested JSON, the resulting attribute view in Grafana feels noisy and overwhelming. Users cannot collapse attribute subtrees.
• Scroll & Order Confusion: the default scroll and order functionality is cumbersome to change because of code design choices, and breaks the flow of investigations

Other UX pains points are self imposed:

• The "Live Tail" Gap: Some engineers miss live log streaming. They relied on it to deploy monitoring and incident triage. We offer and encourage real-time metrics use, 30s log view auto-refresh, or kubectl log to live-tail specific pod. 
• All-Field Search: For performance and cost reasons, searching across all log attributes is not supported. Users must explicitly specify the attribute to search, or the system will default the search to the message field.

Due to aforementioned quirks, logging UI prototypes are still occasionally tinkered with during company hackathons. It’s valuable for us to learn from our most engaged users and we look forward to incorporating their ideas. 

Conclusion

Looking back on the migration, building a bespoke logging solution in-house is undeniably hard. However, it solved our core problem: Snoolog handles our growing scale reliably, and by reusing ClickHouse, we achieved this highly cost-effectively compared to SaaS alternatives.

Is it a perfect system? No. We have to be honest that our custom UI isn't as polished as dedicated vendor offerings. Users frequently ask for UX improvements, and one of our biggest ongoing feature requests is the ability to easily perform full-text search across all JSON fields rather than specifying individual attributes. We're still iterating to close those gaps.

But we developed Snoolog in the open. We ran company-wide bake-offs and published all raw feedback - even the critical stuff. This radical transparency earned the organization's trust. Ultimately, by controlling our own data layer and UX, we control our own destiny, with a platform that can scale alongside Reddit for years to come.

Written by Austin Jackson.

TL;DR: In our previous blog post, we covered how Reddit built its Observability (O11y) data pipeline – the system that gets security logs from 50+ sources into Google BigQuery. This post picks up where that one left off: now that the data is flowing, how do we detect threats? We’ll walk through our detection-as-code framework, automated alert orchestration, AI-powered triage, MITRE ATT&CK coverage mapping, threat emulation, and the full detection engineering lifecycle.

The Big Picture

A quick refresher: Reddit’s security Observability platform (O11y) ingests logs from dozens of sources, including: identity providers, endpoint agents, cloud platforms, internal services, and more – processes them through Cribl and Apache Kafka, and lands everything in Google BigQuery.

The data pipeline is the foundation, but the value comes from what we build on top of it. Every detection at Reddit is a YAML file committed to a Git repository. That file defines what data to query, how often to query it, and what to do when something suspicious turns up. Those YAML files get translated into scheduled jobs that query BigQuery and, when results are found, kick off automated actions: Slack alerts, PagerDuty pages, Jira tickets, AI-powered analysis, and more.

Detections as Code

Every detection lives as a YAML file in a Git repository, goes through code review via pull requests, and is version-controlled. This gives us peer review, change history, rollback, and CI/CD (Continuous Integration / Continuous Deployment) applied to our security detections.

The Detection YAML Spec

Here’s a real example, a detection that alerts when a new IAM user is created in AWS:

name: AWS IAM CreateUser
enabled: true
environment: prod
team_ownership: infrastructure-security

action:
  pagerduty:
    service_id: "<pagerduty_service_here>"
    severity: "critical"
  slack: ["<slack_channel_here>"]
  jira:
    project: "<jira_board_here>"
    assign_to:"[email protected]"
  email: ["[email protected]"]
  ai_agent: "<ai_agent_here>"
  distributed: false

detection:
  engine: airflow
  datasource: aws
  severity: 1
  detection_confidence: high
  detection_impact: high
  cron: "*/5 * * * *" # Run every 5 minutes
  runbook: "<runbook_link_here>"
  tags:
    - "attack_persistence_T1136.003"
  query: >-
    SELECT
      insert_time,
      event_time,
      event_name,
      event_source,
      error_code,
      ... (many more fields here)
    FROM
      `reddit-o11y.siem.aws`
    WHERE
      event_name = 'CreateUser'
      AND event_source = 'iam.amazonaws.com'
      AND error_code is NULL
      AND JOBS_TABLE_FILTER

The YAML file has three main sections:

Top-level metadata – the detection name, whether it’s enabled, the environment (prod vs. nonprod), and the owning team.

The action block – what should happen when the detection fires. Detection authors have full control over alert routing: PagerDuty for paging on-call analysts, Slack channels for collaborative triage, Jira for ticket tracking, email for notifications, and an ai field that routes alerts to an AI agent for automated triage (more on that later). There’s also a distributed feature that can DM the involved user directly in Slack to ask “Did you actually do this?” – useful for user-verification scenarios.

The detection block – the core logic. This includes the execution engine, data source, a severity score (0 = critical through 4 = informational), confidence and impact ratings, a cron schedule, a runbook link, MITRE ATT&CK tags, and the BigQuery SQL query itself. Severity, confidence, and impact work together to control alerting behavior; only detections with severity 0-1 and will trigger PagerDuty pages.

The Detection Pipeline: From YAML to Alert

How do YAML files in Git become running queries that catch threats?

1. Git to Airflow: Detection YAMLs are pulled into Apache Airflow and each one is automatically translated into a DAG (Directed Acyclic Graph) – Airflow’s unit of work. The DAG inherits its cron schedule from the YAML spec.
2. Airflow queries BigQuery: When a DAG runs, it executes the detection’s SQL query against Google BigQuery. We have detections running on schedules from every minute to once a week.
3. Results trigger actions: If the query returns results, Airflow sends an HTTP POST to Tines, a security automation platform, with the results and the full detection YAML spec. If no results, nothing happens.

The Sliding Window: Handling Overlaps

There’s a critical subtlety with scheduled queries: cron is approximate, not exact. A detection set to run every 30 minutes will run roughly every 30 minutes, but jitter, delays, or catch-up runs after an outage could mean missed or double-scanned events.

Our solution is the JOBS_TABLE_FILTER placeholder. Detection authors place it in the WHERE clause of their SQL, and at runtime the pipeline automatically replaces it with a precise time-bounded filter:

WHERE
  event_name = 'CreateUser'
  AND error_code IS NULL
  AND insert_time BETWEEN '2026-01-15T10:00:000Z' AND '2026-01-15T10:05:000Z'

The pipeline tracks the exact timestamp where the previous run left off and uses the current time as the end boundary. This creates a true sliding window – no gaps, no overlaps. Every event is scanned exactly once, regardless of scheduling variance. If Airflow goes down for an hour and recovers, the next run picks up right where the last successful run left off.

The O11y Action System: Automated Alert Orchestration

When a detection fires, the alert enters our O11y Action System – a Tines automation workflow that orchestrates the full response based on the detection’s YAML spec. Here’s a high-level overview of how this system works:

Scoring: The engine evaluates severity, confidence, and impact to determine which actions fire.

Suppression: The system de-duplicates alerts, checking whether we’ve already seen a given detection + result combination within the past 8 hours. If so, the duplicate is dropped – nobody likes getting the same alert fifty times.

Alert Actions: Once an alert passes scoring and suppression, the system fans out:

• Slack is the primary workspace. The Reddit Security Bot posts a structured message with the alert name, a Jira ticket link, the detection runbook, a link to the detection YAML in GitHub, severity, team ownership, and an alert silence toggle. The alert results will also be placed into the Slack alert thread for responders to easily reference.

• PagerDuty triggers for the most critical alerts – the “drop what you’re doing” signal.
• Jira tickets are auto-created on our SOC (Security Operations Center) board for tracking and archival purposes.
  

Slack2Jira: Bridging the Gap

Analysts work in Slack – that’s where they first see alerts, discuss findings, share screenshots, and decide on next steps. But Jira is where we need information for tracking, reporting, and archival. Nobody wants to copy-paste Slack conversations into Jira manually.

Slack2Jira is a Tines automation that bridges the two:

• Every alert already has an auto-created Jira ticket (via the O11y Action System).
• When an analyst reacts with the 👀emoji, the Jira ticket moves to “In Progress.”
• Every message and file in the Slack alert thread is automatically copied to the Jira ticket as a comment – including images and attachments. Slack markdown is converted to Atlassian Document Format for clean rendering.
• When an analyst reacts with the ✅emoji, the ticket moves to “Done.”

The result: the Jira SOC board becomes a complete, searchable archive of every alert and its full investigation trail, without analysts leaving Slack.

AI-Powered Triage

Security teams face a universal challenge: more alerts than humans to investigate them. We built AI into the pipeline to give analysts a head start.

The ai field in the detection YAML routes alerts to an AI agent. When a detection fires, the agent analyzes the results and produces a structured response: alert summary, contextual analysis, risk scoring, and recommended next steps. This is posted directly into the Slack alert thread, so analysts get a detailed briefing before they even start investigating.

Our agents also have tool-use capabilities – they can resolve endpoint identities, look up user details across security platforms, and investigate authentication patterns. The extra_prompt field lets detection authors provide per-detection context to guide the AI toward more relevant analysis.

Importantly, AI doesn’t make decisions for us. It’s a first pass that surfaces context, an initial hypothesis, and recommended next steps. Human analysts always review, validate, and decide on the response for critical security alerts.

MITRE ATT&CK Mapping and Coverage Tracking

The MITRE ATT&CK Framework is a comprehensive knowledge base of adversary tactics, techniques, and procedures (TTPs). Every detection we write is tagged with the relevant techniques in the tags field.

tags:
  - "attack_initial-access_T1566.001"   # Phishing: Spearphishing Attachment
  - "attack_execution_T1059.004"        # Command Execution: Unix Shell
  - "attack_persistence_T1098.003"      # Account Manip: Additional Cloud Roles

Our detections repositories CI/CD parses these tags across all detections and auto-generates a MITRE ATT&CK Navigator layer – a visual heatmap of our detection coverage across tactics. Alongside the Navigator layer, the CI/CD tooling generates coverage metrics for automated reporting, giving us a clear view of where we have strong coverage, where we have gaps, and how our coverage is trending over time.

Threat Emulation: Trust, but Verify

Detections can drift over time: a vendor changes their log schema, a BigQuery view gets updated, a tuning rule becomes too aggressive, or an infrastructure change alters the data pipeline. If a detection silently stops working, you might not notice until the attack it was designed to catch actually occurs.

Our threat emulation system addresses this by injecting known true-positive log examples directly into the pipeline. These synthetic events should trigger specific detections, and if they don’t, we know something has drifted. Think of it as a heartbeat monitor for the detection system – continuous validation that our detections are responding to the threats they were built to catch.

This is especially valuable after tuning. When we add exclusion rules to reduce false positives, threat emulation ensures those rules haven’t accidentally suppressed the true positive cases we care about.

The Threat Detection Lifecycle

Threat detection is a continuous cycle, not a one-time effort.

1. Threat Intelligence: We consume threat intelligence from threat feeds, industry reports, vendor advisories, and our own investigations. We prioritize based on relevance to Reddit’s environment and actionability given our log sources.
2. Threat Hunting: Our security team proactively hunts for signs of compromise using BigQuery, looking for patterns that don’t currently warrant automated alerts: unusual activity, known adversary behaviors, and artifact chains suggesting multi-stage attacks. Successful hunts that indicate threat patterns will become new detections.
3. Detection Engineering: An engineer scaffolds a detection YAML, writes the SQL, tags it with MITRE ATT&CK techniques, and opens a PR for review.
4. Testing & Tuning: New detections route to dedicated test Slack channels. We observe alert volume and quality, add exclusion rules for benign activity, adjust thresholds, and refine logic to maximize signal-to-noise ratio. Once reliable and accurate, the detection graduates to production.
5. Operationalize: Tuned detections move to production Slack channels monitored by on-call analysts. Full alert routing activates: Slack notifications, auto-created Jira tickets, PagerDuty pages for critical detections, and AI triage analysis.
6. Respond: When detections fire, analysts triage using Slack threads, AI analysis, and runbooks. Routine findings are handled directly. Serious events engage our incident response processes. Findings feed back into the cycle to improve future detections.
   

Wrapping Up

Reddit’s threat detection system is built on the principle that security should be treated like software engineering. Detections are code – reviewed in PRs, tested in staging, deployed through CI/CD. Alert routing is declarative, defined alongside the detection logic. AI handles initial triage so humans can focus on judgment calls. And the system is continuously validated through threat emulation.

This is the detection layer built on top of the O11y data pipeline we described previously. Together, they form a code-driven security operations platform that scales with Reddit.

What’s next? We’re approaching building streaming detections on Kafka for near real-time detection, expanding our AI agents toward more autonomous investigation, and looking at contributing back to the open-source community.

More from the Reddit Security team coming soon. Stay tuned for posts on streaming detections, agentic AI in security operations, and the evolution of our data ingestion pipeline.

Written by Martin Preedy, with heartfelt thanks to Chan Park, Drew DiBiase, Jenna Wei, and Andrew Meyers

TL;DR

Our Internal Audit team automated SOX testing for 175 controls in 3 months, using advanced OCR + agentic AI, cutting testing time on average by 60% per control. Here’s how we did it, what we learned, and why we’re so excited about empowering the profession to reach new heights.

The Problem: SOX Testing Was Where Automation Went to Die

If you've ever worked in SOX testing, you know the drill. The work is critical, repetitive, and about as automatable as a philosophical debate.

Why? Evidence comes in every format imaginable: PDFs with tables that barely parse, Excel files with merged cells, system screenshots, scanned documents, and unstructured data with no consistent schema. Traditional RPA noped out. The technical debt of building for every edge case made automation economically ridiculous.

Add high complexity and rigorous PCAOB standards and documentation requirements, and we were still stuck with smart humans manually testing controls - which works but doesn't scale.

The Technical Solution

This wasn't a "throw documents at ChatGPT and hope for the best" situation, but modern AI is the core enabler due to its fundamental ability to cut through the chaos of unformatted SOX evidence. Large Language Models, trained on the entire internet's most unruly data (including Reddit), can actually handle the 'insanity' of real-world documentation that traditional automation attempts couldn't touch.

But reading messy documents is only half the battle. True automation at scale requires a governed system that captures deep, relevant context and mirrors the full auditor workflow: reading evidence, applying test criteria, performing procedures, reviewing the work, and producing proper documentation.

And that multi-step process demanded specialized, purpose-built agents:

• Evidence agents that extract and structure data from source documents
• Testing agents that evaluate evidence against test criteria
• Review agents that perform quality control and flag edge cases
• Documentation agents that generate work papers with full audit trails

This was the game changer

What We Did

First, we had to tackle the build vs buy conundrum and knew building was the fast road to fatigue—buying was the only way to tackle this complexity and succeed quickly. After rigorous head-to-head pilots evaluating several platforms, we selected Midship for its advanced technology, flexibility for customization, and the team’s willingness to iterate with us as a true product partner.

Then we really got to work:

• Automated 175 controls in 3 months, over 40% of our SOX scope
• Covered every control and test type - business process controls, IT general controls, interfaces, automated controls, Entity-Level Controls (ELCs), key reports, and SOC reports. Test of design and test of operating effectiveness. Multi-sample tests, multi-table tests…
• Used Midship to ingest evidence, run AI testing, and produce work papers formatted in our external auditor’s template
• Created clear explanations for every test result with tickmarks and annotations showing exactly what the AI evaluated and why, and where it got its info
• Retained a robust human-in-the-loop review process (because quality issues invalidate the entire AI use case)

What We Learned

Setup is 80% of the battle: Getting the configuration right up front is critical to test accuracy and minimizing manual override on the back-end. It can be tempting to shortcut this stage but it’s infrastructure - you build it once and reuse it forever.

Data quality still matters: Garbage in, garbage out applies to AI too. The better the existing documentation (control and test metadata, test attributes and existing work-papers etc.) and evidence quality, the more bang for your buck.

Intelligence and context is fuel: Using existing test attributes as generic prompts gave us good results. Adding extra context gave us great results. The team became really good prompt engineers and harnessing that intelligence is the fuel that makes repeatable agentic workflows scale. Deep, relevant context means accurate conclusions and proper documentation every test run.

Output quality is make-or-break: The AI can be 99% accurate, but if the output looks like AI slop, humans can’t validate it and external auditors won’t trust it. We invested heavily in output design – building custom templates to mirror what our external auditors were used to, visual tickmarking and annotations, and digestible audit trail documentation.

AI doesn’t make sense for every control… yet: Not all controls are created equal. In general, the longer it takes to perform a test manually, the better the ROI. Testing an automated control once a year? Not as much to gain, so we’ll do those later.

Why This Matters

This changes the game:

Quality: The combined “machine + human” approach raised the bar on quality. AI caught things humans missed, proving the results were better than before, not just faster. Important for external auditor buy-in.

Immediate results: Instant test results mean we get more time to remediate deficiencies and more flexibility scheduling testing and managing workloads. And external auditors get our work sooner for reliance purposes.

Efficiency: 60% reduction in testing time per control on average. That’s not shaving some time off - it fundamentally changes the economics of SOX testing.

Scalability: Now we have a governed, infrastructure engine for other recurring testing programs. Because we built for SOX - with the highest complexity and documentation standards - everything else is easier.

Higher value work: By automating high-volume mechanical stuff, we’re freeing up capacity for strategic work that matters more to the business.

Empowerment and a brighter future: No-one ever said, "When I grow up, I dream of making sure this data in this system matches that one." Instead of human OCR machines, we’re helping Internal Auditors become AI strategists and risk-based decision makers, and giving them development opportunities in new areas.

What’s Next

There’s so much opportunity ahead of us and we’re excited to see how far we can take this:

Max out SOX automation

We’re only 40% of the way there. We’re aiming for 90%+.

Automated Evidence Collection

We’re exploring automated evidence collection - grabbing populations, sampling, and pulling evidence without human intervention. That gives us zero-touch compliance - a big win for Engineering and other control owners - and opens up end-to-end automation and scheduled job testing.

Self-Service Testing

Empowering process owners to run their own pre-tests and grade their homework before independent testing. Applying a shift-left mentality to assurance.

Continuous Monitoring and Assurance

Moving from periodic testing to continuous monitoring.

Scale Everywhere

Taking this beyond SOX to every recurring testing program we run.

Keys to Success

Find meaning in your work and set a lofty, inspiring vision

This isn’t about cost-cutting or reducing headcount. It’s about fundamentally rethinking what’s possible and creating the AI testing infra that powers our function to do more. We didn’t want to just check the box on AI - we wanted to go after our biggest opportunity and be first. Not for bragging rights, but to prove it could be done, shape how it’s done, and share what we learned with others.

Innovation mindset

This wasn’t comfortable or easy. As a small team, we went outside our comfort zone and took on a beast of a side quest while working on first-year SOX compliance… which is kinda nuts! But fortune favors the bold (and slightly delusional).

Get your tech selection right

There's no way we could've moved this fast, this well, without an exceptional vendor partnership. There's a lot of noise in this space and we waded through some bold claims. We had to be super-diligent in evaluating vendors - maybe even auditor-level-skeptical.

Our selection criteria:

• Accuracy rate across different control types, evidence formats and variables (this varied wildly across vendors)
• Output quality and ability to generate audit-ready documentation directly in our external auditor's template format - no reformatting, no translation layer.
• Real software - not a black box. We needed a product our team could actually use, end-to-end. Too many vendors were skittish about us getting hands on keyboards.
• Functionality and features to handle the nuances of real world testing. Comprehensive test templates, multi-test tables, editable tickmarking and annotations, output template builder, and those other UI features that can handle edge cases and really improve quality of life.
• True partnership with a vendor willing to build with us, take our feedback seriously, and rapidly iterate on the product. This wasn't about finding finished software, it was about finding a partner who'd evolve with our needs.

We ran rigorous pilots with multiple vendors using a variety of 10-15 real controls. We tested the tooling ourselves and the differences were stark. Failing tech kills momentum like nothing else. This decision is make-or-break.

Final Thoughts

A year ago, automating SOX testing with AI sounded like science fiction. Today, it’s production code that processes real testing for real financial statements. We’ve accomplished more in the last few months than I’ve seen in 20 years of “SOX automation initiatives.” There’s challenges ahead, but the velocity is genuinely shocking and the possibilities are endless.

There are 4 wonderful people I can’t thank enough for all they’ve done - for being willing to fail, iterate, and try things that sounded crazy. Chan Park, Drew DiBiase, Jenna Wei, and Andrew Meyers work so hard and so smart, and they’re the best in the biz.

If you’re in audit, risk, compliance, or any adjacent field and you’re not experimenting with AI automation to solve your biggest problems, you’re leaving a massive opportunity on the table. The technology is ready. The question is whether your organization is ready to embrace it.

-----

*P.S. - To the inevitable question “but what about hallucinations?” Yes, we account for that. That’s what the review process, confidence scoring, and auditor-ready work papers are for. AI is a tool, not a replacement for professional judgment.

*P.P.S. - Yes, pretty much everyone was skeptical at first, including me. The antidote to fear was results.

Written by Michael Petro

BigQuery serves as the central compute engine of Reddit’s data platform. It powers ingestion, batch ETL, feature engineering, experimentation, analytics, and so much more. While BigQuery is performant and extremely scalable, these qualities make it easy to spend enormous amounts on compute without the right guardrails. In this blog post, we’ll walk through how we flattened Reddit’s BigQuery slot cost growth, and reduced our average slot hour cost by 21%.

Background

Cloud infrastructure billing models typically fall into one of the two pricing paradigms: consumption pricing or capacity pricing. In a consumption pricing model, you pay for resource consumption regardless of traffic shape; infrastructure scales to demand and you only pay for usage. In a capacity pricing model, you pay for capacity availability; you pay a premium for scalability and spiky consumption.

For most of BigQuery’s history, it has followed a consumption pricing model. The initial on-demand-only model billed by data scanned. Later, slots (BigQuery’s abstraction for a unit of compute), exposed capacity in the interface. Eventually, flex slots, which supported flexible capacity, allowed for large capacity bursts for short periods without committing to long-lived static capacity. 

In 2023, Google launched BigQuery Editions and re-centered around capacity pricing. Google deprecated Flex Slots, removing the ability to buy cheap short-term bursts of capacity without static capacity commitments. Additionally, they increased the price of on demand querying by 25%, pushing customers away from consumption billing. These changes made room for a new elastic capacity model.

In the editions model, a reservation is a pool of slots made up of baseline and autoscaling slots. Baseline slots are statically billed and allocated to a reservation, acting as a capacity floor. Autoscaling slots are additional slots that scale up and down (between baseline and the max reservation size) to meet variable demand, and pay-for-use.

Committed use slots are purchased at a discounted rate by committing to a specific term (one or three years). These slots can be assigned to reservations through baseline slots and any unused committed use slots are shared across other reservations through idle slot sharing.

Reddit’s Slot Management Strategy

About a year into adopting Editions, increasing usage and spend forced us to revamp our approach to slot management.

Assumptions

We rely on the following assumptions to break down this complex capacity model into clear decisions

• Reservation breakdown doesn’t affect performance
  • Given BigQuery’s low-latency autoscaling, a reservation’s effective performance is driven mainly by its total size, not the split between baseline and autoscale slots. 
• Reservation size is a usage lever
  • Increasing reservation size also tends to increase total consumption: as runtimes decrease, teams schedule more jobs and larger jobs. Planning to add autoscale capacity while holding usage constant is typically unrealistic.
• Total baseline should match total committed capacity
  • We assume every committed slot purchased should be allocated as baseline somewhere.  If we over-allocate baseline without commitments, we pay autoscale rates for always-on capacity without the benefit of scaling down. If we under-allocate, unused commitments flow into the idle pool and can increase overall consumption/spend.

Key Decisions

While the BigQuery editions capacity model offers granular control, it introduced 3 key questions regarding allocation:

1. Reservation Size

What should be the total size of a reservation (max reservation size)?

We abstract baseline, autoscaling, and committed use slots away from users. Reservation size is the only user-facing performance and cost lever. 

At Reddit, reservations are mapped to Domains (department/cost centre). Each domain has a slot budget, which they allocate across reservations that are tiered by criticality (From Tier 1, which is highly critical to Tier 4, which is for adhoc analysis). This decentralizes decision-making, allowing domain leaders to self-serve and reallocate slots within their budget. By budgeting at the domain level (rather than individual team or workload), it creates an internal opportunity cost: a slot used on a low-priority workload is a slot unavailable for a high-priority one. 

Additionally, budgeting on total slots and abstracting away baseline/autoscaling incentivizes teams to smooth out slot consumption through smart scheduling. Increasing a reservation’s size to run a workload at a peak time “costs” far more in slot budget compared to changing its schedule.

  domain: ads
  slotBudget: 3800
  reservations:
    - name: rtb-inference
      tier: "1"
      slots: 500
      teamName: ads-realtime
    - name: campaign-optimization
      tier: "1"
      slots: 1500
      teamName: ads-ml
    - name: advertiser-reporting
      tier: "2"
      slots: 1000
      teamName: ads-reporting
    - name: auction-analytics
      tier: "2"
      slots: 800
      teamName: ads-auction

2. Committed Use Purchasing

How many total committed use slots should we buy?

We have an ETL pipeline that analyzes historical slot usage across the entire platform and simulates committed use and autoscaling cost across various commitment levels. It generates recommendations for committed use purchases with savings estimates, identifying commitment volume with the minimum total cost.

3. Baseline Slot Allocation

How can we allocate our total committed use slots across reservations?

Given a set of reservations, each with a set number of slots (from 1), and a global total number of committed use slots (from 2), we have to decide how many committed use slots should be allocated (as baseline) to each reservation. That is, we need to determine the baseline/autoscaling breakdown for each reservation (such that total baseline equals total commitments).

We developed an algorithm for dynamic baseline slot allocation that runs hourly, allocating baseline slots to the reservations that are most likely to use them based on historical slot usage data. This allocation process determines the breakdown of baseline slots and autoscaling in each reservation, not the total reservation size. This maximizes baseline slot usage and minimizes autoscaling.

Outcomes and Conclusion

This structured approach to BigQuery slot management has been extremely successful at Reddit. Over the past year, we’ve flattened BigQuery compute cost growth and reduced our unit cost of slot usage by 21% (due to more committed use, less autoscaling).

We simplified the interface by abstracting away baseline slots and autoscaling from our internal stakeholders. We created an incentive structure to smooth out slot usage through budgeting by total slots, encouraging users to be capacity aware and schedule workloads at off-peak times to see better performance. Then, smoother usage helps us justify committed use slot purchases to reduce unit cost.

Upcoming Challenges

While our current approach to BigQuery capacity management is fairly cost efficient, we have identified 2 key areas for improvement around reliability and resource allocation:

Idle Slot Dependence

One challenge we have is idle slot dependence i.e. some users/workloads become reliant on idle capacity. When baseline slots go unused in the reservation they’re assigned to, they’re shared across other reservations, allowing other reservations to exceed their capacity. Despite fairly efficient baseline slot allocation, we see frequent idle baseline slots because it’s often cost optimal to aggressively purchase committed use slots. While idle slot sharing minimizes wasted capacity, users can inadvertently build workflows dependent on this idle capacity. When utilization is high across the org and idle capacity dries up, users who are dependent on idle slots experience significant performance degradation. We have plans to partially address this with domain-level reservation groups, and potentially limiting access to idle slots.

Starvation Order

Another current gap in our platform is the ability to effectively manage resource starvation across tiers. Ideally, higher-tier or priority workloads take capacity precedence when SLOs are not met. However, under the current BigQuery spec, we can’t enforce priority-based resource allocation, while keeping needed capacity levers and limits.

Written by Daniel Peters, Aleksandr Plentsov, and Anand Natu.

The Why

One of Reddit’s core differentiators as a platform is the tremendous variety and depth of authentic human conversations that happen on the site, covering a huge variety of topics ranging from shopping advice to niche media fandoms. Subreddits allow entire communities to organize around individual topics (e.g. r/malefashionadvice), and posts within these subreddits go even deeper on a specific issue or question (e.g. the best men’s t-shirt under $50). 

From an ads standpoint, contextually relevant ads are naturally aligned to this structure; we can leverage deep, specific context to place ads where they’re genuinely valuable to users, and therefore are most likely to be efficient and performant for advertisers. This blog post describes our efforts as a company to implement context-aware ad selection into our delivery systems, and what we’ve learned along the way, by

1. Motivating the problem (why and how is contextual advertising good for Reddit users & advertisers?)
2. Defining the solution path (how did we improve the contextual relevance of ads?)
3. Identifying learnings and opportunities for further work

Today, we have three main categories of placements on Reddit as shown in the visual below: Mixed feeds (e.g. Home feed), Subreddit feeds (e.g. feed for r/espresso), and posts (e.g. an individual conversation page within r/espresso). 

Intuitively, it’s easy to hypothesize that posts represent the best opportunity for contextual advertising, since the context is very specific (e.g. showing an ad for an espresso grinder on a post asking which one is best). To prove out this hypothesis, we sought to validate the effect of contextual ads on business outcomes - specifically ad performance (e.g. do relevant ads drive better click-through / conversion rates).

The How: Definition & Ground-Truth

Our first step in proving out the above relationships was to create a ground-truth definition for contextual relevance for posts; given a <post, ad> pair, how relevant is the ad to the context of the post? Our first iteration leveraged existing content understanding artifacts, specifically the IAB taxonomy labels we apply to posts (see this blog post for more details); wherein ads and posts were considered relevant to each other if they had the same IAB taxonomy label, with more granular labels in the 3-tiered taxonomy hierarchy translating to a higher degree of relevance. This let us quickly prove out a promising offline correlation to performance, i.e. <ad, post> pairs with matching IAB categories demonstrated better performance metrics, with a monotonic increase in performance from no match to a Tier 1/2/3 match. This motivated further work to address multiple limitations with the IAB approach, specifically:

• IAB labels are a proxy for contextual relevance, but not an explicit definition of it, making them structurally unsuitable as ground-truth. 
• IAB labels often lack granularity for certain relevance assessments, e.g. two different posts about Kubernetes and Twitter both fall within the same IAB Tier 3 category, meaning the taxonomy has no room to further differentiate these posts (even though we know they’re about materially different topics).
• IAB labels are rigid and don’t allow us to characterize posts that fall within intermediate / intersectional states (e.g. a post about Auto Insurance could be relevant to Automotive or Insurance IAB categories).

Accordingly, we needed a purpose-built ground-truth labeler for contextual relevance; LLMs were a promising choice for this task, since language models are well-suited to the nuanced semantic analysis and inherent ambiguity of this problem. 

We evaluated several variations of models and prompts by measuring agreement against a golden dataset of human labels. We found that using Gemini 1.5 Flash (now Gemini 2.5 Flash Lite) provided the right balance of quality & cost. Our prompt used a few-shot approach, with simple definitions of our relevance criteria (No/Low/Medium/High) and a handful of examples for each. We found that these LLM labels aligned to human labels at a comparable rate to the intrinsic alignment between any two human labelers. We further improved alignment by labeling more golden data and performing SFT (supervised fine-tuning) of the LLM. 

Finally, we built an Airflow pipeline to sample a large set of real <ad, post> pairs daily, and label them with the LLM prompt using BigQuery’s ML integration. These labels served two purposes:

1. We used them to continuously evaluate the relevance of the ads we were serving
2. We could also use them to build up a data set for evaluating & training a relevance model

Assessment of these LLM labels with respect to performance lift also indicated that they were better predictors of relevance than IAB labels:

The How: Inorganic Experiments

After developing an LLM labeling schema for ground-truth post<>ad relevance labels, we shifted focus to improving the delivery funnel’s ability to serve contextually relevant ads. The funnel consists of the following sequential components:

• The targeting layer considers advertisers' criteria to select eligible ads;
• Then, light rankers narrow that list down; 
• Heavy rankers predict calibrated probabilities for performance outcomes: CTR / conversion rate
• The final ad is selected in the Auction to maximize the utility function (roughly, P(outcome) * Value, e.g., p(CTR) * Bid).

Each of these stages represents a possible source of error / root-cause as to why a contextually relevant ad is or is not served for a given impression. Because of this complexity, the fastest way to prove out a treatment online was to apply an intervention in the auction to systemically induce more relevant ad serves. Using IAB tags as an online relevance proxy, we ran two experiments:

1. First, we ran a “filter” experiment wherein non-relevant candidates (i.e. w/ no IAB category match) were excluded outright from heavy ranking on the treatment slice.
2. After the filter yielded positive results, we developed a more balanced approach by applying a Utility boost to relevant ad candidates based on their degree of relevance (Tier 1/2/3 IAB category match). This led to more balanced performance improvements, especially for lower-funnel objectives (conversions). 

Selecting for User Intent

The lower-funnel performance bias we observed came with another hypothesis about the relationship between contextual relevance and user intent. Breaking out experiment traffic based on predicted user intent showed non-uniform results, wherein passive / low-intent users showed worse ad engagement,  while high-intent users disproportionately benefited. One of the best proxies we have for high user intent is the referral source of the impression; millions of users visit Reddit every day from search engines like Google. That implies both the presence of high intent through search, and the Reddit post context becomes a descriptive proxy for that intent. Accordingly, applying the auction boost conditionally (preferentially for search-referred traffic) helped us further refine the treatment and drive performance.

The How: Organic Treatments

Relevance using Embeddings

After proving key relevance hypotheses through the above experiments, we finally set out to tackle the challenge of improving contextual relevance via the core delivery systems outlined above. This involved developing a predictive model that could vend relevance scores for <post, ad> pair, while meeting the scale and latency demands of online inference in the auction, while simultaneously addressing the shortcomings of the incumbent approach (using IAB categories).

Embeddings are a classic solution to this problem; we used our large dataset of LLM labels to evaluate several pre-trained embedding models by generating embeddings from post & ad text, and measuring how well post<>ad cosine similarity predicted the underlying LLM relevance labels (using PR AUC). For instance, in one iteration, we found that Stella (stella_en_400M_v5) performed best. 

Fine-Tuning

The generic embedding approach described above met our performance needs for online ad serving, and had better generalizability and representative power than IAB tags. From there, we refined the construction of these embeddings to more explicitly capture elements of contextual relevance:

• Complementary subreddit features: Subreddit context is often a crucial signal to understanding posts, so we used pre-existing subreddit embeddings as an additional feature for the experiment variants.
• Leveraging ad landing pages to enhance the ad representation with an LLM-generated summary of the landing page contents.
• Leveraging product attributes for product-centric ad formats (e.g. Dynamic Product Ads) to improve representational power (e.g. product brand, type etc.) 

To implement these improvements, we built a multi-tower model using pre-trained Stella as our encoder for text features, and learning or reusing representations for other features.

Our initial training set consisted of millions of <ad, post/product> pairs, sampled daily from real served impressions. Training on this dataset was suboptimal for two reasons: 1) the relevance label distributions were constrained by the existing level of relevance on the platform (which we know has room for improvement) and 2) a small number of posts and ads were over-represented in the set of labeled pairs. To address these gaps, we rebuilt the dataset to include pairs that hadn’t actually been served in real impressions. This let us better control the distribution of labels, and ensure every different type of post/ad/product was represented in the training set. We started by sampling N diverse ads/posts/products using embedding similarity as a diversity measure, ensuring adequate representation. We then constructed <ad, post/product> pairs from this set, using embedding similarity to try and build a consistent number of positive and negative samples for each ad & post in the dataset. This new training set was labeled with the same LLM prompt, and performed much better both qualitatively + quantitatively.  

Results & Integration

As we’ve done with pretrained embeddings, we used the LLM's labeled data for evaluation of each relevance treatment, with the following (normalized) results demonstrating the significant improvement of fine-tuning:

Besides simply using more inputs, there are various intuitive explanations for the better performance against general-purpose embeddings, including (i) divergence in attributes that are important for general semantic similarity vs. contextual relevance, and (ii) the asymmetric (“post”=>”ad”) nature of the relevance problem compared to general text representation. Finally, we validated these fine-tuned embeddings against an outcome variable (performance) and recovered the same trend:

Today, these fine-tuned embeddings have been systemically integrated into all of the major modeling steps in the funnel: for targeting and retrieval, and as features in light + heavy rankers for different objectives (clicks, conversions, etc.). This has resulted in online improvement which is directionally consistent with our offline results and hypotheses, incremental to the gains from our inorganic MVP solution (boosting). 

Since we can compute this similarity at scale, we also integrated it in our online experiment platform and started to use it as an “FYI” metric in all the tests we run - we don’t use it for launch decisioning today, since there’re other factors at play (behavioural relevance, performance, etc.), but it helps inform / validate hypotheses about the relationship between contextual relevance and other auction & ranking variables. 

What’s next

We’ve made a lot of progress on the relevance front in the last few years, but there is so much more to do. We continue to work on improving guidelines, metrics, and embeddings; look for better ways to integrate them into the funnel and break the feedback loops and biases in the data we train our rankers on; we seek a better understanding of when relevance, in general or contextual relevance specifically, is a must… stay tuned!

Acknowledgements

It took a village to work on this!

• Eng: Ted Ni, Andrea Trianni, Alessandro Tiberi, Clement Wong.
• Product: Looja Tuladhar, Lillian Kravitz. 
• DS: Ryan Sekulic.

Written by Stas Kravets

TL;DR:

The performance of GraphQL service is crucial in a distributed system since it is usually a common facade for the whole ecosystem. In turn, GraphQL stability depends heavily on the performance of its dependencies. In this blog post, we will discuss how to protect GraphQL from dependency failures, high latency, and traffic spikes with timeouts, circuit breakers, and load shedding.

Background

GraphQL is a modern way for web clients to fetch data from multiple services in an ergonomic manner. The client sends an information request query to the GraphQL service, which then collects the required data from different parts of your ecosystem, stitches it together, and returns the final result to the client (also known as "declarative data fetching").

The entire business domain is represented as a graph of entities, their relations, and operations. Clients do not need to know anything about the services running in a distributed system, how they depend on each other, what their endpoint addresses are, and so on. Very elegant, in theory.

Let’s use, as an example, an imaginary document processing service. Each user in the system has an account with the payment information and multiple documents. There are also statistics for the specific account, documents, and period.

In many cases, the information needed by the client is just a few parallel calls away. For example, this might be enough to display the home screen:

1. Fetch user name and e-mail address from the Account service
2. Fetch the user document list from a Document service
3. Fetch user payment information, e.g., Free or Premium, paid until date

This is fast and simple: the client makes a single call, GraphQL authenticates the user, and then uses internal calls to fetch the data in parallel. This works very fast because GraphQL is co-located with other backend services.

Now, imagine we need something more complex, like “Show the statistics for the last payment period”, the request will look like this:

• Retrieve the user account and the payment information
  • Get the statistics based on the payment information

That means the sub-requests are no longer parallel but sequential; see the diagram below.

Clients do not need to care whether calls are parallel or sequential; GraphQL handles it all.

But now let’s think about what could go wrong in such a setup? The answer can be boiled down to two letters: IO.

Difficult Dependencies

Because GraphQL is a stateless server application that calls multiple other services to collect, convert, and combine data, its availability depends on the combined availability of those backend services.

These services can misbehave in different ways:

1. Be slow.
2. Return validation (HTTP 4xx) errors or internal (HTTP 5xx) errors.
3. A combination of the first two.

The way you approach each of these problems varies based on your SLA, traffic volume, and the criticality of a particular query. Let’s discuss them separately.

Timeouts

How long are you ready to wait for a response? In some cases, such as waiting for a $1,000 transfer to complete, you can be very patient and wait, keeping your hands off the Refresh button. For others, like waiting for the achievements page to load, even 2 or 3 seconds is too long. But there are three things we can be sure about:

1. Nobody wants to wait forever.
2. Waiting is not free.
3. In GraphQL, the waiting time is either the maximum response time across parallel requests or the sum of the response times for sequential requests. Most often, it is both.

The second point is critical: if your service is under heavy load and experiences a sudden slowdown, requests will begin to pile up. Imagine you’re in line at the grocery store and the payment system is frozen. Even if other cashiers are available to help, they will face the same issue, so the queue grows quickly, and some customers might just leave.

In math, this is called Little’s Law: the average number of customers equals the average effective arrival rate multiplied by the average time that a customer spends in the system. If your response time keeps increasing, the GraphQL server will eventually run out of memory or I/O resources and collapse. And it might happen just because one important backend is slow.

How do we address this? All modern network clients (HTTP, Thrift, gRPC) support setting a call timeout, which specifies how long to wait for a response. This is very important for “front line” services: those that are first in the sequential query resolution call chain. It is important to remember that: 

1. The timeout should be reasonably low, something like P99 of normal latency multiplied by two. Setting it very high will do nothing.
2. There is no guarantee that timeout detection will trigger exactly at the specified time in every case. This might sound funny, so let us explain.

If you write a simple client-server application with just one API: wait for the specified number of milliseconds (e.g., 100ms) on the server side, set the client timeout to the same value, and run it, your observed P99 of request latency will likely be very close to what you expect: 100ms.

You will see something very different if you try to do the same with 100,000 RPS on a service that is actively handling other tasks. Now the operating system is so busy with computation that it has way less resources for timeout detection, so actual timeouts will be longer than expected.

In a highly loaded Python application, we observed that only the P50 response latency was within the expected timeout. This inefficiency in concurrency is part of the reason we migrated our GraphQL stack to Go.

Most of the time, you use a Linux distribution like Ubuntu or Debian on your server. These OS’s are not Real-Time Operating Systems, and therefore, they do not guarantee that your time-bound operations will work at the exact specified schedule. The only way to improve your timeouts is to lower the load, which in turn means you will need more hardware. The best way to save on hardware is to use a high-performance programming language. You will need to strike a balance between the effectiveness of timeout detection and operating costs.

So, why bother setting timeouts if they only work some of the time? The answer is simple: even a 50% chance of the request timing out correctly might make a big difference, allowing your service to recover.

There is another interesting caveat. Imagine all your backends are working fine, but your query is so large that it still takes forever to load. This leads us to use two timeouts: one per backend, as discussed above, and one per query.

The query timeout is typically longer than the backend timeout (in seconds rather than milliseconds for the backend). Go makes setting timeouts very easy with context cancellation: just add a middleware to do that at a very beginning of each request. Query timeouts prevent hanging requests when multiple backends are slow – at some point, you just abandon the query resolution if it takes too long and return an error to the clients. Let’s talk about the errors next.

Errors

Errors are a part of our daily life. From the GraphQL point of view, it is usually a backend error, and we have three choices:

1. Return the error to the client, specifying which path in the query has failed.
2. Return a default value (e.g., empty list) instead, while logging the issue and/or increasing the error rate metric.
3. Retry.

The first one is the simplest: “Now it's your problem, pal.” Depending on the query's nature, this might be fine. Not all query fields are critical, after all. 

In other cases, the client may retry, and it is crucial to ensure that clients do not cause a “retry storm” that effectively breaks the backends. This requires some degree of standardization of the client-server interaction, so the client knows when to retry, how many times, etc. Beware of cascading retries, though! Imagine your client wants to retry, a proxy before GraphQL also wants to retry, and some backend clients are configured to retry. You might end up with the normal backend traffic volume amplified by an order of magnitude.

The default value might also be helpful with non-critical fields. Sometimes it is also good to return a warning to the client, stating that certain fields have failed, but ensure the response body remains valid.

The backend endpoint design can also affect query resolution performance. The preferred approach is to implement it in a batched manner. For example, “give me the documents with IDs (1,2,3,4)” - just 1 backend request instead of  “for each ID in (1,2,3,4), give me the document” - N=4 requests. The latter approach is called a “fan-out” style and is much more error-prone because you have to wait for additional requests to complete to return the data. In a worst-case scenario, if three of the four calls were successful and one failed, you’ll need to make all four calls again. If this is combined with the retries, your service is unlikely to survive. We have implemented linters to prevent contributors from inadvertently exposing their own services to risk.

Some errors are transient, and in this case, a retry can resolve the issue. But what if something is really wrong and the backend is completely unresponsive or attempting a recovery? In this case, it is good to take a break.

Circuit Breakers

Trying to handle every request perfectly in a highly loaded environment might be economically unreasonable. If some of your backends become unresponsive, you start triggering availability errors due to timeouts. Maybe the dependency itself is so severely broken that it does not want to talk to you in a normal, 200-way manner, no matter what.

In this case, it makes no sense to try calling it again. Rather, it's better to “fail fast” and return an error right away, giving the backend time to recover. This pattern is often called “Circuit Breaker”, and it has saved us many times. The circuit breaker is configured to trigger after a certain backend availability threshold - for example, if 30% of requests fail within the given time period. When triggered, it returns either an error or a default value without calling the backend.

Then the breaker enters the “testing” state after a predefined delay. In this state, it begins routing a small portion of backend traffic to verify that the backend has recovered and can serve at a normal rate. This way, we give the service a chance to recover, for example, by horizontal scaling (adding more service instances), or by reducing the database load in case the storage suddenly becomes a bottleneck. 

What is important here is not just the threshold configuration, but error classification. Some errors are not really availability issues but validation (4xx) errors, e.g., BAD_REQUEST or UNAUTHORIZED.  In this case, it's important to ensure these types of failures do not trigger your circuit breakers - that's a failure of the client request, not of your system.

A small note - if you use the Thrift protocol, it is important to assign/map standard error codes to exceptions, like in HTTP/gRPC. It will also help to fine-tune your availability metric by excluding these validation errors from the overall statistics. We observed cases where, from a GraphQL perspective, backend availability improved by 20% after proper error classification.

Load Shedding

Now, when things are very bad, it is not just one service that is broken, but many. This is what we experienced with the Amazon DynamoDB Service Disruption incident. Many things misbehaved at that time - delays and errors were so widespread that circuit breakers and timeouts just could not handle it all, and the GraphQL service itself became unstable.

In such cases, you have to sacrifice some traffic altogether to remain responsive. We use an internal concurrency limiter library for Load Shedding. It functions as middleware, counting only GraphQL service internal errors (compared to an individual backend circuit breaker, which analyzes only this particular dependency).

If there are too many traffic-specific errors in the GraphQL (e.g., timeouts), we begin returning a 429 error for some requests until the system stabilizes. The concurrency limiter uses the AIMD algorithm to identify congestion and recover.

It works relatively simply: you start with a safe value and gradually increase the number of concurrent requests you can handle by 1 on each success. You multiply the threshold by a number less than 1 (e.g., 0.5) for each error, sharply decreasing the threshold to shed the load quickly. The result is a sawtooth-like shape when problems occur.

Traffic Classification

The most intelligent approach to load shedding is to distinguish between critical and non-critical queries, sacrificing the less important for the sake of the most important. This will require you to assign a priority to each GraphQL query you execute and to instruct the concurrency limiter to discard non-critical traffic first.

This query-level priority can provide additional benefits, beyond just smarter load shedding in GraphQL. First, you can propagate it to your dependencies so they can perform a similar, prioritized load-shedding. Outside of incidents, the backend also gains visibility into its role serving critical traffic, and can tune its performance and reliability accordingly. Quite often, we faced situations in which some backend owners were unaware of the significance of their service and later had to tune their alerts to make them more sensitive.

Another benefit is the opportunity to enable more detailed observability for critical traffic. We're always tuning our observability to operate within a finite cardinality budget, and while we want fine-grained precision when studying the Home Feed, this granularity is often overkill for background requests and niche functionality.

Conclusion

In a distributed system, using GraphQL for client convenience and optimization offers tangible benefits but introduces new problems to address. Most issues stem from backend dependencies, so all communication with them requires multiple layers of protection:

• Timeouts and Circuit Breakers are industry standards for managing individual dependency latency and unresponsiveness, helping the service recover by failing fast. Every dependency and every request should have a timeout configured.
• Make sure you classify errors correctly; handling validation errors is very different from handling internal service errors. For example, it makes no sense to retry on the first, but a lot of sense to retry on the latter.
• Load Shedding serves as a final defense during massive incidents (e.g., system-wide disruptions), using algorithms such as AIMD to throttle some traffic and keep the core service responsive.
• Traffic Classification is the most intelligent layer of protection, requiring business and engineering alignment to prioritize critical queries over non-critical ones, ensuring the most important features remain available during high-stress periods.

All of those measures will require a balance between the amount of traffic you are willing to sacrifice and the stability of the service. Unfortunately, this is not a “once and for all” decision; it is rather a dynamic threshold that requires periodical re-evaluation on both the engineering and business sides.

Written By Abinodh Thomas, Senior Software Engineer.

In our previous post, From Fragile to Agile: Automating the Fight against Flaky Tests, we detailed the inception of our Flaky Test Quarantine Service (we adoringly call it FTQS). That system marked a pivotal shift-left moment for us at Reddit. We successfully moved from a reactive, chaotic environment where our on-call engineers were constantly fighting fires caused by non-deterministic tests, to a structured, automated workflow by identifying flaky tests and quarantining them via a static configuration file committed to the repository. 

For a long time, this solution was excellent. As you can see in the previous post, it stopped the bleeding and it had a major positive effect on our CI stability and developer experience. But as our engineering team scaled and the number of tests we were running (and covering with FTQS) increased over the last two years since that post was written, the static nature of the solution became a bottleneck.

The Paradox of Configuration-as-Code

You might be wondering, why did we use a static file in the first place?

There is immense value in keeping test configuration right alongside the rest of the code. A static file honors the principle of Configuration-as-Code, ensuring transparency and version control. It guarantees that the configuration a developer has is based on the latest information they know about. Basically, it prevents a dangerous type of "time travel" error - imagine a test that was broken, then fixed in the mainline (main/develop) yesterday. If you’re working on a feature branch that you cut three days ago after the test was quarantined, you obviously do not have the fix in your branch since your branch was cut before the fix landed. If you relied on a single external source of truth, the system would know the test was fixed, but it wouldn't know if that fix was actually in your branch. The result? The test runs, fails, and leaves you confused about why an unrelated test is blocking your Pull Request (PR). A static file in the repository is a powerful solution to this problem as it protects us from this issue, by ensuring that we only run tests that we know are stable in that branch. 

But this strength became our weakness.

The "Rebase-for-Update" Friction

Consider the lifecycle of a feature branch in a high-velocity monorepo:

• Alice branches off of main in the morning to work on a cool new feature.
• Alice does not know that the main branch has a flaky test (Text_X) that will block her when she opens a PR and CI runs all tests.
• Later that afternoon, Test_X gets quarantined by FTQS, which commits an update to the quarantine configuration file in the main branch to stop the test from running.
  • Anyone that branches off of main now will no longer run Test_X.
• Next day, Alice pushes her work and creates a PR. Her CI build runs the flaky test Test_X as her quarantine configuration file is outdated, it fails, and her PR is blocked.   

Alice is now in a bind. To get the new quarantine list, she has to rebase her branch on main. This has several disadvantages: she is forced to perform a high-risk Git operation, potentially resolving complex merge conflicts in files she never touched, just to perform a low-value administrative task - ignoring a test. It typically invalidates the cache, which leads to increased build times. It also increases Alice’s cognitive load, as she now has to spend time investigating if the test that failed in her branch is a flake that has been actioned already, or if it is due to some change she introduced in her branch. Any CI builds that are triggered from her feature branch which hasn’t been rebased yet also waste resources, as we know the test is going to ultimately make the build fail. 

We realized we had a conflict of needs. We needed:

• History Consistency: Feature branches need to respect their current history (don't run tests I can't pass).
• Real-Time Knowledge: Feature branches should know about new problematic tests that are unrelated to their changes (don’t run tests that I know will fail).

Essentially, we needed a system that could decouple the list of tests to quarantine from the source code while maintaining strict synchronization with the state of the codebase, a sort of "Point-in-Time" Quarantine System.

The goal was to enable a CI job to ask a sophisticated temporal question:

"I am a build running on a feature branch that was branched off main from commit abc1234. Based on what we know now, which tests were flaky at that time, or have become flaky since, that I should ignore?"

This post details the architecture, implementation, and theoretical underpinnings of the Sequence-based Dynamic Test Quarantine System, a platform-agnostic service that linearizes Git history to serve precise, context-aware quarantine lists.

The Solution: Linearizing the Git Graph

Git is a Directed Acyclic Graph (DAG). It’s great for distributed work, but terrible for ordering events. Time in Git is ambiguous as clocks skew, and rebasing changes timestamps. We couldn't rely on timestamps to tell us if a test was flaky at the time a branch was cut.

We solved this by abstracting the Git history into a Monotonic Integer Sequence. We treat our mainline history as an append-only log similar to a database write-ahead log or a blockchain ledger:

• Commit A ➜ Sequence 0
• Commit B ➜ Sequence 1
• Commit C ➜ Sequence 2

This linear Code Timeline allows us to transform the quarantine problem from a graph traversal problem into a simple range intersection problem. Instead of asking, "Is Commit A an ancestor of Commit B?" (a computationally expensive graph traversal), we can simply ask, "Is Sequence(A) < Sequence(B)?".

It is important to note that this system relies on a linear history for the default branch. At Reddit, we enforce Squash Merges for all pull requests merging into main. This ensures that our history is effectively an append-only log of changes, allowing us to map every commit on main to a strictly increasing integer without worrying about the complex topology of standard merge commits.

System Architecture

The system consists of four primary, decoupled Go components that run as background services. This separation of concerns allows us to scale ingestion, validation, and serving independently.

• Sequencer: The source of truth. It maintains the SHA ➜ SequenceID mapping.   
• Sequencer Feeder: An ingestion engine that listens to GitHub webhooks and polls for new commits to populate the timeline.
• Sequencer Validator: The auditor. It periodically checks our database against GitHub to ensure that our linear history isn’t corrupt.   
• Quarantine Phase Store: The application layer that manages the lifecycle of a flaky test (Start Seq ➜ End Seq)

Technical Deep Dive

The following sections explain how each of these components works in detail:

Sequencer

The Sequencer is the heart of the timeline. Its only job is to maintain the SHA ➜ Seq mapping.

• Implementation: It uses a combination of an in-memory ring buffer cache with FIFO eviction for fast lookups of recent commits, and a PostgreSQL database for persistent storage.
• The Extend Function: This is the primary way to add new commits. It is designed to be idempotent and safe for concurrent calls. When called, it fetches the current max sequence number and increments it. Additionally, it includes a retry loop to handle race conditions where multiple processes might try to write to the timeline simultaneously. 
• The Lookup Function: First checks the in-memory cache (typically 99% of active feature branches will hit the cache). On a miss, it falls back to a database query and populates the cache.

Since main/develop is a high-traffic branch, we occasionally have multiple merges attempting to claim a Sequence ID simultaneously. To handle this, the Sequencer utilizes optimistic locking (using database-level atomicity) to ensure that two commits never grab the same ID. If a race condition occurs, one transaction fails safely, and our retry loop kicks in to grab the next available integer.

Sequencer Feeder

To keep the timeline current, we need to feed it commits. The Feeder ensures the Sequencer has a complete and up-to-date history of mainline branches.

• Backfill: On its first run for a repository, it fetches the last x months (configurable) of commit history from the GitHub API, sorts them by date (oldest to newest), and feeds them into the Sequencer via the Extend function. Before serving requests, the feeder gates on two readiness flags, dbSeeded and cacheWarmed, to ensure the timeline is properly initialized.
• Webhook: To achieve near real-time sequencing, the Feeder exposes an HTTP endpoint listening for GitHub push events. This allows it to process commits within <2 seconds of a change landing in the mainline branch.
• Polling: It runs on a configurable interval to fetch the most recent commits. It uses a lastProcessedSha anchor to avoid re-processing old commits. Poller helps us ensure that the (sacred) timeline has not been compromised if we drop webhook events or if the GitHub API is temporarily unavailable.
• Recovery Mode: If the polling falls behind, the system enters a recovery mode where it fetches a larger number of commits to find the anchor and bridge the gap.

Sequencer Validator

When you flatten Git history into a linear sequence, data integrity is very important, as any mistake here can cause a test that shouldn’t have been run be run in a build (or vice-versa). The Validator acts as the guardian of the timeline, ensuring the numbers in our database accurately reflect real Git history.

It runs periodically, fetching a window of recent commits from the database and comparing commits (e.g., seq 100 and seq 101) using the GitHub compare API. It looks for two specific anomalies:

• Drift: The sequence order in our database does not match the ancestry in Git (e.g., seq 101 is not a descendant of seq 100). This usually happens due to force pushes or history rewrites.
• Distance Anomaly: The difference in sequence numbers (e.g., 105 - 100 = 5) does not match the actual number of commits between the two SHAs as reported by GitHub.

If anomalies are detected, it logs detailed errors and emits metrics for manual intervention (likely wiping the history and backfilling it). For continuous validation, a sample of API requests also triggers asynchronous ancestry checks (via GitHub compare API) to verify phase boundaries are correct.

Quarantine Phase Store

The Quarantine Phase Store is the application layer that sits on top of the Sequencer infrastructure. It translates raw flakiness data into actionable Phases. A phase consists of a start_seq (when the test broke) and optionally, an end_seq (when the test was fixed).

• Opening a Phase: When our data pipeline detects a new problematic test, it goes through the test metrics to identify the earliest known record in recent history when this test started having problems at scale. In the vast majority of cases, this corresponds to the change that made the test flaky. We record the sequence related to that commit SHA as the start_seq.
• Closing a Phase: When a JIRA ticket associated with a flake is moved to "Done" (the signal we use to determine if a fix has been implemented), we verify the fix and record the sequence related to the current HEAD commit as the end_seq.

The Serving Algorithm: Context-Aware Intersection

The beauty of this system is how simple the client interaction becomes. The client (generally, a CI job) can determine which tests to skip by making a single GET request with the Merge Base Commit SHA of its feature branch, which is the most crucial piece of information as it represents the point-in-time in git history the feature branch was cut. 

Once the system receives this SHA, it then finds its sequence number (e.g. 500) from the CommitSHA <-> Monotonic Integer Sequence map in the Sequencer. The service then performs a temporal query:

"Find me all tests that started flaking before Sequence 500, and either haven't been fixed yet, OR were fixed after Sequence 500."

The system achieves this by querying the database for all quarantine phases where the given sequence number falls between the phase's start_seq and end_seq (or the end_seq is NULL, for a test that hasn’t been fixed yet).

Now let’s look at some scenarios that shows how powerful this system is:

• Scenario A (The Future Flake): If Test_F started flaking at Sequence 505, and we are at Sequence 500, the system EXCLUDES it from our quarantine list. Even though the test is flaky in the future, our code is based on a point in history (Sequence 500) where the test was considered stable. If it fails in our branch, it is likely that our changes caused a regression.
• Scenario B (The Fixed Regression): If Test_G was fixed at Sequence 400, and we are at Sequence 500, the system EXCLUDES it from the quarantine list. Since our feature branch was cut after the fix was merged, the branch includes the fix. If Test_G fails for us, we likely broke it again (a regression).
• Scenario C (The Active Flake): If Test_H started flaking at Sequence 450 and isn't fixed yet (or is fixed later at Sequence 600), and we are at Sequence 500, the system INCLUDES it in the quarantine list. Our feature branch is based on a version of the code where the test is known to be broken. Even if the test has since been fixed, since the fix was merged in after we cut our branch, we can ascertain that the test will fail if we run it, so we skip it.

This dynamic context-awareness means developers never have to rebase just to get an update to their quarantine config. They get the correct list for their specific point in history, every single time.

An Illustrative Example

The diagram below provides a practical example of how the dynamic quarantine system determines which tests to skip for different developers working on separate feature branches

Deconstructing the Diagram

1. The Timeline: The top of the diagram represents the mainline branch's history moving from left to right. Each commit (e.g., e93ebae...) is mapped to a unique and sequential integer (0,1, 2, etc.). This is the core timeline created by the Sequencer.
2. Quarantine Phases: The red bars represent the quarantine phases managed by the Quarantine Phase Store. Each bar has a start and end point on the sequence timeline, indicating the exact period during which a test is considered flaky.
   • TEST A is flaky between sequences 1 and 5, and again from 7
   • TEST C is flaky between sequences 3 and 6
   • TEST F is flaky from sequence 4
3. Developer Scenarios: The three developers - Charlie, Bob, and Alice, represent engineers who have created feature branches from the mainline at different points in time.

How the System Determines the "Skip/Ignore" List

The system generates the quarantine list by drawing a vertical line through the timeline at the sequence number of the developer's merge-base commit. Any flaky phase that this line intersects is added to the list.

• Charlie branched from commit e93ebae... (sequence 0). The line at sequence 0 does not intersect any red bars. Therefore, his quarantine list is empty.
• Bob branched from commit e1b0e98... (sequence 6). The line at sequence 6 intersects the red bars for TEST C and TEST F. Therefore, his quarantine list is [TEST C, TEST F].
• Alice branched from commit a161ed9... (sequence 7). The line at sequence 7 intersects the red bars for TEST A and TEST F. TEST C is no longer flaky at this point. Therefore, her quarantine list is [TEST A, TEST F].

Fallback Mechanism

The final component of this system is a fallback mechanism when this system is down or unavailable. We achieve this by maintaining a configuration file in the repository that is updated at a regular interval. Before running tests, the system will attempt to call the API and get the most recent quarantine configuration for its merge base. If the connection succeeds, we use the configuration returned by the API, and if it fails, we fallback to the in-repo configuration file. 

In a follow up post, we will go in-depth about our Test Orchestration Service (which we adoringly call TOAST), about how it does test quarantining (among other cool things!), and how this dynamic quarantine system fits inside it. 

An Important Caveat

One of the most important parts of the system is the component that determines when a problem started by sifting through the test run metrics. For this to work, we need to be able to accurately connect a regression to the test code, or code that test covers. For instance, if a test is written in such a way that it talks to an external system, like a server, and it gets flaky due to networking issues, we cannot accurately tell whether or not the test failed due to an external issue. At Reddit, we have put a lot of effort into ensuring that most of our tests are self-contained, use mocks, and do not talk to external systems. However, we still have a handful of tests which could potentially fail due to other reasons. We have systems in place to detect failures like these (that happen across multiple feature branches, irrespective of their git history), where they are “globally” quarantined instead. 

Conclusion

By moving to this sequence-based dynamic model, we achieved three major wins:

• Zero Rebasing: Developers no longer need to rebase just to pick up updated quarantine configs. They can simply re-run the failing CI job to ignore/skip an updated list of flaky tests.
• Precision: We provide a precise, up-to-the-minute list of tests that should be quarantined.
• Future-Proofing: This code timeline concept gives us a foundation for future analysis, such as pinpointing exactly when bugs were introduced.

If you are struggling with flaky test management in a high-velocity monorepo, consider linearizing your git history. It turns a complex graph problem into a simple integer comparison. If this kind of complex distributed systems engineering excites you, check out our careers page. We're hiring!