A few years ago, I worked through Sutton & Barto as part of the University of Alberta's Coursera course on reinforcement learning. I found the math to be incredibly complex, with various branches into algorithms that seemed to be ancillary to the main progression toward deep RL. Some of the most important concepts were left as "exercises for the student."

I'm currently working on a video series that teaches the basics of using deep RL (namely PPO) to train an ESP32-based balance bot. Additionally, I wanted to solidify my understanding of the underlying math. As a result, I am actively working on a set of blog posts that build up to PPO, starting from the very basics.

In other words, my goal is to create a quicker and more approachable text to bring newcomers up to speed on the math behind modern deep RL algorithms. I've listed the posts below, and I would appreciate any feedback you might have! I'll update this list as I continue to add articles.

1. What is Reinforcement Learning?
2. Rewards, Returns, and the Discount Factor
3. Policies, Markov Decision Processes (MDPs), and Trajectories
4. Expected Return, Value Functions, and Bellman Equations
5. The Bellman Optimality Equations
6. Dynamic Programming
7. Monte Carlo Methods
8. Temporal-Difference (TD) Learning
9. TD(λ) and Eligibility Traces
10. Q-Learning
11. Deep Q-Networks (DQN)

I'm training an AlphaZero-style agent (Gumbel MuZero via DeepMind's mctx, in JAX, single RTX 5070) and found that the MCTS search cost grows superlinearly with the simulation budget: at 16 / 32 / 64 simulations per move the training throughput was 143 / 47 / 9 episodes per second. Doubling the budget should roughly double the cost, not triple it — so something in the search was superlinear.

Isolating the tree machinery (network replaced by constants, then the environment removed too) showed it is O(N²⁾ all by itself: the pure per-simulation tree overhead measured 0.47 / 0.52 / 1.05 / 1.95 ms at 8 / 16 / 32 / 64 sims — the per-simulation cost doubles every time the number of simulations doubles.

The compiled HLO showed why. mctx's backward pass carries the whole search tree through a lax.while_loop and, within one iteration, both gathers from and scatters into children_values / children_visits. XLA:GPU cannot prove the scatter can alias, so the compiled loop body contains a full copy of both [batch, nodes, actions] buffers on every step of the leaf-to-root walk. Buffer size grows with the simulation count and the walk depth grows with it too, so the whole thing is quadratic.

The fix carries only an O(num_nodes) path through the loop and applies one scatter per array after the loop. Search results are bitwise-identical (same rng gives the same actions and action_weights), and the full upstream test suite passes, including the golden-tree comparisons. End-to-end training speedup: x1.10 at 16 sims, x1.58 at 32, x3.27 at 64; XLA compile time at 64 sims dropped from 63 s to 19 s.

Two gotchas that cost me time: 1. Top-K action sampling — the "textbook" fix for large action spaces — gained nothing once the copies were gone; the copies were the A-scaled term, not the per-action math. 2. Consumer-GPU clock ramp-up (495 -> 2932 MHz) makes cross-process benchmarks lie by up to x1.7 — all A/B numbers above are interleaved in a single process.

Full write-up with HLO dumps, benchmark methodology and the things that didn't work, plus the PR, are linked in a comment below.

I recently spent some time modernizing NVIDIA’s CuLE (2019), the GPU-native Atari environment. It was tied to an older software stack (CUDA 10, atari-py, older PyTorch), so I wanted to see if it could be brought up to date.

The fork now builds with CUDA 12.x, PyTorch 2.x, Python 3.12, Gymnasium, and ale-py (so ROMs come bundled).

I also compared it against EnvPool on my machine (RTX 4090 + i5-13600K). For PPO, CuLE reaches about 38k SPS vs EnvPool’s 21k (~1.8× faster). For raw environment stepping, the crossover is around 512 environments—below that EnvPool is faster, but above it CuLE continues to scale while EnvPool plateaus, reaching up to ~114k steps/s at 4,096 Breakout environments (~3× faster on pure stepping).

To make benchmarking easier, I also added training scripts based on CleanRL and LeanRL (PPO, DQN, Rainbow, C51, PQN, SAC), alongside the original CuLE implementations.

I’d be interested to hear if anyone here is still using CuLE or GPU-native Atari environments, or if there are additional benchmarks you’d like to see.

Repository:
https://github.com/MehrdadMoghimi/cule

Hey everyone,

RLHF alignment using standard Proximal Policy Optimization (PPO) can be notoriously tricky to stabilize during LLM post-training due to policy collapse and high sensitivity to hyperparameters.

I built Tpo-torch to explore Target Policy Optimization (TPO) as a cleaner, more stable alternative for preference alignment directly in PyTorch.

Key Focus Areas:

• Mitigating policy collapse without requiring aggressive KL-divergence penalties.

• Modular, lightweight, and readable implementation designed for research and custom fine-tuning pipelines.

• Integrated stability benchmarks comparing policy drift against standard PPO.

I'll drop the GitHub repository link in the comments below! I'd love to hear feedback from anyone experimenting with alignment, preference optimization, or RLHF.

Repo link : [https://github.com/Griffith-7/Tpo-torch.git\](https://github.com/Griffith-7/Tpo-torch.git)

Fala, pessoal. Queria trazer uma dúvida/observação pra quem mexe com RL, porque faz tempo que ela não sai da minha cabeça.

Eu rodo um mundo simulado (o re·genes) onde vivem várias “espécies” de agentes na mesma ecologia. Cada organismo come, gasta energia, reproduz e morre num servidor que fica ligado 24/7. Duas das espécies são interessantes pra essa comunidade:

A primeira é um Q-learning tabular clássico. Reward por delta de energia: +50 quando ganha, −1 quando perde, −0,1 neutro. Aprende durante a vida, guarda a Q-table entre encarnações. RL de textbook, funciona do jeito esperado.

A segunda não tem reward NENHUM. Nem fitness function. Os cérebros são redes neurais que só mudam por mutação e crossover entre gerações (neuroevolução, NEAT). A única pressão é: sobreviveu e pariu, passa os genes. Morreu, fim. Ninguém nunca disse o que é bom.

Eu esperava que o Q-learning dominasse, porque aprende em vida enquanto o outro precisa morrer pra aprender. Não foi o que aconteceu. Os cérebros evoluídos dominam o mundo faz meses. Mas o detalhe que quebrou minha cabeça foi outro: semana passada fui olhar dentro do genoma do campeão (geração 260) e ele tinha 458 neurônios registrados, dos quais exatas 3 conexões funcionais. Um arco reflexo. 99% do genoma era lixo acumulado, igual DNA não codificante humano. Ou seja, a “inteligência” que vence não é inteligência quase nenhuma, é o reflexo mais barato que resolve o problema.

Fica a provocação pra quem entende mais que eu: será que a gente não superestima o quanto de cognição os ambientes realmente exigem? No meu mundo, com reward ou sem reward, o que vence é sempre a política mais simples que não morre. O Q-learning pelo menos precisa do reward certinho pra funcionar, a evolução nem isso.

Se alguém tiver curiosidade, o mundo roda ao vivo e aberto (re-genes.is), dá inclusive pra plugar o seu próprio agente na arena e testar contra os dois. Mas mais do que isso, queria mesmo ouvir o que vocês acham: tem literatura boa sobre “recompensa mínima necessária” ou open-endedness que eu deveria ler?

Hi all!

I built this two wheel balancing robot and hacked together the hell out of it... Battery has black wire for both positive and negative (but sort of safe because XT connector is asymmetric), I have no buck converter to power the ESP from the 12V LiPo so added powerbank, cut open ethernet wires for my CAN bus and used 12V instead of 16V battery to power the motors.

Next version will be a lot cleaner and robust when the new hardware arrives, if people are interested I will post a new video with driving over slope and different materials (already working on this robot).

Stack used:

- OnShape for 3D design
- onshape-to-robot for XML creation
- mjlab for training policy

Hardware:

- ESP32 runs policy at 200Hz (~18k param total)
- MPU6050 accelerometer
- 3.3v CAN bus transceiver for motor control
- CubeMars GL40II Gimbal motors (direct drive)
- 12V LiPo battery
- All parts are custom design and printed on Bambulab H2D
- PS4 controller

Controller automatically connects to ESP on power up and shows different lights for different modes, I can calibrate through controller, stop/start policy and clear fault codes after falling.

Future state prediction is usually discussed as a training signal. I wonder whether the error after each action could also tell a policy when its current plan is becoming unreliable.

Raw error would be a poor trigger because many contact outcomes are naturally uncertain. It would need calibration by task phase and comparison with a policy confidence baseline. If calibrated error rises, the controller could shorten its action horizon, request a new observation, or stop.

LingBot-VA 2.0 uses Foresight Reasoning to predict future visual states before producing actions, but that does not mean its deployed policy exposes prediction error as a control gate. A study could save the predicted state, compare it with the next observation, and check whether a rising residual precedes failed contact or a distribution shift. It would matter only if unsafe commitments fall without an excessive number of false stops.

Solo dev project. I wanted to make imitation learning for 2D games accessible without writing any code.

You play the game, the app records screen + key inputs, and trains an EfficientNet-based model via behaviour cloning to play like you. The clip shows the same bot at 60s, 2, 4, and 8 minutes of training — you can see it go from crashing constantly to playing cleanly.

There's also human-in-the-loop fine-tuning: while the AI plays, you tap a key to take over for a moment, it records the correction, and keeps learning.

Training runs locally on your GPU or in the cloud. It's early and rough in places, but the core loop works.

Curious what this crowd thinks — especially on the fine-tuning approach, and where you'd expect behaviour cloning to break on games like this.

Link in the comments.

We just put out a short conceptual piece on something we're calling synthetic counteradaptation, basically trying to name a loop that keeps showing up in human-AI interaction but doesn't have a clean framework yet.

The idea: an AI system develops a strategy or protocol that looks strange or bad by human standards. Humans study it, extract whatever's useful, and change their own behavior. Now the AI is adapting to a population of humans who have themselves adapted to the AI. This is different from a one-off transfer of knowledge because the loop doesn't close — it keeps running as both sides keep moving.

The example we lean on most is Go. AlphaGo's move 37 against Lee Sedol (the shoulder hit) was dismissed by commentators in the moment as a mistake. Within a couple years pros were incorporating it and similar shoulder-hit ideas into their own play, which changed the pool of strategies that later Go engines and players were training and competing against. The "novel move gets absorbed into human play" part is well documented; what we're pointing at is the second-order effect, that the target the AI is adapting to has itself shifted because of the AI.

Why I think this matters for multi-agent RL specifically: most of our evaluation setups implicitly assume a static human or a fixed opponent pool. Self-play against a frozen population, or a one-shot human baseline collected at a single point in time, can't capture this because the whole phenomenon is that the human side of the interaction is non-stationary in response to your agent. If your agent trains against or evaluates against humans-as-of-2023, and then gets deployed against humans who've read about your agent's own strategies, you're facing a moving target that your training process never modeled.

We don't have experiments in this paper, it's a conceptual framework paper, we walk through Go plus some mixed-motive social interaction and geopolitical simulation cases to show the same pattern recurring. But I think it has direct implications for how people think about opponent pools, curriculum design, and what a "human baseline" even means if you're claiming your system will be used repeatedly by people who can study and adapt to it.

Curious if others here have run into this in practice, especially anyone doing repeated human-AI play studies or long-horizon deployment work where the human side visibly shifts strategy over time. Happy to be told this is already handled somewhere and I've just missed it.

https://arxiv.org/abs/2606.15503

Full disclosure I've shared this in a few discord channels. I thought I should cast a wide net if I am to get some feedback.

Hey yall

wanted to share a project ive worked on for the past 5 months
It initially started as a way to make money on the side, but turned into a research project. Looking for some feedback both on the code and research methodology. I didn't go to school for AI and I've never done any research before (I got help from Claude).

it's a Soft Actor-Critic (SAC) reinforcement learning agent that trades US stocks using features derived from OHLCV (open/high/low/close/volume), VWAP, and transactions data, sourced from Massive (formerly Polygon.io). Early versions (1–4) operated on 4-5 years of minute-level data; later versions (5–7) moved to 22 years of daily bars. Versions 1-6 traded one stock and version 7 traded a five-ETF basket (equities, treasuries, gold, oil, and the dollar)

It did not beat buy and hold

It made money: +12% across three independently seeded runs on a held-out 2024–25 test window. The catch is that simply holding the same basket returned ~29% over that window, at the same risk. It captured under half the return of doing nothing, for an identical drawdown.

I should also mention I was only able to backtest on a bull regime.

here's the code and full write up

https://github.com/markk628/Project-Prometheus

Does the spider, casting his web, know why he is doing it? Can he foresee the insects that will fall on it?

I am not an expert in RL, but casually reading the Dreamer paper... it seems like this is the premise for Dreamer intelligence: that long horizon tasks get solved thanks to latent imagination.

I don't think that's what animals are doing. I don't think spiders are visualizing the future when laying the web in the perfect spot to catch insects...

But why I don't think so? I don't know... so what do you folks think?

I think they must be feeling some form of spider pleasure in anticipation. But they don't know why.

Same as dogs when they sense tsunamis and don't wanna go out. I don't think they actually know a tsunami is coming. They just know something is wrong.

What do you think? I am really unsure of my thoughts.

Hey, I've been experimenting with reinforcement learning lately, and since I am a game developer, I chose to do it in Godot using Godot RL Agents, which connects the environment to Python trainers.

I wanted to recreate the kind of space battles I enjoyed as a kid in games such as Homeworld, but using learned policies instead of scripted combat behavior.

The setup eventually became two separately trained policies. The pilot receives a 232-dimensional observation and outputs seven continuous controls for thrust, rotation and braking. The turret receives a smaller 45-dimensional observation and outputs aim X, aim Y and fire.

Both policies were trained with TQC from SB3-Contrib, a distributional extension of SAC. I used three critics with 25 quantiles each, truncating the highest quantiles to reduce value overestimation.

For the pilot, 25 agents train simultaneously with normalized observations, replay buffers and a curriculum that progressively increases asteroid density, target movement and the required number of attack cycles. A shared 3D voxel flow field provides route information, but it never controls the ships directly. The policy still has to learn the approach, attack, egress and reapproach sequence from continuous actions.

Once the pilot became usable, I froze that policy and trained the turret on top of it. Hits are sparse, so the turret also receives a ballistic lead solution and shaped rewards for reducing aim error. The curriculum gradually shrinks the accepted hit sphere and raises the success requirement from two to five hits, while penalties discourage firing out of range, during egress or through friendly ships.

Of course, I came to realize really quickly that this exact setup would not scale well to hundreds of RTS units with my current architecture. I didn't design this necessarily to be used at scale, but I couldn't help myself and I still tried to push it to the limit. The version you see in this video at the end of 25 vs 25 ships is the most I could get for the current setup.

I wonder whether imitation learning could scale better with a smaller observation pipeline and without extensive reward engineering. I could move RL one level up and use it only for tactical decisions, leaving navigation and aiming to conventional systems, but that would avoid the challenge I actually want to solve with this project.

Hi guys, I need three frontier models for my research benchmarks. Which option would you recommend: using APIs, models from Hugging Face or something else? I’m open to any suggestions or recommendations.

This currently uses MuJoCo as the physics engine and a way to define scenes / environments. So if you can define things there you can define instructions by playing it as a game and later on run that on hardware

Hi, i need some help in the form of ideas. Im trying to train a ppo but the model ends up just not playing, i've tried penalty for doing nothing but does not work.

https://reddit.com/link/1uzxzer/video/6ey80xdk20eh1/player

It can evaluate 2D UI control performance using a mouse. I hope it will be a new environment like Pick And Place.

I have been playing this game for a while and think it's a great environment to learn RL. Does anyone know the rules used to generate the numbers on the top blocks of the game or they are just random from 1 to 5 (as I saw)? Also, do you have any advice before I start?

I have been reading "Reinforcement Learning: An Introduction" by Sutton and Barto. Does anyone know which chapters from this book might help me the most with this project?

I’m taking a break from research and doing a little side project. I want to bring RL to a video game because staring at DMC all day is no fun, but there’s just so many to choose from… What are some games you would love to see tackled with RL, for any of the following reasons?
- hasn’t been done before
- has been done before, but not correctly in your opinion
- curious if it can be solved
- you just love the game
- maybe already solved before, but not from pixel observations

Only thing I am not actively looking for is multiagent (POMGs will bring me more frustration than happiness 🧌). Thanks!

5 detectives chase an invisible fugitive on a 199-station graph, seeing only which ticket he plays. I trained both sides.

The detectives are an R-GCN that eats an HMM-style belief state plus distance maps. BC warm start, then PPO: 25% → 79% win rate against the same MCTS Mr. X. Then Mr. X got a GNN too (21% → 35%), and finally I wrapped him in PUCT with policy priors and a value head, 16 sims per move, frozen GNN detectives playing the replies in the tree. That alone: 35% → 91.7% over 60 eval games.

Play it in the browser 
Video (20 min, scenes rebuilt from real logs): https://youtu.be/V0osfVtJUuI 
Code and models: https://github.com/Jacopo888/scotland_yard

Fair warning: most of this was pair-programmed with AI tools. Solo side project, it wouldn't exist otherwise.