r/StreetFighter • u/Trevor_Z • Sep 02 '25
Discussion I trained a neural network to predict the outcome of a round using only controller inputs
Enable HLS to view with audio, or disable this notification
TL;DR: The way you press buttons could be enough to predict the result of many of your matches, without any information from the game itself.
(Just to be clear, there's no hacking involved, and since the model doesn't interact with the game, there's no potential for cheating here. So far this is just an interesting experiment that might have some utility in the future as a research or maybe training tool.)
I'm a neuropsychologist from Brazil, and while doing some related research with another game last year, I got curious: how well could raw directional and button inputs predict SF6 round outcomes (win/lose), without any other information like what character you're playing, what moves you executed, opponent actions, health bar state, etc.?
The idea is that there are in-game behavioral patterns that indicate quality of play by themselves and correlate with winning chances. They should also correlate with psychological factors—if someone is confident or intimidated, overexcited or cool, that should be reflected in how they play. Incidentally, Brian F. and RobTV were discussing something in this vein in a couple of recent Extra Trashy videos.
So I've been extracting data from over a thousand of my own matches, and now trained a neural network model on this dataset to make predictions. It worked surprisingly well:
- The model correctly predicted 75% of a test set of 397 unseen rounds, assigning winning probabilities higher than 50% to actual wins and lower than 50% to actual losses at round end.
- Won and lost rounds show significant differences across most generated metrics, suggesting those raw patterns indicating quality of play exist and the model learned them.
The model processes inputs from replays after the actual match, outputting a running win probability that updates with every input from round start to finish.
The main caveat is that, as usual with deep learning, the model is a black box. I have yet to make sure it's actually doing what it seems to be doing, and not taking some surface-level shortcut.
Some additional notes:
- Dataset is me playing only Ryu, from Platinum to 1400MR-and-falling-fast Master. Don't know how well this generalizes to other characters.
- Maybe the model's prediction ability is already high because the way I play is predictable—trying to model other players could yield very different results and require different training strategies.
- Many rounds are decided by luck in the final interactions, so one's "quality of play" alone shouldn't have even close to 100% predictive power.
- The model is very light and xinput allows more than one application to poll the controller, so it's possible to run the prediction while playing.
- Despite my dataset being roughly 50-50 win/loss, the model clearly knows—I mean, clearly thinks I'm a scrub, since it starts most rounds giving me ~30% odds.
- The attached video shows a script plotting a previously made prediction in sync with the replay video, it's not running inference live. Just to show that the model clearly understands something about the game, even without actually knowing anything about it.
- Some more info: https://imgur.com/a/VqUkCNp
19
u/Co1iflower >:D Sep 02 '25
Very interesting work!
I'm curious on how quickly if at all the frequency of button presses affects the predictions. If you were to just mash buttons in neutral, would the prediction fall quickly? I suppose that would defeat the purpose of the test.
Completely anecdotal and non-scientific but I do find that when I personally feel like I'm "playing good" my inputs seems to come out smoother, and my button presses are more precise - as opposed to when I'm frustrated and mashing my combos haha!
12
u/Trevor_Z Sep 02 '25
If I'm not mistaken, the peak in input frequency here is me mashing desperately while being trashed by a combo: https://imgur.com/a/GyBGAU3
It seems to drop the prediction of victory, but since the game is pretty dynamic, the effect doesn't last long.
Also, the idea to do this first came when I was thinking about this feeling of "playing good", I wanted to see if it could be quantified somehow.
3
2
10
u/JegErEnFugl CID | 100Lamp | Merde Means I Love You Sep 02 '25
haha that’s awesome man, stockfish evaluation for sf6. if you’re taking additional data sets i’d be more than happy to provide.
5
u/CharlehPock2 Sep 02 '25
All along I thought people were talking about street fighter 6 when they said sf6, but they meant stockfish 6!
10
u/ImpossibleMorning12 Sep 02 '25
It tickles me that it starts at 25%. The NN is like "nah bro ive seen you play. 1/4 max"
7
5
Sep 02 '25
Have you tried testing it with a user mashing while getting hit or inputting a combo while being hit? Classic rule of data: Garbage in, Garbage Out. This is probably accurate because it’s running on a single data source which likely has a consistent pattern, and from my understanding, you’re not factoring in the inputs of the opponents?
3
u/Trevor_Z Sep 02 '25
Yeah, I talked a bit about mashing in another question, it affects the prediction, but not for long.
The model is absolutely overfitted to my behavior in ranked matches, the tuning is probably off even against the CPU in training, because there are no stakes there. But that's almost by design, given the dataset and the training settings.
And no, I'm just factoring my own inputs. The idea is to infer as much as possible just from the way one presses buttons.
4
u/darvos Sep 02 '25
I wonder if the model only applies to you. Like it's trained your specific patterns and ticks and would perform badly on someone else's game.
2
4
u/Merosian Sep 02 '25 edited Sep 02 '25
What architecture are you using for this? Some kinda regression model? You say it can detect input patterns so so you feed it prior inputs in a sequential manner (transformer, lstm, rnn)?
Would love deets on how you trained it generally (size, depth, hyperparams, etc...). How did you encode the time difference between inputs?
I'm not sure i understand how you can train it to update loss for every new input when all your labels have are the end result, though? Wouldn't it just predict once based on all inputs in the game instead of input by input?
3
u/Trevor_Z Sep 02 '25
Currently it's a bidirectional GRU, it's trained on full rounds, but at inference I query progressively to generate the plot. So it's not calibrated for that, and even then it's able to differentiate between winning and losing rounds in aggregate.
The time difference is encoded here by the duration of each input. Don't know if it's the best way to do it, I'm preparing a few separate runs with different approaches, like frame by frame.
But it's all trial and error, I'm learning as I go.
5
2
u/Deer-Antlers Sep 02 '25
Brasileiro que trabalha com dados aqui :)
Qual modelo você usou pra prever o resultado? Muito bacana o projeto, mas a não ser que você não queira prever outros resultados além de você jogando com Ryu provavelmente há overfitting. Seria muito legal combinar isso com entradas de um player 2 com outros personagens e ver como o modelo performa!
Infelizmente jogo no ps5 então acho complicado poder te ajudar a treinar o modelo lendo os meus inputs, mas se quiser treinar juntos pode me mandar uma DM hehe quem sabe nao contribuo com o projeto também? :)
1
u/Trevor_Z Sep 02 '25
Se vc está perguntando arquitetura, é uma GRU.
Sim, tem muito overfitting, mas como esse tipo de coisa não tem referência nenhuma, eu estou fazendo testes e vendo no que dá. Prever resultados é muito interessante mas quase secundário, meu objetivo maior é avaliar e discernir características dos jogadores.
Tenho um monte de dados que ainda não usei, cada partida minha tem tb os inputs do adversário, vou treinar com isso antes de tentar modelar outro jogador individual, mas valeu!
2
u/Spirited-Year9656 Sep 02 '25
Boa noite, outro brasileiro aqui, na reta final da minha graduação em neurociência! Sempre mantive guardadas algumas ideias para trabalhar com dados naturalísticos de jogos, e você me forneceu um insight bem poderoso com esse post!
A abordagem de modelar a sequência de ações é realmente muito promissora. Considerando a questão do overfitting que você mencionou e a ideia de expandir o modelo, já pensou em enriquecer seus dados de entrada com o registro das estatísticas que o próprio jogo fornece no perfil de cada jogador? (Seria interessante saber como o próprio jogo já armazena essas informações ao longo das partidas pra não precisar "codificar um contador próprio do zero")
Métricas como a frequência de drive rushes, a precisão dos perfect parries, o uso estratégico das barras de super e a execução de drive reversals poderiam adicionar uma camada de contexto fundamental para o seu modelo.
Do ponto de vista neurocientífico, isso seria análogo a fornecer ao cérebro não apenas a sequência motora (os inputs), mas também o "estado interno" e as "tendências comportamentais" do jogador, que refletem processos de tomada de decisão, gestão de recursos e adaptação em tempo real.
Ao incorporar essas estatísticas, sua GRU não estaria apenas aprendendo a sequência de "o que" você faz, mas também "como" e "em que contexto" você o faz, assim, o algoritmo poderia começar a identificar padrões mais complexos.
Por exemplo, ele poderia aprender que uma sequência específica de ataques tem uma probabilidade de sucesso muito maior quando iniciada após um perfect parry bem-sucedido, ou que o uso agressivo da barra de super em certos momentos do round é um forte indicador de vitória, ou um bom jogo em si.
Essencialmente, você estaria permitindo que o modelo aprendesse a "ler o jogo" de uma forma mais holística, espelhando a intuição que um jogador experiente desenvolve ao ponderar não apenas a próxima ação, mas todo o fluxo e a dinâmica da partida.
1
u/Trevor_Z Sep 03 '25
Eu tenho uma pesquisa diferente em mente com as métricas de perfil.
O problema de usar elas aqui é que não dá pra codificar pro treinamento do mesmo jeito que os inputs, porque são justamente dados fora de contexto, agregados. Não sei quando aqueles perfect parries ou aqueles drive rushes aconteceram.
Pensei em extrair tb o estado da barra de energia além dos inputs, mas isso ia dar pistas demais pro modelo de quando o round termina, e do jeito que está sendo treinado agora ele ia acabar ignorando a maior parte do round e focando só nos últimos inputs.
Na real, dá pra modelar tudo no jogo, personagem por personagem, golpe por golpe, quando acerta, quando é defendido... mas fico cansado só de pensar no trabalho que daria.
2
u/Spirited-Year9656 Sep 03 '25
Eu entendo, minha principal dificuldade em colocar um projeto dessa magnitude na prática, de fato é a baixa familiaridade que tenho no nível das hard skills de programação propriamente dita caso fosse extrair informações diretamente do jogo, por exemplo.
Citei a questão das métricas pra avaliar o quão bem um modelo aprenderia com essa "categorização de inputs", imaginando que seria possível utilizar a própria lógica do jogo no cálculo desses parâmetros ao invés de "reinventar a roda" pensando em utilizar esses dados contextuais.
Por exemplo, ver quão bom classificador o algoritmo seria utilizando essas informações adicionais, pra diferenciar um drive rush bem sucedido de um DRC, um target combo comum versus as extensões, identificar se um combo veio de corner, mid screen... São muitas possibilidades...
Ainda assim, parabéns pelo trabalho até então e espero que continue postando mais sobre, caso deseje continuar refinando a proposta!
2
u/airbear13 Sep 02 '25
This is super cool and interesting. I know you’ve only trained the model on yourself so far and there’s a lot of caveats, but I really think your prediction that there’s an underlying playstyle behind these raw inputs that could predict chances of winning makes a lot of sense and could be correct. Are you gonna be publishing this if it works out or is it just for fun?
2
u/gardotd426 ebolarama Sep 03 '25
Are you not gonna open source the script for real?
Sorry I'm a Linux user (only OS I use anymore) and so seeing a post like this but there not being a gitlab or github repo link in the post gave me whiplash lol.
If you dont wanna open source it on github or anything id still love to see it to see if I could get it working on Linux, I believe i could get it to work even if it doesnt just work as-is through Proton translation.
1
u/Trevor_Z Sep 03 '25
Dude, I'm a psychologist who dabbles in programming, didn't even think about creating a repo. :)
Right now this is a side project with a dozen scripts, half of them vibe coded, where I just try stuff.
2
1
u/Jay_Playz2019 'Echo' 1400 Sep 02 '25
This is really cool! How does one even start doing this stuff???
2
u/Trevor_Z Sep 02 '25
I was already doing something similar for academic research with another game, but honestly the beginning is just getting curious about research methods. Start by learning basic statistics and Python/R, then dive into machine learning tutorials. The gaming angle makes it way more engaging than dry textbook examples.
1
1
1
1
u/DudeWhereAreWe1996 Sep 02 '25
Cool. I want to see how it works with a comeback round. Do you think it is currently comparing time on offense vs defense? How long does it take to get back on track if you mash buttons during a super or something? Anyway this is really interesting. I would’ve liked my AI class more if it was modeled around SF6.
1
u/SleepyBoy- Sep 02 '25
The model is detecting combo strings, because they indicate big damage, which precedes a favorable end of the match.
1
u/j_one_k Sep 02 '25
The model seems very overconfident. I'd expect it to start with probabilities closer to 50% and only reach probabilities close to 0%/100% later in the round. The situation in this video where it hits 100% confidence and then revises that down should be very rare, I'd expect.
I wonder if you've got something wonky going on with the way you're splitting up the probability prediction and the level of confidence in the picture on imgur.
1
1
u/mangobae Sep 03 '25
Super cool project! How did you extract the inputs from the replays? Reading and classifying a part of the screen while watching it back?
1
u/Trevor_Z Sep 03 '25
I automated data extraction with a mix of ocr, pattern matching, and classification models.
However... I made the mistake of recording my replays with too much compression, and the videos were full of artifacts and even skipped frames when there was a lot of movement onscreen: https://imgur.com/a/AstXCC6
So this project spent months in the back burner while I was figuring out some error-correction algorithm and obsessively tuning the classification models to rebuild the full sequence of inputs from this defective source. 99% of the work went into fixing the dataset.
1
u/mangobae Sep 04 '25
Very nice, but somewhat sad screen recording is literally the only way. Would be so nice if publishers would offer APIs more often for these things
1
u/Zealousideal-Life467 Sep 03 '25
How to do this from scratch for my data. i have some basic knowledge about NN.
1
u/Trevor_Z Sep 03 '25
The hardest part isn't the training if you have the knowledge, it's building the dataset.
I started recording replays of most of my matches a year ago (really, the first match is dated September 7, 2024), even before I knew exactly how I would do things, because every 3 months or so you lose access to all of your previous replays.
1
u/Zealousideal-Life467 Sep 04 '25
do you have any tool for auto records all your replays. My PC is free when i go to work
1
1
1
0
u/TheShmoodus Sep 02 '25
I wonder if it can tell the difference between a counter DI, and an inputs that's too late to counter, and if you could reverse engineer that to find the frame data of it.
1
u/Trevor_Z Sep 02 '25
I think it can, because the probability of winning increased dramatically during the combo right after the counter DI: https://imgur.com/a/CM7Eby8
Thing is, a well-trained model will interpret not only a move, but the context, what comes before and after it.
With a very precise model, it could be possible to reverse-engineer and reconstruct the sequence of inputs, but I don't think that's the case here.
114
u/TomSelleckIsBack Sep 02 '25
If I had to guess how it's working based on your video with the graph, I would say:
The model has some sense of who is on offense and who is on defense based on the pattern of their inputs. The player who appears to be on offense longer and/or on offense at the end of the round has a higher probability to be the winner.