Everything I do here is just experiments. I'd be really happy to hear any friendly tips or advice you have.
In part 1 https://www.reddit.com/r/StableDiffusion/comments/1v1smgn/im_training_an_image_model_from_scratch_part_1_my/ I trained my own VAE. A VAE is nice but it doesn't actually make pictures, it just squashes and rebuilds them. So this time I sat down to train the real generator, the part that turns text into an image.
The setup: one machine, one RTX 5090. No cluster, no rented pods. One card. So the dataset stayed small (I started with about 37k image and caption pairs). I wasn't trying to ship anything yet. I just wanted to know one thing: can this even learn, and how does it fall apart.
It falls apart constantly. And almost never for reasons that have anything to do with AI.
Attempt one: the model that could only draw snow.
My first version could only "read" the caption as one blurry summary instead of actual words. It trained, the loss dropped for a bit, and then sat still forever. I let it run way too long out of stubbornness.
The results were amazing in the wrong way. "Snowy mountain" actually looked like a snowy mountain. Everything else melted. A portrait came out as a melting face. A puma in snow was grey soup. The model had basically decided that "vaguely textured blob" was the safe answer to everything and fully committed.
The bug that ate 92,000 steps.
This is my favorite one. I had a feature turned on that keeps a smoothed backup copy of the model. Because of one copy paste mistake, every time the trainer stopped to save a preview image, it overwrote the live model with the older backup and never switched back.
So every thousand steps, the model quietly threw away a thousand steps of progress and reset itself. I stared at the weird loss graph for days thinking it was some deep training problem. Nope. I was deleting my own work on a timer. Roughly 92,000 steps of training, gone, because of two lines of code.
Attempt two: a real architecture, and my own code fighting back.
I rebuilt it properly this time so the model actually reads the full caption word by word instead of one blurry summary. And since the small version was clearly learning, I decided to go bigger and feed it a much larger dataset. Turning all that new data into the format the trainer needs is where the fun started.
The model itself was fine. Everything around it was not.
First launch of the new run: instant crash on the very first batch, because my data loader tried to open all 71 of the new dataset files at once and choked. It worked fine back when there were only a few files. Nothing teaches you about scale like scale.
Building the bigger dataset ran out of memory halfway through, then left 47GB of half finished junk files on my drive as a goodbye present.
Printing a single checkmark character crashed an entire training run. Not the model, not the data, just one tiny symbol in a log line. I killed my own training with a checkmark.
My launch script refused to run for an entire evening because of one missing backslash in a path.
Did it actually work?
Yeah, and surprisingly fast. "Red dress" gave me a red dress. "White cat" gave me a correctly shaped white blob. "Red sports car" started as a literal jar (it heard "car," drew a jar, I have no notes) and later turned into an actual red car. Strawberries stayed the wrong color for an embarrassingly long time.
The weirdest part: the loss number barely moved this whole time while the images kept clearly getting better. Turns out for this kind of training the loss just isn't the thing that tells you quality. Watching a flat line for days while your eyes say it's improving is its own special kind of stress.
I stopped it on purpose, not because it broke, but because I'd figured out the next real upgrade needed a better VAE, which means starting the generator over from scratch anyway.
That's the next part. Short version so far: the model was never the hard part. My own code was.
Want part 3? Want to hear about more of my mistakes? Say so in the comments and I'll write up what happened when I tried to rebuild the VAE.
