So I was translating samples from Dolci-Think-SFT-7B and I thought it'd be an easy task, just deploy Gemma on vllm and write a quick translation prompt, specify the source and target languages and that's it but I ended up going through a whole rabbit whole and by the time got out of it, I was pretty unsatisfied and needed to rant about it so I'm sorry haha I'm still trying to make it educational so hopefully you'll learn something.
I think the most important lesson here, which probably many of you know, is that if the payload contains instructions, the model might execute the instructions of the payload instead of your prompt. Which is pretty obvious, but I didn't think about it when it came to a task as "boring" as translation. So the thing is, the model actually executed the problems that the reasoning traces talked about instead of translating them, and like actually produced solutions to programming problems and whatnot, proofs of math problems etc.
Actually that's it, there isn't much to say beyond this, if you're doing translation, be wary of that, obviously you should always be wary of your model's outputs, and you should have proper and rigorous input tagging so you know what you're feeding your model and you know on what to trust it and whatnot, so like if you trust your model to translate "normal" texts, you'd probably not trust it on a new distribution (e.g., reasoning) that you've never tried or evaluated before.
In case you want to go down the rabbit hole with me, what I'll be saying here is specific to these two models that I tried: RedHatAI/gemma-3-27b-it-FP8-dynamic, RedHatAI/gemma-4-31B-it-FP8-Dynamic, I'm not sure whether the bf16 models suffer from this or not, I'd say yes but you never know without trying. And I went with Gemma models because I've heard they were the strongest multilingual models.
I think the useful way to describe the failure is that the boundary between instruction and payload failed. And I'd bet this is more general than translation but also rewriting, proofreading, and summarization and everything that puts an instruction-following model in an unusual position like where the outer prompt requests a transformation, while the text treated as data may contain its own instructions. This makes me think of some kind of indirect or non-malicious prompt injection.
Anyways, to speedrun the rabbit hole, the first annoying thing was that everything looked completely fine from the outside, like the requests finished with `stop` and the output files had the right number of rows and nothing was empty, the throughput numbers looked good. Which makes sense because the inference server can't tell the difference between a right and wrong output.
But I looked at the samples and then I noticed that the broken outputs were often much shorter than their sources so I started using an output-to-source length ratio as a very cheap alarm. It's obviously not a quality metric at all but if you give the model a huge reasoning trace and it returns something tiny then it's probably worth opening the file. So I used that to collect 30 of the worst failures into a small test set and tried a bunch of methods on it and initially made the very tempting mistake of feeling good when something fixed all 30. But as you might expected, a dataset made entirely of known failures only tells you whether your method can recover known failures and it doesn't tell you how often the failure happens or whether the method is good on "normal" examples.
But the main thing that worked and I was hopeful about was chunking, also protecting code fences, mathematics, and other structures. Intuitively (and this might be wrong) splitting the prose into smaller requests worked probably because the translation instruction remained more locally relevant instead of being buried next to thousands of tokens that looked like a problem the model should solve. But then chunking created a whole new list of problems around broken code fences, `<think>` tags, reconstruction, inconsistent terminology, context between chunks, etc. and I ended up with the annoying task of writing a parser that is correct and that separates the document into typed blocks, so it lets Python preserve the parts that shouldn't change (code, math etc.) so that we only send the prose to translate, and then we reconstruct everything by putting things back into place.
Then I ran a larger experiment on a separate 340 row sample, across six languages, six chunk lengths and both models. And I found out that both models behaved pretty cleanly with the smaller chunks, and then the task execution and runaway generation rates suddenly became much worse at the larger labels, which was expected but well, it needed to be properly studied. Gemma 4 moved the cliff further away, which is good, but it didn't make the failure disappear. I really had hoped it'd solve this problem.
And then evaluation became its own rabbit hole because of course it did. I used COMET-QE, but the checkpoint has a limited input length, which meant I had to split long source and translation pairs again just to score them. But if the source and translation don't split at exactly the same places, you need some kind of alignment method, and at some point I had to fall back to splitting both sides independently and pairing the pieces by position. That gave me a score, but the correspondence behind that score is weaker, especially as chunks grow in length.
And, why stop here when you can suffer more, like even averaging those scores wasn't straightforward. Like, if every evaluation piece gets one vote, a passage can become more important just because the packer happened to split it into four pieces. So I weighted units by their combined source and candidate length. That fixes the arbitrary vote problem but it doesn't fix alignment uncertainty. If a long positional pair is badly aligned then length weighting also gives that alignment error more influence. So the quality curve at large chunks was also a curve built from increasingly weaker correspondence evidence, which is not exactly the clean conclusion I wanted haha.
I even went to the extent to use a blinded model judge (GPT 5.5 on medium) to compare candidates and apply a more detailed error rubric but I still don't want to pretend that this replaces native speaker review across six languages, I wanted to go beyond and properly calibrate this and estimate confidence on the judge, like I tried to find the most interesting "small" subset of samples that I can give to human native speakers to judge but then I thought that this is hellish and not worth it and should just stop at some point.
So yeah, I went into this thinking translation is solved and that it'd be an easy task, but well. I guess we learn the most from what we ignore the most.
If you want to read all of this in a clean way and more detailed with figures and everything you can read about it here: https://reinforcedknowledge.com/posts/when-translation-starts-solving/
I don't know why I wrote about all of that honestly but if you go take a peek, what I think is genuinely valuable is the last section because it's more about what to pay attention to in general (beyond this very specific task of translating a dataset) when doing ML work because that directly impacts your ability to improve, without honest evidence you don't know which direction you're going and whether you're actually improving or not. And it's also inspired by mistakes I've noticed people make. I also talk about how to improve this beyond what I've used here, and I think it's similar to LLM development, the biggest hurdle is good evaluation and evaluation that you can rely on, this was what took most of my time and obviously I haven't solved the problem and wish for someone to solve it haha I'm just giving my ideas at the end of what we need so that we can move the needle a little bit forward when it comes to translation.
