Reasoning Models Show Generalization on Rare Text Encodings
The shift from memorization-by-prevalence to genuine generalization changes how practitioners should reason about edge-case reliability in deployed models. Builders evaluating models on adversarial or low-frequency inputs can expect reasoning models to outperform earlier LLMs where no direct alternative comparison exists.
Riley Goodside shared a case study showing that current reasoning models can read text typed in unusual character encodings despite scarce training data of that form. He contrasts this with earlier LLMs, which could handle ROT13 but failed on ROT12 because task prevalence in training data determined success. The observation suggests training-data frequency now matters much less for these tasks.
Good case study of generalization in reasoning models; consider how little training data there is of text typed this way. In earlier LLMs that prevalence would have decided whether a model could do this—e.g. they could read ROT13 but not ROT12. That’s much less what matters now.
