Two outcomes of adaptation
How much pretrained knowledge does a model retain when it adapts to a new task after reaching lower pretraining loss with Muon? A new Carnegie Mellon University preprint tests that question with OLMo-style models ranging from 30 million to 600 million parameters. Fine-tuning on GSM8K, StackMathQA, Alpaca and MusicPile is evaluated through loss on both the new task and held-out DCLM data. This makes the cost of adaptation visible: progress on a new task can accompany deterioration on the pretraining distribution.[1]
The experiment's most useful choice, in my view, is to compare attainable learning-retention tradeoffs rather than nominate one fine-tuned checkpoint as the winner. Lower learning rates can preserve prior knowledge while limiting adaptation; higher rates can incur the opposite cost. The researchers sweep those choices and compare the best attainable frontiers. That directly tests the assumption that the starting model selected for minimum pretraining loss is also the most useful starting point for subsequent adaptation.[1]
A possible mechanism for the reversal
The ranking changes with the data budget. Muon offers the more favorable tradeoff at small budgets, while AdamW overtakes it after sufficient pretraining data. In the StackMathQA experiment, the crossover token-to-parameter ratio is 320 for a model with 30 million parameters and 10 for a model with 600 million parameters. The point varies by task. The reversal also persists when settings are searched for the best adaptation frontier, beyond the minimum-pretraining-loss selection rule. That narrows an explanation based solely on how the starting checkpoint was selected.[1]
The authors' proposed explanation lies in the arrangement of answer scores. They divide the correct answer's distance above the mean incorrect score by the spread of incorrect scores. A larger ratio provides more room against small output perturbations. Its movement with the data budget parallels the change in adaptation tradeoffs, suggesting a connection between forgetting and the starting output geometry. Differences in pretraining loss and the adaptation task can also affect the frontier. The imperfect alignment of the crossover points makes a single-factor explanation harder to sustain.[1]
Scope of evidence and the decision endpoint
A substitution matters at the theoretical step: in the simplified linear-memory model, Muon is compared with gradient descent rather than AdamW. A relation proved for one update and an empirical trend in language models have different scopes. The simple model helps investigate why the robustness comparison can change with the data budget. It does not prove the full adaptation behavior of a large language model trained with AdamW. That boundary identifies which step is established analytically and which connection is supported by experiments.[1]
My practical reading of this preprint is that optimizer selection should be tied to the evaluation endpoint. For a model intended for later adaptation, the starting loss belongs alongside post-adaptation learning and retention curves. Experiments extending to 600 million parameters do not provide a ready-made selection rule for every larger model. They do supply a testable comparison of adaptation settings at a specified model size and data budget. The decision depends on what knowledge survives modification as well as where the initial training loss reaches its minimum.[1]