How does the same Wordle model score 37 per cent and 84 per cent?

One adapter under changed conditions

The Wordle experiment AWS published on 23 October gives the same trained model two different solve rates. Its final validation measurement during training is 37 per cent. At the settings chosen for serving, the reported rate is 84 per cent. Wordle asks a player to find a hidden word using letter clues. The interesting question begins with the conditions behind those measurements. Understanding what the model can do requires knowing how each result was obtained.^[1]

Training validation fixes temperature, which controls randomness in response selection, at 0 and uses low reasoning effort. The 84 per cent serving evaluation instead uses temperature 1.0 and medium effort. The same adapter, a small set of weights added through training, is therefore tested under different generation conditions. Reading that difference as a gain from additional training would be mistaken. Temperature and reasoning effort change together, so this comparison also leaves their individual contributions unresolved.^[1]

The value of a matched comparison

The more useful comparison for training’s contribution lies elsewhere: AWS reports 61 per cent for the base model and 84 per cent for the trained model at medium effort and temperature 1.0. Generation settings match in that pair. My inference is that this matched comparison carries the claim of improvement. Putting the trained model’s 37 per cent beside the base model’s 61 per cent also changes generation settings. That ranking could give a misleading impression of training’s effect.^[1]

There is a strong reason to validate at temperature 0. The authors explain that deterministic selection makes training curves more comparable. Dismissing 37 per cent as a measurement error would therefore also be inappropriate. It describes performance under a particular way of using the model. Serving with sampling enabled answers a different question. A useful evaluation identifies the conditions offered to users and tests the base model under those conditions, rather than simply choosing the higher number.^[1]

A test of 128 words

The example’s scope is narrow. The authors use 640 training words and 128 validation words with no overlap. The game environment and reward function receive the secret answer; the model does not. That arrangement meaningfully tests solving words outside the training list. Success elsewhere remains a separate question. Moreover, 84 per cent averages two runs scoring 80 per cent and 88 per cent. A single headline number can hide that variation between runs.^[1]

These are the company’s own Wordle results, with independent replication and general model superiority unestablished. The useful methodological contribution is making the base and trained models comparable across settings. A stronger test would publish repeated results for both versions under settings fixed in advance, together with performance on a new word set. That design could distinguish how far the learned improvement carries to new words from the advantage supplied by generation settings.^[1]