What operation must the student perform?
A sentence names two people, one of whom gives an object to the other. The language model must complete the recipient’s name. In a new preprint, six-layer DistilGPT-2, distilled from GPT-2, favors the wrong person in this test. Yet the same student retains much of its teacher’s measured behavior when completing a year interval with an ending later than its beginning. I find that contrast more useful than a single overall grade. Compression can disrupt one chain of computation while relocating another. For a researcher choosing a smaller model, the practical question is which of those patterns resembles the operation their own application requires.[1]
The teacher has 12 layers and the student six, with the same embedding width. Researchers intervene on internal activations between clean and corrupted prompts to test how particular components contribute to answers. That is more informative than comparing whether the models produce similar-looking prose. The experimental object still deserves a narrow description: there is no random sample representing today’s compact models. There is one teacher–student pair with a particular training history and two previously studied behaviors. Depth, initialization and distillation training change together in that pair. My reading therefore avoids assigning every observed difference exclusively to the removal of layers.[1]
Different denominators behind two numbers
The mean correct-minus-incorrect name logit difference is minus 0.467. Falling below zero matters because the incorrect name receives the higher score in the tested prompts. In contrast, the reported 87.7 percent for the year task is the fraction retained of the teacher’s probability mass on greater year endings. I would not read it as answering 87.7 percent of questions correctly. The two tasks use different endpoints: a score difference between names and probability assigned to a set of endings. Before putting these numbers beside each other, a researcher needs to establish each denominator and the particular behavior it represents.[1]
Representation comparison also depends on alignment. Comparing the student’s sixth layer with the teacher’s sixth asks a different question from comparing the final layers of both networks. Relative-depth matching can yield higher similarity. I read this as a warning about where a computation takes place: a shorter network may compress some operations into a different sequence. Similar activation patterns alone still do not establish preserved function. That is why the paper’s behavioral interventions matter. Treating representational resemblance and causal contribution as interchangeable evidence would erase the distinction that makes this comparison useful.[1]
Measuring through the teacher’s lens
Another boundary lies in the instrument used to read features. A sparse autoencoder trained on GPT-2 is also applied to the student’s activations. The student is consequently measured in a coordinate system learned from its teacher. A weak feature signal could reflect genuine loss, but it could also reflect a poor fit to that basis. Multiple measurements help address the issue, yet I would hesitate to call every low similarity score a lost capability. Favoring the wrong name is a more direct behavioral result. Uncertainty about feature alignment should limit the breadth of the mechanistic interpretation without erasing that observed failure.[1]
My practical response to this preprint is to specify the critical operation before assigning a compact model a general competence label. If a research workflow depends on matching people correctly, fluent output and preserved performance elsewhere cannot replace testing that match. For a workflow centered on year comparisons, the favorable result here warrants replication with the relevant input patterns and failure cases. A broader conclusion would require students with different training histories and a wider range of behaviors. For now, the useful finding is that one student produces two distinct outcomes. The cost of compression becomes measurable when the operation that must survive is explicitly identified.[1]