DistilGPT-2 loses name selection while retaining much of a year-comparison skill
A single preprint comparing GPT-2 with its smaller student finds sharply different outcomes on two controlled tasks. DistilGPT-2 favors the wrong person’s name in one test but retains much of the teacher’s year-comparison measure. The study probes internal components as well as answers, and covers only this model pair and these behaviors.
Artificial Intelligence··Morning
The smaller model favors the wrong recipient
A single preprint compares GPT-2 with DistilGPT-2, a smaller language model trained from its outputs through knowledge distillation. The work submitted on 27 October tests two controlled behaviors and examines internal components. GPT-2 has 12 layers, while its student has six. In prompts involving two people and an object being given, DistilGPT-2 favors the incorrect name. The authors’ interventions on internal activations probe the components involved in this failure.[1]
Much of the year-comparison measure survives
The other task asks the model to complete a year interval with an ending later than its beginning. The probability DistilGPT-2 assigns to later year completions amounts to 87.7 percent of GPT-2’s corresponding measure. The teacher’s late-layer components have no direct structural counterparts in the student. The authors identify a shift toward earlier components, but describe verification of the complete causal chain as unfinished.[1]
Relative layer matching changes the similarity result
Researchers also examined representations, attention patterns and general language-modeling measurements. Matching relative positions through each network gives different similarity results from matching identical layer numbers. Features expressed differently by the student can escape a comparison built on a feature basis learned from GPT-2. These two behaviors were tested in one teacher–student pair, leaving other model families and compression methods outside the experiment.[1]
Related columns
For more information on this topic, you can read the related columns.
