When the rest of the recommendation list counts

The list after the first correct skill

A model extracting skills from a vacancy can place a correct skill near the top and still fill the remaining recommendations with unrelated material. An evaluation focused on a handful of human positives sees these behaviors unevenly. WorkRB’s new graded protocol examines that gap: it assigns relevance from 0 to 4 across five datasets and scores the first 100 ranked results. Finding a known positive and keeping the rest of the list useful become distinguishable dimensions of performance.^[1]

In the authors’ comparison, the binary winner places 16th under graded scoring, while the graded winner places 12th under binary scoring. That reversal makes it harder to treat success under the original metric as a complete account of the recommendation task. Choosing a model depends partly on which portion of its list matters. Recovering the first correct skill and reducing unrelated suggestions are two different demands that can be placed on the same system.^[1]

Locating the source of the gap

One explanation is that stronger systems recover more relevant skills. Another is that, after finding similar positives, they suppress unrelated recommendations more effectively. WorkRB’s error composition gives the second explanation concrete support: the graded winner includes an average of 9 irrelevant skills in its top 100, compared with 34–37 for trained baselines. That exposes a filtering task that positive-only ranking can miss. The result still leaves room for better positive retrieval to contribute to the gap.^[1]

Who decides that a skill is irrelevant is part of the result. Language models generate the grades through sequential checks of domain, activity, granularity and replacement of an existing gold skill. Blind human comparison gives weighted kappa of 0.69, with 59 percent exact agreement and 90 percent within one grade. Those findings support the grading procedure while leaving the documented generosity on positive grades in view. A judgment that a system cleans its ranking therefore also inherits the error structure of its labels.^[1]

Where the human anchor belongs

Candidate coverage draws another boundary. BGE-M3 first selects 2000 skills, covering 96.6 percent of the human gold positives. Other relevant skills can fall outside the annotated window. Shared relevance biases between judging models and successful rankers may also let a similar preference shape both the recommendation and its score. The circularity acknowledged by the authors complicates any direct identification of a high score with genuine skill understanding.^[1]

A useful next experiment is therefore specific: establish a wholly human-annotated anchor that the judging models have never seen. It can examine both irrelevant suggestions near the top and relevant skills missed by candidate selection. WorkRB currently covers English-language tasks and does not measure hiring fairness. Graded scoring describes the recommendation problem in finer detail. Testing that description against human judgments independent of the label-producing models gives a clearer basis for interpreting its gains.^[1]