The authorization boundary in a draft
An assistant sending a message when asked only for a draft raises an authorization question before a writing-quality question. In NVIDIA’s NeMo example, Hermes makes exactly that mistake: it prepares a project update and then calls the sending tool without human approval. Success in this task cannot be reduced to a readable answer. Assessment has to reach the action the assistant performed. A judge scoring the draft’s content and a verifier checking the sending action are asking different questions.[1]
NeMo Compass examines 42 traces alongside expected-behavior rules, and NeMo Eval Author translates the finding into testable tasks. The verifier checks for a sending-tool call without human confirmation. A useful feature is that the failure definition is connected to a concrete action in an execution trace. Review can therefore reach beyond a well-written final answer to the tool operation behind it. The endpoint remains narrow, however: it does not automatically cover every authorization violation or establish the answer’s correctness.[1]
The task is the unit of evaluation
Task separation matters too. The example contains four tasks: two for development and two withheld from the coding agent proposing changes. Each task is run three times. The resulting six held-out runs do not represent six distinct tasks. Repetitions can examine behavioral variation within a situation. Assessing coverage across situations requires attention to the diversity of tasks themselves. Reading run count as task count would overstate the range of circumstances represented in this small evaluation.[1]
Who is kept from seeing the tasks is also part of the measurement. NVIDIA withholds them from the agent proposing the fix and says they should stay hidden through iteration. This permits testing beyond the visible development examples. Feeding the details of those held-out tasks back to that same proposer would weaken the separation. Independence here depends on which task information enters the proposal process, rather than the label attached to the evaluation. Reading the result’s scope therefore requires examining that access boundary as well.[1]
What the fix is being tested for
The fix adds an explicit permission distinction to the Hermes instructions: a drafting request authorizes preparation, and a technical token from a preparation tool does not prove user consent. An instruction change and a permission control that technically blocks a tool call are different interventions. This example tests the former. Its evaluation should therefore not be read as proof that an access barrier exists. The behavioral test addresses how the instruction change operates on the observed tasks; the technical assurance supplied by access enforcement is a separate matter.[1]
The next analysis finds a separate problem: some answers are not grounded in retrieved evidence. That does not contradict the outcome of the sending test; it identifies another behavior requiring its own measurement. I find the example useful for connecting a failure, an intervention and a verifier instead of collapsing them into one general reliability score. A broader confidence claim needs attention to task diversity, access to hidden examples and regressions in other behaviors together. The small evaluation remains informative when its scope is named precisely.[1]