Eigen RadarAI
Analysis

Plain-word output reduces silent order errors in a small-model study

Researchers compared six ways of writing the same pizza order with two small language models. Ordinary words reduced errors that return a valid-looking but incorrect order, while shorter output alone brought no consistent accuracy gain. The preprint also measures how a strict refusal rule rejects some genuine orders, including requests containing polite expressions.

Artificial Intelligence··Evening
A researcher at a tall table in a university language-technology lab checks a laptop beside a closed pizza box with an order slip taped to its lid, seen from a low angle in daylight.

An incorrect order that still parses

A new preprint studies pizza-order mistakes that survive a parser’s format checks. Researchers trained two small language models to express identical orders in six lossless formats, including bracketed structures, compact symbols and ordinary words. The PIZZA benchmark supplied a human-written test set of 1,357 orders. The experiment covered five training-pool sizes and five seeds for each model and format, giving 300 runs. Each format represented the same underlying order, allowing its surface wording to be compared.[1]

Word form cuts silent mistakes

On Qwen, the plain-word format returned valid-looking wrong orders in 4.5 percent of tests, against 10.3 percent for the native expression format. On the second model, LFM2, the corresponding rates were 10.0 percent and 17.9 percent. The word format also generated substantially fewer tokens on Qwen. Comparisons between nearly equal-length symbolic and word-based formats favoured the words, while shortening the output by itself gave no consistent accuracy benefit. Negation accounted for much of the difference.[1]

Refusals also catch genuine requests

Every training run included off-menu requests and spoken self-corrections. Models learned to reject most held-out off-menu examples and to apply many corrections, such as a customer changing an ingredient. The strict rule also rejected genuine orders: rates across the tested formats ranged from 2.1 percent to 10.1 percent. Many contained greetings, thanks or other wording absent from the synthetic training grammar. All main results come from one ordering benchmark and two models with fewer than a billion parameters.[1]

References

  1. News sourcearXivOutput format changes silent order errors in small-model study↩1↩2↩3