Plain-word output reduces silent order errors in a small-model study
Researchers compared six ways of writing the same pizza order with two small language models. Ordinary words reduced errors that return a valid-looking but incorrect order, while shorter output alone brought no consistent accuracy gain. The preprint also measures how a strict refusal rule rejects some genuine orders, including requests containing polite expressions.
Artificial Intelligence··Evening
An incorrect order that still parses
A new preprint studies pizza-order mistakes that survive a parser’s format checks. Researchers trained two small language models to express identical orders in six lossless formats, including bracketed structures, compact symbols and ordinary words. The PIZZA benchmark supplied a human-written test set of 1,357 orders. The experiment covered five training-pool sizes and five seeds for each model and format, giving 300 runs. Each format represented the same underlying order, allowing its surface wording to be compared.[1]
Word form cuts silent mistakes
On Qwen, the plain-word format returned valid-looking wrong orders in 4.5 percent of tests, against 10.3 percent for the native expression format. On the second model, LFM2, the corresponding rates were 10.0 percent and 17.9 percent. The word format also generated substantially fewer tokens on Qwen. Comparisons between nearly equal-length symbolic and word-based formats favoured the words, while shortening the output by itself gave no consistent accuracy benefit. Negation accounted for much of the difference.[1]
Refusals also catch genuine requests
Every training run included off-menu requests and spoken self-corrections. Models learned to reject most held-out off-menu examples and to apply many corrections, such as a customer changing an ingredient. The strict rule also rejected genuine orders: rates across the tested formats ranged from 2.1 percent to 10.1 percent. Many contained greetings, thanks or other wording absent from the synthetic training grammar. All main results come from one ordering benchmark and two models with fewer than a billion parameters.[1]