A new preprint presents semCER, a measure that distinguishes a harmless transcription slip from an error that changes meaning. A language model judges the severity of individual character changes. Readers assessed mistakes in three languages: English, Vietnamese and Chinese. A separate comparison covering 16 languages finds different consequences for altered accents and tone marks.
Researchers Nguyen Nguyen and Yongyi Zang introduce semCER in a new preprint on optical character recognition, which converts an image of writing into text a computer can process. The conventional character error rate counts edits without distinguishing a small typo from a changed name. semCER keeps the existing alignment and denominator, but a language-model judge assigns a severity weight to each edit.[1]
Five severity levels guide the judging model. Its final weight incorporates the likelihood assigned to every level, avoiding reliance on the top-rated level alone. The researchers asked people reading English, Vietnamese and Chinese to assess individual errors. On transcription pairs that the measures ranked differently, readers of English and Chinese preferred semCER more frequently. Among Vietnamese readers, semCER and the normalized measure attracted equal support.[1]
In the semCER analysis across 16 languages, accent and tone changes received different severity weights. A changed mark in Spanish carried little weight, while in Telugu its weight could approach that of an incorrect letter. The released materials contain 61 million edits assessed by the judge, reader ratings combined across respondents and a smaller scoring model. For scoring an entire transcription, the authors still find the normalized metric a strong baseline. The new score helps sort individual mistakes by severity.[1]
