Words, people and time
A meeting transcript can contain the right words while assigning a decision to the wrong person. IBM’s Granite Speech update makes that distinction visible within one model family. The new 5.0 model emphasizes transcription and translation, while the revised 4.1 Plus is evaluated separately on words, speakers and timestamps. My reading of those tables is that speech recognition should be judged in the unit of the task it serves. A searchable transcript, a speaker-attributed meeting transcript and a subtitle aligned to audio each ask a different accuracy question.[1]
IBM reports that long-form word error rate for 4.1 Plus falls from 13.53 to 10.22. Word diarization error rate changes from 4.40 to 4.10, while concatenated minimum-permutation word error rate moves from 30.50 to 28.48. These are the company’s own results. Each measure has a previous value against which it can be compared; ranking the magnitudes of different measures would discard their scoring definitions. There is no reason here to expect improvements in recognized words to correct speaker assignments at the same rate. Content recognition and speaker attribution can have different sources of error.[1]
Timestamps add a third axis. IBM says it updated 4.1 Plus weights while preserving the architecture, prompt format and API, a concrete convenience for software migration. The same interface does not establish that every type of output error behaves as before. A researcher locating a particular utterance needs its position in the audio; one studying who said it also needs the speaker assignment. Evaluating those fields separately can expose a problem hidden by an aggregate word error rate. The relevant endpoint follows the use of the transcript, even when the model can be swapped without changing application code.[1]
Vocabulary and language conditions
Custom vocabulary exposes another distinction. The new 5.0 model accepts lists of up to 2,000 terms. For revised 4.1 Plus, IBM reports keyword-biased word error rate changing from 11.79 to 6.48. Recognition of a research field’s terminology can therefore be tested separately from ordinary conversation. How the list is selected becomes important: names a real user knows beforehand and a list assembled after inspecting test outcomes represent different conditions. This is a question about experimental design, rather than a finding of leakage in IBM’s evaluation. It determines which intended use the result can reasonably represent.[1]
Language coverage deserves the same precision. The new 2 billion parameter model transcribes six languages; the smaller Turbo CTC supports seven. Those lists differ. Turbo CTC transcribes Italian, whereas the new larger model offers Italian as an English-to-Italian translation output. Mentioning a language in a product description does not establish identical support for every task in that language. I would use this distinction to define the evaluation set at the outset. Specifying the language, task and recording conditions together makes it possible to say what operation a reported score actually measures.[1]
What a replication should measure
A useful next test would run old and new weights on the same audio and publish word errors, speaker attribution and timestamps together. Vocabulary lists could be fixed before testing. That comparison would examine how far the gains in IBM’s table carry to other speakers and microphones. A smaller gain could reflect changed recording conditions; another explanation would be a metric that does not adequately represent errors in the new task. The threshold I want is visibility into which part of a meeting transcript improved, and by how much: separate answers for words, people and time.[1]