ASR hallucination can put words into a transcript that the speaker never said. A new paper, Grounded in Sound, tests whether acoustic feedback during training can reduce those insertion errors. Its results on meeting audio make a useful case for measuring insertions separately from overall word error rate.
What changed in the ASR hallucination results?
The authors report relative insertion reductions of 28.3% on close-talking AMI-IHM meeting speech and 22.3% on far-field AMI-SDM. On AMI-SDM, word error rate fell from 35.89% to 34.71%. The other five evaluation tiers showed no statistically detectable WER difference against the schedule-matched WER-only baseline.
Those percentages describe different measurements. A 22.3% reduction in insertions is not a 22.3% reduction in all transcription errors. The far-field WER improvement was 1.18 percentage points. A product evaluation that records only total WER would miss much of the change in error composition.
The scope is specific. The experiments use a Qwen2-Audio-7B policy adapted with LoRA and a frozen 0.3B wav2vec2-CTC judge. The authors describe a mechanism demonstrated in one model setup. The results do not establish equivalent gains for other models, languages or production call conditions.
How does the acoustic judge work?
During training, the policy samples groups of eight candidate transcripts. A frozen, separately pretrained CTC model scores them against the audio. The training reward combines that acoustic score with a term based on word error rate. The judge's frame-level outputs are reused across the candidates for each utterance.
The method section places the additional judge in training. At inference, the policy performs one greedy decode without that external scorer. The paper does not report an end-to-end latency comparison with production models.
The reward also has limitations. Its normalization depends on candidate length, and the judge becomes less reliable under severe acoustic degradation. The paper reports added deletions alongside fewer insertions. For an application, both error types deserve attention: omitting a requested item and inventing one can have different consequences.
Can inference-time rescoring reproduce the result?
The authors tested a 32-best CTC rescoring configuration on the baseline policy. That configuration did not recover the insertion gains reported for the policy trained with acoustic feedback. This is evidence about the evaluated configuration; it does not rule out every possible rescoring method.
The authors update model weights through RL post-training. The paper does not evaluate configuration changes to hosted transcription APIs. Its judge-free confidence experiments offer a separate direction to investigate. For a production evaluation, we recommend testing confidence-based abstention on representative audio and reporting the coverage tradeoff.
What should a voice-agent team measure?
A practical evaluation should retain the audio and reference transcript, then report insertions, deletions and substitutions alongside WER. Group the results by recording condition. Include the kinds of overlap, background noise and microphone distance the application actually encounters.
We would also review examples where a transcript changes an action. An invented quantity, omitted negation or altered name may matter more to a workflow than several harmless wording differences. This is an evaluation recommendation, not a result measured by this paper.
Turn timing belongs in that evaluation as a separate concern. Ink 2 exposes turn lifecycle events including turn.start, turn.eager_end and turn.end. Those events let an application coordinate listening and responding. They do not establish that a transcript is factually correct, and this paper does not evaluate Ink 2 or show that endpointing settings reproduce its acoustic-reward results.
What are the limits of the evidence?
The study's evaluation setup and limitations are worth reading before adopting the method. It trains on LibriSpeech with additive noise augmentation and evaluates six conditions, including AMI meeting speech. The 33,282 utterance-condition instances are not all independent recordings: the LibriSpeech utterances recur under different noise conditions, and both AMI conditions come from the same set of meetings.
AMI was excluded from RL training, but it did inform a diagnostic decision about which acoustic conditions to include in training. Describing it as having played no role before final evaluation would overstate the separation.
The useful result is a reduction in insertion errors under a defined training and evaluation setup. For teams evaluating speech recognition, the immediate lesson is to inspect which errors changed. A lower aggregate WER alone cannot tell you whether the model became less likely to invent words.