Word boosting in AllSpeak Captions

Rare names are an awkward problem for speech recognition.

A speech recognizer, or Automatic Speech Recognition (ASR) model, listens to audio and produces a transcript based on patterns it learned from training data. That’s a problem for words it saw rarely or never: an unfamiliar name doesn’t sound like nothing to the model, it sounds like whatever more common word it acoustically resembles.

Consider an application that already knows a list of names it is likely to hear, for example, a classroom roster. When one of those names comes up in the audio, the recognizer will often substitute that more familiar, acoustically similar word instead: a student named “Kotkin” becomes “Cotkin,” a name it has seen far more often in training data.

Knowing the roster should help. But blindly pushing the recognizer toward roster names creates a different, and potentially worse, problem: we might insert a name the speaker never said.

What we wanted was something narrower. The roster can tell us what to look for, but the audio should still get the final say.

CTC-based word spotting (CTC-WS) gives us a natural way to do this. CTC, or Connectionist Temporal Classification, gives us frame-level distributions over tokens, which makes it useful for searching the acoustic evidence for known terms. For a more complete introduction to CTC, see Hannun (2017).

In Fast Context-Biasing for CTC and Transducer ASR Models with CTC-based Word Spotter, Andrusenko et al. (2024) propose searching these frame-level log-probabilities for a supplied set of context terms and returning candidates supported by the audio. For Transducer ASR, the authors use a Hybrid Transducer-CTC model in which the Transducer and CTC heads share an encoder. The CTC head provides the frame-level log-probabilities needed for word spotting, while the Transducer produces the transcript.

There’s just one problem: our recognizer does not have that architecture. It is a standalone TDT model, with no CTC head sharing its encoder. Rather than change or retrain it, we pair it with a much smaller, independently trained CTC model listening to the same audio.

The TDT model still owns the transcript. The CTC model is not there to produce a second transcript and vote on which one looks better. It has one much narrower job:

Is this known name actually here?

That sounds simple, but it introduces an inconvenient problem.

One audio, two clocks

Both models listen to the same audio, but they do not share an internal timeline. Each model has its own encoder and its own sequence of acoustic representations. A frame reported by the CTC word spotter therefore belongs to the CTC model’s clock. We cannot assume that the same frame index refers to the same representation, or even exists, inside the TDT model. Suppose that the TDT model produces:

and Jeff Cotkin from NVIDIA

Meanwhile, CTC-WS spots the known name Jeff Kotkin over frames 50–59 of the CTC model.

Great. Should we replace Cotkin with Kotkin? Not yet.

Frames 50–59 belong to the CTC model’s clock. They tell us where the CTC model found evidence for Jeff Kotkin, but not which words on the TDT model’s side those frames correspond to. Before changing the transcript, we want to ask a stricter question:

Over the audio corresponding to “Jeff Cotkin”, does the CTC model actually prefer “Jeff Kotkin” to “Jeff Cotkin”?

To answer that fairly, we need to put Jeff Cotkin and Jeff Kotkin on equal footing: same model, same frames. This leaves us with an alignment problem. How do we find a word produced by one model on the frame timeline of another? We don’t try to synchronize the clocks directly. Instead, we look for something the two models sometimes agree on: the words.

The figure above shows the overall pipeline. The two models begin independently: the TDT model produces the primary transcript, while the CTC model provides the frame-level acoustic evidence used for word spotting, alignment, and comparison. A greedy CTC transcript gives us something else we need: words that can be matched against the TDT output. The two paths first meet through those shared words. These anchors let us align the relevant TDT words onto the CTC clock, after which the original wording and the spotted candidate can be compared over the same acoustic frames.

Meeting in text

The models do not share frame indices, but they often share words. Suppose their outputs around our example look roughly like this:

TDT:	and Jeff Cotkin from NVIDIA
CTC:	and Jeff Kotkin from NVIDIA

They disagree on Cotkin versus Kotkin, but there is plenty they agree on: and, Jeff, from, and NVIDIA. Those shared words give us landmarks. We call them anchors.

An anchor does not tell us whether Cotkin or Kotkin is right. It tells us something simpler: both models agree that this word happened here in the utterance. Shared words on either side of a disagreement therefore give us a local region in which to investigate. The models still have different clocks. But now we know where to build the bridge between them.

From candidate to correction

Now we can follow Jeff Kotkin all the way from a roster entry to a possible transcript edit.

Finding a candidate

The useful thing about CTC for our purposes is that it gives us a distribution over tokens at every acoustic frame. We can picture those outputs as a matrix: time runs from left to right, while each row represents a token in the CTC vocabulary. Each cell contains the model’s log-probability for that token at that frame.

The table below shows a schematic example around Jeff Kotkin. For readability, we use character-like tokens; the real model operates over its own tokenizer vocabulary. Because these are log-probabilities, values closer to zero indicate greater probability.

Schematic CTC log-probabilities using character-like tokens for readability. Values are illustrative; the real model operates over its tokenizer vocabulary. Highlighted cells show the greedy choice at each frame.

A greedy CTC decode takes the highest-scoring token at each frame, shown by the highlighted cells. Consecutive repetitions are collapsed and blank tokens are removed. In this example, the frame-level path

J J e f <blank> f <blank> K o t k i n <blank>

collapses to Jeff Kotkin. Notice the blank between the two fs: without it, consecutive f predictions would collapse into a single token.

The important part for us is that greedy decoding selects only one path through the matrix. The other log-probabilities are still there. At frame 57, for example, K wins the greedy decision, but C still has acoustic support. Greedy decoding chooses one path through the matrix; it does not erase the alternatives.

Locating the TDT words

At this point, CTC-WS has given us a candidate and told us where it appears on the CTC model’s timeline. What it has not told us is which part of the TDT output that interval corresponds to. Frames 50–63 mean something to the CTC model, but we cannot use them as coordinates in the independently trained TDT model.

This is where the anchors from earlier become useful. We compare the TDT output with the CTC greedy transcript and look for shared words surrounding the candidate:

TDT:	and [Jeff Cotkin] from NVIDIA
CTC (greedy):	and [Jeff Kotkin] from NVIDIA

For this candidate, the nearest shared words outside the spotted region are and and from. Those become our local anchors, giving us a small region in which to locate the TDT words.

The CTC greedy transcript gives us the anchors, but the underlying log-probabilities give us more than that single decoded path. The CTC model does not need to have greedily produced Cotkin for us to ask whether the audio supports Cotkin.

We take the TDT model’s fixed text between the anchors (in this case Jeff Cotkin) and ask for its best legal CTC alignment through the same frame-level log-probabilities. This is a CTC forced alignment: instead of asking what text the model would greedily produce, we fix the text and ask where that text best fits the acoustic evidence.

The same schematic CTC log-probabilities, now forced-aligned to the fixed TDT text Jeff Cotkin. Highlighted cells show the highest-scoring valid CTC path constrained to that text.

Earlier, the first table showed the greedy path through these log-probabilities. The table above shows the same matrix again, this time forced-aligned to the TDT wording Jeff Cotkin. Unlike greedy decoding, forced alignment does not take the highest-scoring token independently at every frame. Instead, it finds the highest-scoring CTC path consistent with the fixed text. At frame 57, for example, the greedy decoder prefers K, while the best path consistent with Jeff Cotkin instead passes through the lower-scoring C.

The result is a set of intervals for the TDT words expressed on the CTC model’s timeline. We never need to convert a TDT frame number into a CTC frame number: the TDT model supplies the words, the CTC model supplies the frame-level acoustic evidence, and the anchors tell us where to align them.

Letting the audio decide

Forced alignment has now given us the missing piece: the CTC frame interval corresponding to the TDT wording. We can finally make the comparison we wanted from the beginning.

Over those same frames, we score two text hypotheses using the same CTC log-probabilities. The first is the wording produced by the TDT model; the second is the candidate proposed by CTC-WS. For each hypothesis, we find its best valid CTC path through the aligned interval and sum the frame-level log-probabilities along that path. The resulting acoustic score tells us how well that fixed text fits those frames. Because both are evaluated by the same CTC model over the same acoustic frames, the different clocks of the two recognizers no longer enter into the comparison. The rule is simple: same model, same frames, different text.

Hypothesis	Text	Acoustic score
Original TDT	Jeff Cotkin	−18.4
Candidate	Jeff Kotkin	−13.1 (stronger)

Illustrative acoustic comparison of the two hypotheses over the same aligned CTC frames.

Each score asks how well that fixed text can be explained by the CTC acoustic evidence over the aligned interval. The candidate has now done more than merely appear in the context list or trigger the word spotter: it has beaten the wording that the TDT model actually produced.

If that advantage exceeds the required acoustic margin, the candidate becomes an accepted correction. We replace only the aligned TDT word span:

Before:	and Jeff Cotkin from NVIDIA
After:	and Jeff Kotkin from NVIDIA

Everything outside that span remains exactly as the TDT model produced it.

To edit or not to edit

The example above follows the happy path. But a candidate does not always make it that far.

Missing a correction leaves the original TDT transcript unchanged. A false correction is more dangerous because it can insert a real roster name into a sentence where it was never spoken, and the result may look convincing precisely because the name is plausible. So if we cannot confidently locate and defend a candidate, we keep the TDT text. Before we allow a candidate to change the transcript, we ask a series of deliberately conservative questions. If the answer to any of them is no, the original TDT text wins by default.

Can we locate it?

A CTC-WS candidate arrives with an interval on the CTC model’s clock. We need nearby shared anchors to determine where that interval belongs in the TDT transcript. If the surrounding anchors are missing or too far away, or if the local forced alignment fails, we refuse the correction. A good candidate is not useful if we cannot reliably determine which TDT words it is allowed to replace.

Does it cover whole words?

Even a successful alignment does not necessarily give us permission to edit. The CTC-WS spot may overlap an aligned TDT word without covering that word completely. Consider:

TDT Output:	post-biopharma
Candidate:	Biopharma

The candidate may have strong acoustic support for Biopharma, but the spot covers only part of the aligned TDT word post-biopharma. Replacing the whole word would therefore change text outside the span supported by the candidate. Partial-word coverage means no correction.

Can we score both sides?

A fair comparison requires both hypotheses to be expressible in the CTC model’s vocabulary. If we cannot construct and score a valid CTC path for both of them, there is no fair acoustic comparison. Neither side gets a free win, so we keep the TDT transcript unchanged.

Does it beat the original?

A candidate still has to win the same-frame acoustic comparison. For example:

TDT Output:	financials
Candidate:	Financial

If financials receives the stronger acoustic score over the aligned CTC frames, the candidate is refused. Being spotted is not enough to overrule a better explanation of the audio.

Does it conflict with another edit?

Candidates can overlap, especially when the context list contains similar phrases. Once an accepted correction has claimed a TDT word span, another candidate cannot rewrite those same words. Rather than stack competing edits, we refuse corrections that conflict with an already accepted span. Each accepted change remains local and explainable.

Safety gates between a spotted candidate and an accepted correction. Failure at any gate leaves the original TDT wording unchanged.

The pattern is deliberately conservative. A candidate must pass every gate before it is allowed to change the transcript; failing any one of them leaves the original TDT output untouched. The system is allowed to miss a correction, but it should not invent one merely because a plausible context term was nearby. The context list tells us what to look for. The audio gets the final say.

Conclusion

The TDT model provides the transcript. The CTC model provides the acoustic evidence. The shared words provide the bridge.

References

• Andrei Andrusenko, Aleksandr Laptev, Vladimir Bataev, Vitaly Lavrukhin, and Boris Ginsburg. “Fast Context-Biasing for CTC and Transducer ASR Models with CTC-based Word Spotter.” Interspeech 2024, pp. 757–761. https://doi.org/10.21437/Interspeech.2024-1002
• Awni Hannun. “Sequence Modeling with CTC.” Distill, 2017. https://distill.pub/2017/ctc/