Providing an existing transcript as input?

[samuelbradshaw]

If I already have a transcript (plain text, without timestamps), is it possible to provide that as an input, along with the audio, and get a timestamped transcript back?

[misutoneko]

Yes! The --prompt switch can be used to fix spelling.
There's a catch though: It only works with very short clips (224 tokens max?).
So some kind of pre/postprocessing is needed to get around that limitation.

Btw I've only tried this with whisper_timestamped, but it should work with whisper.cpp too.
And I only used it for some short clips that I got from VAD preprocess, not for the whole audio track.
VAD alone probably won't be enough if your aim is to go through the whole script.

EDIT: I've now done some more testing, actually I couldn't replicate the results from whisper_timestamped in whisper.cpp, it always left some spelling mistakes (yes I tried tweaking the parameters but it didn't help). The model size seems to have an effect, so you can't just give it a prompt and expect a tiny model to handle it, you need to use the bigger models.

[bobqianic]

It only works with very short clips (224 tokens max?).

Correct.

[bobqianic]

Is it possible to provide that as an input, along with the audio, and get a timestamped transcript back?

I don't think that's possible right now. The --prompt option currently just adds some tokens at the beginning of a transcription. And when a new transcript is generated, old tokens in the context window get replaced. (I'll need to double-check this though). Even if we created an option to use text from a file, like your transcript, consistently, it would still be hard to figure out which section of the text should be used as a prompt since we lack timestamps. A more feasible approach might be to utilize sentence embedding models to gauge the similarity between sentences, which would help us assign timestamps to the appropriate sentences.