Processing long recordings (~1h) seems not work correctly

[pcbua]

Hi,
I'm running Whisper CPP on several long recordings 20min - 2h with default settings, 15 threads (-p 1 -t 15).
I've tried multilanguage models: base, small, medium, large.

It seems to work at beginning, but after some time it tends to repeat lot of utterances or detected events.
These utterances are not repeated in origin recording.

Does Whisper CPP requires special parameters for long recordings?
I'm using WAV files, not streaming.

Examples (music & pause below is wrong, there is some speech):

[00:55:14.980 --> 00:55:24.980] I'm going to be able to help you.
[00:55:24.980 --> 00:55:32.980] I'm going to be able to help you.
[00:55:32.980 --> 00:55:50.980] I'm going to be able to help you.
[00:55:50.980 --> 00:56:08.980] I'm going to be able to help you.
[00:56:08.980 --> 00:56:26.980] I'm going to be able to help you.

[00:23:21.000 --> 00:23:24.000] and that the baby was not in the room,
[00:23:24.000 --> 00:23:27.000] and that the baby was not in the room,
[00:23:27.000 --> 00:23:30.000] and that the baby was not in the room,
[00:23:30.000 --> 00:23:33.000] and that the baby was not in the room,
[00:23:33.000 --> 00:23:36.000] and that the baby was not in the room,
[00:23:36.000 --> 00:23:39.000] and that the baby was not in the room,
[00:23:39.000 --> 00:23:42.000] and that the baby was not in the room,
[00:23:42.000 --> 00:23:45.000] and that the baby was not in the room,
[00:23:45.000 --> 00:23:48.000] and that the baby was not in the room,
[00:23:48.000 --> 00:23:51.000] and that the baby was not in the room,
[00:23:51.000 --> 00:23:54.000] and that the baby was not in the room,
[00:23:54.000 --> 00:23:57.000] and that the baby was not in the room,
[00:23:57.000 --> 00:24:00.000] and that the baby was not in the room,

[00:04:25.000 --> 00:04:40.000] [Music]
[00:04:40.000 --> 00:04:44.000] [Music]
[00:04:44.000 --> 00:05:11.000] [Music]
[00:05:11.000 --> 00:05:21.000] [Music]
[00:05:21.000 --> 00:05:31.000] [Music]
[00:05:31.000 --> 00:05:41.000] [Music]
[00:05:41.000 --> 00:05:51.000] [Music]
[00:05:51.000 --> 00:06:19.000] [Music]
[00:06:19.000 --> 00:06:29.000] [Music]
[00:06:29.000 --> 00:06:39.000] [Music]
[00:06:39.000 --> 00:06:49.000] [Music]
[00:06:49.000 --> 00:06:59.000] [Music]
[00:06:59.000 --> 00:07:09.000] [Music]
[00:07:09.000 --> 00:07:19.000] [Music]
[00:07:19.000 --> 00:07:29.000] [Music]

[00:09:22.000 --> 00:09:24.000] (Pause.)
[00:09:24.000 --> 00:09:26.000] (Pause.)
[00:09:26.000 --> 00:09:28.000] (Pause.)
[00:09:28.000 --> 00:09:30.000] (Pause.)
[00:09:30.000 --> 00:09:32.000] (Pause.)
[00:09:32.000 --> 00:09:34.000] (Pause.)
[00:09:34.000 --> 00:09:36.000] (Pause.)
[00:09:36.000 --> 00:09:38.000] (Pause.)
[00:09:38.000 --> 00:09:40.000] (Pause.)
[00:09:40.000 --> 00:09:42.000] (Pause.)
[00:09:42.000 --> 00:09:44.000] (Pause.)
[00:09:44.000 --> 00:09:46.000] (Pause.)
[00:09:46.000 --> 00:09:48.000] (Pause.)
[00:09:48.000 --> 00:09:50.000] (Pause.)

I see a lot of similar cases.
Love Whisper CPP,
Piotr

[albino1]

Search this github and the actual Whisper one for "hallucinations", "duplicating", "repetition", "repeating", etc, there's a ton of posts about this. Whisper pushed a potential fix a few weeks back, but it hasn't made its way into whisper.cpp yet, so no idea if it helps.

[albino1]

It falls apart all the time for me, and in completely different ways with different models. For me it mostly fails with non-English languages, probably like 80% of the time in my tests, or 100% if they're long enough, but I've had it happen plenty of times with English as well.

Here's some examples of people mentioning this on the actual OpenAI Whisper page, as well as here on whisper.cpp, and on various forks like this GPU accelerated version.

[pcbua]

Thx for the info.
To be more precise, it's not a real hallucination, it's a dummy repetition.
A typical workaround might be to do some kind of soft reset from time to time.
A hard cut is also possible, but requires a heuristic of where to cut and it will resets the context which will decrease the accuracy.

Perhaps there are internal Whisper CPP parameters that can be soft reset.

[albino1]

Repetitions are the problem I'm seeing, but I've seen people refer to them as hallucinations which is why I mentioned it.

There's several suggested workaround here, like this one for example to redo sections, or this one to adjust the entropy threshold. Maybe give some of those a try and see if it fixes your specific issue.

Either way, I hope you're right and they can figure it out, because this project is amazing when it works.

[rdhox]

I had this problem after 4min on an russian wav file that I was translating to english, but I also had no problem with another long wav (more than 1hour) from russian to english that works perfectly. So I doubt it's linked to the length of audio. When I checked the moment where it start to repeat itself, I didn't see any strange noise or anything that could make whisper "bug".

But the problem is systematic, not random.

[pcbua]

True, it is not related to the length, I've spotted it in 4s (example above: "[00:00:04.000 --> 00:00:06.000] >> I need to --").
Maybe it is just rare on short recordings - this could be the reason why developers didn't noticed it during release.
When you test on short recordings only, such instability can be missed in final metric (WER), especially when it is rare.
But if it happens during long recording, it tend to not work after it happen and significant part of the recording is wrong so WER goes up rapidly.