Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Corrupt video file makes server 2.4 crash #1508

Closed
Sidonai-1 opened this issue Nov 21, 2023 · 11 comments
Closed

Corrupt video file makes server 2.4 crash #1508

Sidonai-1 opened this issue Nov 21, 2023 · 11 comments
Labels
ffmpeg-producer type/bug

Comments

@Sidonai-1
Copy link

I have a corrupt video file encoded in UHD ProRes 4444. The download from Dropbox probably failed and the last seconds of the file are corrupted.

On 2.3.2 with a UHD channel ffmpeg complains, throws an error but server stays alive.

Starting CasparCG Video and Graphics Playout Server 2.3.2 4de6d18f Dev

[2023-11-17 11:40:04.415] [error]   [ffmpeg] [prores @ 000001C698940100] invalid frame header
[2023-11-17 11:40:04.415] [error]   
[2023-11-17 11:40:04.442] [error]   Exception: C:\Program Files (x86)\Jenkins\workspace\casparcg-server-dep\master\src\modules\ffmpeg\producer\av_producer.cpp(153): Throw in function auto __cdecl caspar::ffmpeg::Decoder::()::<lambda_cf087faeeb9dabf84379bbd46f77baec>::operator ()(void) const
[2023-11-17 11:40:04.442] [error]   Dynamic exception type: class boost::exception_detail::clone_impl<struct caspar::ffmpeg::ffmpeg_error_t>
[2023-11-17 11:40:04.442] [error]   [struct boost::errinfo_api_function_ * __ptr64] = avcodec_send_packet(ctx.get(), input.front().get())
[2023-11-17 11:40:04.442] [error]   [struct boost::errinfo_errno_ * __ptr64] = 1094995529, "Unknown error"
[2023-11-17 11:40:04.442] [error]   [struct caspar::ffmpeg::tag_ffmpeg_errn_info * __ptr64] = -1094995529
[2023-11-17 11:40:04.442] [error]   [struct caspar::tag_stacktrace_info * __ptr64] =  0# 0x00007FF6BDEBA37E in casparcg
[2023-11-17 11:40:04.442] [error]    1# 0x00007FF6BDEE9690 in casparcg
[2023-11-17 11:40:04.442] [error]    2# 0x00007FF6BE0F12E0 in casparcg
[2023-11-17 11:40:04.442] [error]    3# 0x00007FF6BE0F4602 in casparcg
[2023-11-17 11:40:04.442] [error]    4# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    5# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    6# 0x00007FF6BE0F49E2 in casparcg
[2023-11-17 11:40:04.442] [error]    7# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    8# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    9# 0x00007FF6BE0E8CB3 in casparcg
[2023-11-17 11:40:04.442] [error]   10# 0x00007FF6BE0E8A4B in casparcg
[2023-11-17 11:40:04.442] [error]   11# 0x00007FF6BE0F4B5A in casparcg
[2023-11-17 11:40:04.442] [error]   12# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   13# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   14# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   15# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   16# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   17# configthreadlocale in ucrtbase
[2023-11-17 11:40:04.442] [error]   18# BaseThreadInitThunk in KERNEL32
[2023-11-17 11:40:04.442] [error]   19# RtlUserThreadStart in ntdll
[2023-11-17 11:40:04.442] [error]   
[2023-11-17 11:40:04.442] [error]   

[2023-11-17 11:40:04.442] [error]    0# 0x00007FF6BDEBA37E in casparcg
[2023-11-17 11:40:04.442] [error]    1# 0x00007FF6BDEB9CAF in casparcg
[2023-11-17 11:40:04.442] [error]    2# 0x00007FF6BE438ED7 in casparcg
[2023-11-17 11:40:04.442] [error]    3# 0x00007FF9C61A1030 in VCRUNTIME140
[2023-11-17 11:40:04.442] [error]    4# is_exception_typeof in VCRUNTIME140
[2023-11-17 11:40:04.442] [error]    5# RtlCaptureContext2 in ntdll
[2023-11-17 11:40:04.442] [error]    6# 0x00007FF6BE0F0825 in casparcg
[2023-11-17 11:40:04.442] [error]    7# 0x00007FF6BE2838E3 in casparcg
[2023-11-17 11:40:04.442] [error]    8# configthreadlocale in ucrtbase
[2023-11-17 11:40:04.442] [error]    9# BaseThreadInitThunk in KERNEL32
[2023-11-17 11:40:04.442] [error]   10# RtlUserThreadStart in ntdll
[2023-11-17 11:40:04.442] [error]   
[2023-11-17 11:40:39.238] [info]    Received message from 127.0.0.1: STOP 1-20\r\n
[2023-11-17 11:40:39.238] [info]    Sent message to 127.0.0.1:202 STOP OK\r\n

On the 2.4 builds that I tested the error makes Caspar crash and it closes.

Starting CasparCG Video and Graphics Playout Server 2.4.0 80f8ee7 Dev

[2023-11-17 11:31:46.937] [error]   [ffmpeg] [prores @ 0000020B673FE500] invalid frame header
[2023-11-17 11:31:46.937] [error]   
[2023-11-17 11:31:46.939] [fatal]   #######################
[2023-11-17 11:31:46.939] [fatal]    UNHANDLED EXCEPTION: 
[2023-11-17 11:31:46.939] [fatal]   Adress:00007FF9CE02CF19
[2023-11-17 11:31:46.939] [fatal]   Code:3765269347
[2023-11-17 11:31:46.939] [fatal]   Flag:1
[2023-11-17 11:31:46.939] [fatal]   Info:0000000FC21FF730
[2023-11-17 11:31:46.939] [fatal]   Continuing execution. 
[2023-11-17 11:31:46.939] [fatal]   #######################

I tried both UHD and a custom video mode in the 2.4 build and the result is the same.

I cannot share the video file publicly at the moment, but I can send it to @Julusian via the forum if needed. Here are the details:

General
Complete name                            : D:\2_TRAILER_RO.mov
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   0000.00.01 (qt  )
File size                                : 15.9 GiB
Duration                                 : 2 min 17 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 993 Mb/s
Frame rate                               : 25.000 FPS
Encoded date                             : 2023-09-26 06:52:16 UTC
Tagged date                              : 2023-09-26 06:52:16 UTC

Video
ID                                       : 1
Format                                   : ProRes
Format version                           : Version 1
Format profile                           : 4444
Codec ID                                 : ap4h
Duration                                 : 2 min 17 s
Bit rate mode                            : Variable
Bit rate                                 : 990 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:4:4
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 4.777
Stream size                              : 15.9 GiB (100%)
Writing library                          : avi1
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : in24
Duration                                 : 2 min 17 s
Bit rate mode                            : Constant
Bit rate                                 : 2 304 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 24 bits
Stream size                              : 37.8 MiB (0%)

Other
ID                                       : 3
Type                                     : Time code
Format                                   : QuickTime TC
Duration                                 : 2 min 17 s
Frame rate                               : 25.000 FPS
Time code of first frame                 : 00:00:00:00
Time code of last frame                  : 00:02:17:12
Time code, stripped                      : Yes

Environment

  • Server version: 2.3 vs 2.4
  • Operating system: Windows 10
@PeterAkakpo
Copy link

i have confirmed that too

@Sidonai-1
Copy link
Author

We keep having this problem with some XDCam files, when they reach the end the server crashes.

Also some ProRes 422 files that have been incorrectly downloaded or are corrupt in some other way.

Do we know if this is something related to the ffmpeg version or could this be fixed on the CasparCG side?

This is becoming more of an issue recently.

Thanks.

@Bohoaush
Copy link

It is a CasparCG bug introduced when the ffmpeg producer has been rewritten to use multiple threads b657428. Previously any error from ffmpeg would be catched, so the playback would stop, but the server would keep running, now this would crash the server. I currently use a fix like this in production (got the code from #1469, but not the latest iteration). You might get audio sync issues after playing broken part of a file, but that's obviously better than the whole server crashing.

@Sidonai-1
Copy link
Author

It is a CasparCG bug introduced when the ffmpeg producer has been rewritten to use multiple threads b657428. Previously any error from ffmpeg would be catched, so the playback would stop, but the server would keep running, now this would crash the server. I currently use a fix like this in production (got the code from #1469, but not the latest iteration). You might get audio sync issues after playing broken part of a file, but that's obviously better than the whole server crashing.

Yeah that'd be much more desireable!

If you think the code is compatible with the current master you could make a pull request with the fix and I can test the next build when it gets approved?

Thanks.

@Bohoaush
Copy link

I already tried creating PR all the way in 2022, but current repo maintainer Julusian is hesitant to merge because he lacks deep understanding of ffmpeg part of Caspar (per his words) and is not feeling comfortable about potential impact for other formats.

If you are running Linux, I can give you my build, however I am unable to build for Windows, but you can of course build it yourself.

@Julusian
Copy link
Member

I'm curious whether the fix c676f8f has any impact on this, but don't appear to have a file I can test with. Could someone retry with the latest master build and let me know?

@Bohoaush
Copy link

You can try seeking with https://soubory.tvnoe.cz/s/seekTOcrash
Looking at the change I think after this change it wouldn't crash, but it would stop playing. I will test as soon as I will have some time.

@Julusian
Copy link
Member

@Bohoaush That file plays and seeking results in it logging an "unknown error" and seeking. It might be pausing for a couple of frames too, but it does seek and continue playback.
I can do that multiple times without issue.

@Bohoaush
Copy link

I can now also confirm that with the current master it can seek this file and even correctly play actually corrupted file (can send, but not publicly and the smallest one is 4GB) and the playback continues normally. Thank you @Julusian & @niklaspandersson

@Julusian
Copy link
Member

@Sidonai-1 I'm going to leave this open for a bit, until you can report whether it resolves everything you have been seeing. Please report back once you have had a chance to test it

@Sidonai-1
Copy link
Author

@Sidonai-1 I'm going to leave this open for a bit, until you can report whether it resolves everything you have been seeing. Please report back once you have had a chance to test it

I can confirm the new master (I tried 2.5.0 895a5a0 Dev) complains 8 times (with the video I tested at least)

[2025-01-16 15:11:11.287] [error]   Exception: D:\a\server\server\src\modules\ffmpeg\producer\av_producer.cpp(181): Throw in function auto __cdecl caspar::ffmpeg::Decoder::{ctor}::<lambda_d43a1656b995bdd7bd9d1a3fb738b118>::()::<lambda_fc29d71e538320da01736326bed0d02f>::operator ()(void) const
[2025-01-16 15:11:11.287] [error]   Dynamic exception type: struct boost::wrapexcept<struct caspar::ffmpeg::ffmpeg_error_t>
[2025-01-16 15:11:11.287] [error]   [struct boost::errinfo_api_function_ * __ptr64] = avcodec_send_packet(ctx.get(), packet.get())
[2025-01-16 15:11:11.287] [error]   [struct boost::errinfo_errno_ * __ptr64] = 22, "Invalid argument"
[2025-01-16 15:11:11.287] [error]   [struct caspar::ffmpeg::tag_ffmpeg_errn_info * __ptr64] = -22
[2025-01-16 15:11:11.287] [error]   [struct caspar::tag_stacktrace_info * __ptr64] =  0# 0x00007FF7B6D91342 in casparcg
[2025-01-16 15:11:11.287] [error]    1# 0x00007FF7B6DBB689 in casparcg
[2025-01-16 15:11:11.287] [error]    2# 0x00007FF7B6EF9E2E in casparcg
[2025-01-16 15:11:11.287] [error]    3# 0x00007FF7B6EF95D0 in casparcg
[2025-01-16 15:11:11.287] [error]    4# 0x00007FF7B6FBE13A in casparcg
[2025-01-16 15:11:11.287] [error]    5# configthreadlocale in ucrtbase
[2025-01-16 15:11:11.287] [error]    6# BaseThreadInitThunk in KERNEL32
[2025-01-16 15:11:11.287] [error]    7# RtlUserThreadStart in ntdll
[2025-01-16 15:11:11.287] [error]
[2025-01-16 15:11:11.287] [error]
[2025-01-16 15:11:11.287] [error]    0# 0x00007FF7B6D91342 in casparcg
[2025-01-16 15:11:11.287] [error]    1# 0x00007FF7B6D90DFD in casparcg
[2025-01-16 15:11:11.287] [error]    2# 0x00007FF7B70692F0 in casparcg
[2025-01-16 15:11:11.287] [error]    3# _CxxFrameHandler4 in VCRUNTIME140_1
[2025-01-16 15:11:11.287] [error]    4# 0x00007FFA5ABB2572 in VCRUNTIME140_1
[2025-01-16 15:11:11.287] [error]    5# RtlCaptureContext2 in ntdll
[2025-01-16 15:11:11.287] [error]    6# 0x00007FF7B6EF95D0 in casparcg
[2025-01-16 15:11:11.287] [error]    7# 0x00007FF7B6FBE13A in casparcg
[2025-01-16 15:11:11.287] [error]    8# configthreadlocale in ucrtbase
[2025-01-16 15:11:11.287] [error]    9# BaseThreadInitThunk in KERNEL32
[2025-01-16 15:11:11.287] [error]   10# RtlUserThreadStart in ntdll
[2025-01-16 15:11:11.287] [error]

But it does not crash.

I'll close it as completed.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ffmpeg-producer type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Julusian @PeterAkakpo @Bohoaush @Sidonai-1