(WIP) Use VT for manually decoding frames. Fixes #533; Improve latency and Smoothness #534
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Two main changes
1 - Use VideoToolbox to manually decode each frame instead of submitting it directly to AVSampleBufferDisplayLayer; I'm not proud of this change, but it was needed to fix #533. There may be some way to fix the issue without needing this change, but I still didn't manage to do it.
2 - Latency and smoothness changes
2.1 - Use Direct Submit in VideoDecodeRenderer (reduces latency)
2.2 - Use PTS information correctly per frame instead of using the DisplayImmediately flag in each sampleBuffer. Together with the change above, I was able to replicate smooth low latency stream as I get into the Nvidia Shield. I think using the flag messed with frame time and caused jittering.
Right now, I'm breaking the "Smooth Stream" option that we added some months ago, but wanted to create the PR either way for us to discuss options @cgutman