HTML5X Plugin subtitles styling on UWP

[slonopot]

Hey,

There's an issue with subtitles styles on UWP devices. When enabled, captions are displayed with a default style (tiny black Times New Roman) instead of the expected large white font like other platforms.

I've attempted to fix it manually by cloning the plugin and updating styles, it seems that the webview does not even attempt to load common.css and ignores any styles in the html file itself.

I can't investigate this any further because the webview debugging is not enabled, but you can enable it in the next build and I'll take a look myself.

Thanks!

[benzac-de]

Hey,

Thank you for the hint.

Please note that the HTML5X plugin does not yet implement subtitle styles (only the platform-dependent standard styles are displayed, as you already mentioned). For UWP devices, I recommend testing your HTML5X plugin on the Microsoft Edge browser and checking/debugging whether your subtitle styles are displayed correctly. I would really appreciate it if you would also share your plugin project on GitHub, then I would definitely take a look at it.

Best regards,
benzac

[slonopot]

Thanks for your response!

I'm not exactly creating my own plugin since yours is working just fine. What I wanted to achieve is to figure out why does this issue even exist. I wanted to find a way to add custom styles, preload extra fonts in case the webview can't pick them up from cache or other contexts. My last observation was that the webview ignores any styles in the player iframe (if it's an iframe).

I've tested Edge with the web version of MSX and it loads the styles just fine, take a look:

And it's a desktop UWP version:

As I said earlier, I am unable to debug it any further since I'm unable to see why does it ignore styles. After some research I figured that the most stable multiplatform approach for captions would be completely disabling them in the player and displaying them as an overlay manually. I guess I can clone your player and do this, but then I still have an issue when the webview in UWP does not try to load styles (i.e. I can see in nginx logs that it loads html5x.html and does not load common.css).

I'd appreciate a debug mode for the webview in UWP, this will at least allow me to not blindly test stuff.

[benzac-de]

Thank you for the explanation.

Just to understand it correctly: You have downloaded the html5x.html, the html5x.js, and the common.css files, put them on your server and are now trying to load them via MSX? But only the HTML and JS files are loaded and not the CSS file? Sounds to me like the CSS file is incorrectly referenced in the HTML file.

Would it be possible to send me a test link to the hosted HTML5X version so I can take a look?

Regarding the UWP debug mode: Unfortunately, I cannot currently test the MSX UWP version because I no longer have a suitable device. That is why I cannot simply activate the debug mode without testing it first. I would also not like to activate the debug mode in the production version (rather in a separate test version).

[slonopot]

You have downloaded the html5x.html, the html5x.js, and the common.css files, put them on your server and are now trying to load them via MSX?

I got your html5x.html, changed the relative links to full links to your server for JS files and to mine for common.css, got the common.css, added some changes to see if it even works, uploaded html5x.html and common.css to my server and tested it in MSX. When I use the web MSX version, it loads html5x.html and then common.css. When I try UWP MSX, only http5x.html is loaded from my server. Since the player works, JS files are loaded properly from your server. When I embed styles in the html file, they are also ignored (but I don't have a way to tell that since I can't see the webview console). When I open my html5x.html in Chrome, all files and styles are loaded properly from the correct places which can only mean that it's the webview that fails to load the styles both from your server and mine (and any other).

I no longer have a suitable device

I can give you access to a Windows 11 VM if you are willing to assist me with this.

[benzac-de]

Thank you for the further explanation, now I understand everything. Also thank you for providing a VM access, but I think we will be able to solve this issue without it.

When I embed styles in the html file, they are also ignored

If even that does not work, it seems that the Microsoft Edge browser used for UWP devices does not support CSS subtitle styles.

I have now uploaded a second HTML5X plugin (html5x2.html) to my server that sets basic CSS subtitle styles. Could you please check if this plugin works with your UWP device?

Usage: video:plugin:http://msx.benzac.de/plugins/html5x2.html?url={URL}

[slonopot]

Nope, doesn't work. Again, blindly pushing stuff won't help.

Another interesting bit here is that this webview caches styles. I don't know how and why would it survive restarts, but it does not reload them. Once reset, the app loads all files properly and my initial idea that it ignores styles is invalid.
Take a look:

For now, I believe, the theory should go like this: we have a default browser that has some default styles for the player, captions included. Since webview is intended to be shipped with the app bundle, it might be stripped to lose even the caption styles. Since styling the captions is a challenge itself, we can't just randomly try to apply styles and we should figure out how does it look in the webview to directly apply settings as the platform requires.

Since styles are actually loaded, at this moment I can either create my own plugin with custom subtitles display that will be based on yours and will be multiplatform and better customizable, or I can find a way to fix it for UWP only and be good to go for now. If you are unable to make a testing build for UWP at the moment, I'll go with the first option and see how would it work.

[benzac-de]

Thanks for testing the new plugin. I think we have already gained a lot of insight into the problem.

Can you please tell me which browser version is used on your UWP device (Settings → About → User Agent)?
Based on the latest information, I am pretty sure that this version does not support the CSS pseudo class ::cue (https://caniuse.com/?search=%3A%3Acue), which is used for styling subtitles. If so, only the "custom subtitles display" approach you suggested would work (i.e. reading out textTracks yourself and writing them into a div that can be styled however you want).

UPDATE: I now have a way to test the UWP version. On a Windows 10 Pro machine, the current MSX UWP version uses an Edge/18.19045 build. Therefore, the ::cue class is not supported. However, the standard subtitles look okay (medium-sized white font with black border).

[slonopot]

Mine are Edge/18.22631 on Windows and Edge/18.25398 on Xbox and both have the problem.

I think it's still possible to apply custom styles if the actual element is known, but it will be a workaround anyway, probably temporary if it works as expected for the older version.

[slonopot]

Let's pick this up once again. I've built a version of your html5x that supports styled subtitles, here it is in the testing branch, used as https://slonopot.github.io/msx-html5xs/html5xs.html. Basically it moves the player to the background, makes an overlay with a single element which would contain track cues. Subtitles that are loaded from the internet in the latest Chrome are actually loaded the moment they are enabled, thus the hack with waiting for cuechange to apply all events here and there.

That's not the problem though since I was testing the build with your example from here and it was perfect, but when I switched back to my main source to finally be satisfied with the results -- nothing happened. Turns out I was mixing up HLS streams with ordinary MP4 files, HLS streams are not supported by Chrome, Firefox won't allow me to disable CORS for testing. I can only test this combo of HLS and html5x on Xbox and in Safari. In Safari it actually works as expected, on Xbox it does not show any captions at all (meaning the ordinary ones get disabled by the player but the replacement is missing somewhere). It may also be cached, I'll see if that's the case.

Which technology is used as a webview for UWP builds? I'd still like some debugging, not sure how to work this out on Xbox though unless you'll provide some testing build for the developer mode. If it's something old, would you consider switching to the more recent engine?

I also previously figured the reason html5x is the only plugin to support subtitles is the technical limitations of all the rest, but now I look at hls.js and see that it supports such function. I'll try to extend either hls.js or video.js (or both) with subtitles support, maybe they will be more cooperative on this matter.

[benzac-de]

Thank you for picking up this topic again. Nice work with the html5xs plugin. I will test it when I get the chance. To your questions.

Which technology is used as a webview for UWP builds?

I am using a JavaScript UWP project template that has already been deprecated. The app package only contains HTML, CSS, and JS (no C# wrapper code is involved). I am not sure if a debug package version would even be possible with this approach.

If it's something old, would you consider switching to the more recent engine?

I have already thought about switching to the new UWP WebView2 approach. However, currently I do not have the time and the motivation to start something like this, especially because the old approach has worked well so far.

now I look at hls.js and see that it supports such function

I have also thought about adding subtitle support to various plugins, since almost all of them support subtitles. However, this will take longer, as every implementation is different and would have a long trial and error phase (especially if subtitle styling should also be available). Here, too, I do not have the time and the motivation to start something like this right now. However, I would be happy to assist you with potential implementations and to answer any question.

[slonopot]

Thanks for your responses, I don't think there's anything I can't fix myself apart from bumping a browser version on UWP (but targeting Chrome 70 is still necessary for crossplatform compatibility with cheap trash TVs, I guess). I only really need UWP to run on Xbox to use it as a stream player, the alternative would be Kodi, it's more functional overall but it's also too much for a simple grid, couple of buttons and a player.

I guess I'll try to copy your approach to audio tracks and subtitles from html5x to hls.js, merge it with my styling and see if it will work with HLS streams in Chrome and later on other platforms. I'll post it here once it comes together.

As a side question about formatting I'd like to understand why do I have issues with templates in pages, I only want to place equal buttons one after another but the only error I get when I try to copy your approach is that there's no content or nothing is selectable. I've resorted to an ugly solution here, line 57, but I'll appreciate if you'll point me in the right direction to make it work with a template.

[slonopot]

I guess I finished the job with html5x(s) upgrading, I left some notes in the readme to better understand the context. I've been able to debug the plugin with ajax since all other methods were not leaving any messages, turns out the Xbox browser engine would load only one cue at a time for a stream unlike Safari where all the loaded cues are always available. The subtitles were actually working, they were just never been able to trigger since it's already too late to show anything.

This mod works just fine with VTT and the MP4 test file in Chrome, I believe it will also work on the majority of other platforms except UWP. When the remote MP4 file is loaded in UWP, the player just rejects it, maybe it's a strict CORS policy of the webview.

I pushed the last update, I will redirect my efforts to hls.js and see if it will process cues in a more predictable way

[benzac-de]

Very good job on the plugin and the description.

A few notes on CORS in this context: When I was developing the HTML5X plugin, I noticed a big CORS issue. In order to load subtitle track URLs from a different domain as the document, the crossOrigin property of the player has to be set to "anonymous". However, this also means that the video URL and subtitle track URLs must contain the CORS headers, which is quite unusual on the web (generally, video URLs do not have CORS headers). By setting the "html5x:cors" property to "false" the crossOrigin property will not be set to "anonymous". However, this means that the subtitle track URLs must have the same origin as the document, which means that we need a proxy to load them. I see that this proxy is still linked in your HTML5XS plugin, but it does not work because it does not exist. I have now added CORS headers for the Sintel video and subtitle track URLs. Now I was able to successfully test your plugin with the Sintel demo in a Chrome browser.

[benzac-de]

As a side question about formatting I'd like to understand why do I have issues with templates in pages, I only want to place equal buttons one after another but the only error I get when I try to copy your approach is that there's no content or nothing is selectable. I've resorted to an ugly solution here, line 57, but I'll appreciate if you'll point me in the right direction to make it work with a template.

For templated items, you can use the createAudioTracksPanel function (line: 535) as reference.
For more advanced templating, please have a look at this wiki page: https://msx.benzac.de/wiki/index.php?title=Advanced_Templated_Items

[slonopot]

Back at it again.

I upgraded your hls.js to support both audio and text tracks, it's now live here. The problem is that this didn't help a single bit to let me use the subtitles on xbox. Well, by default hls.js would wrap around default <video> component that is provided by html5 player by default, in this scenario everything works as expected in Chrome and does literally nothing in UWP apps. There's another option to render the cues manually, hls.js provides a CUES_PARSED event that callbacks a list of VTTCue object for later use. The only obvious approach here would be to add a TextTrack in player (player.addTextTrack('subtitles')), get cues in the event, assign onenter and onexit events for styling and append them to this track with player.textTracks[0].addCue. The problem is that this shit doesn't work on UWP again. It turns out that hls.js is running in compatibility mode where it can't use window.VTTCue and replaces it with some internal-typed object. No matter how I attempt to convert it to an object that's compatible with addCue, it always fails.

I've tried to run MSX in IE11 to see if your webview is that -- it is not. However, since we previously discussed that it's an outdated technology, I finally realized that there were times back until 2021 when Edge was using EdgeHTML and not Chromium. Edge on Xbox moved to Chromium later on, but as it always happens with Microsoft -- the remains are still there. I believe that your build still is based on EdgeHTML version of Edge that was undercooked the whole way and that's why it's so complicated to achieve what I want.

At this point I have no way to debug and fix this, this will require a whole custom player that will be compatible with prehistoric browsers and I haven't seen a good one yet. I'd also like to ask you to take a look if it will be possible to move to WebView2 at some point in the near future.

Also, do you accept donations in crypto?

[benzac-de]

Nice, once again very good work on the plugin.

I have just tested your HTML5XS plugin in the MSX UWP app on a Windows 10 machine (Edge/18.19045 agent) and it works. I think it needs to be optimized here and there, but basically the subtitles are displayed with the selected style. The biggest issue here remains the missing proxy to load subtitle files in a CORS-less mode ("html5x:cors": "false"). I will consider adopting your approach and deploying a HTML5XS variant on my server. Then the proxy would be available.

Could you please re-check if your HTML5XS plugin now works with the "Sintel" assets on the Xbox?

Finally, some answers/comments to your questions/remarks.

I believe that your build still is based on EdgeHTML version of Edge

Yes, I think so too.

I'd also like to ask you to take a look if it will be possible to move to WebView2 at some point in the near future.

I have added it to the list of possible extensions.

Also, do you accept donations in crypto?

No, currently I do not have a crypto donation wallet.

[slonopot]

I think it needs to be optimized here and there, but basically the subtitles are displayed with the selected style.

Agreed, this configuration menu does not look good. I still can't figure out the templates there though, maybe I'll rework it completely when I'll be making hls.js version to support customizable subtitles, but that'll happen once subtitles start working on Xbox.

Could you please re-check if your HTML5XS plugin now works with the "Sintel" assets on the Xbox?

Yep, it works now with fixed CORS and static files. HLS, however, is another story.

No, currently I do not have a crypto donation wallet.

Maybe it's time to make one?

[benzac-de]

Agreed, this configuration menu does not look good. I still can't figure out the templates there though, maybe I'll rework it completely when I'll be making hls.js version to support customizable subtitles, but that'll happen once subtitles start working on Xbox.

An important hint here would be that templated items all have the same size within the content root object. If different item sizes should be displayed (as is the case right now), the template object cannot be used.

Yep, it works now with fixed CORS and static files. HLS, however, is another story.

Great, this is good news.

Maybe it's time to make one?

I will think about it.