Fix: differentiate shorts, lives and long videos

[arjitcodes]

Fixes #366 #367

Changes in scraper

1. Add userLongUploadsPlaylist, userShortUploadsPlaylist, and userLivesPlaylist in channel.json, which contain the user's uploaded long videos, shorts, and live videos, respectively, to mirror YouTube's organized experience, even offline, such as dedicated user uploads Videos (longs) tab, shorts tab, and lives tab.

Changes in zimui

1. Add CSS in vjs-youtube.css to correct short video resolution.
2. Add videos shorts lives tab
   • videos contains long videos uploaded by the channel.
   • shorts contains shorts uploaded by the channel
   • lives contains lives by the channel
3. Add the channel-home route, which points to the home tab. The home tab was previously named the videos tab.
4. Update channelHeader for dynamic tabs to display only available tabs, such as shorts, videos, and lives only if available in the channel.

Visuals

Old Home

New Home

Old Player

New Player

[arjitcodes]

@kelson42

[benoit74]

Thank you, I'm amazed by the intent of this first contribution.

Unfortunately, I still have many questions / doubts that need to be clarified before we can dive in code details:

• why are you sometimes speaking about normal videos and at other times about long videos? I'm a bit puzzled. And I don't get why we have only one very specific playlist here, where from my experience we often have many playlists of normal/long videos per channel in many cases
• why are you making new calls to retrieve specific playlist IDs (get_long_videos_playlist_id, ...) where get_channel_playlists_json is already supposed to retrieve all playlists of the channel?
• why are you using so much magic in is_short function where you have a magic cut-off date, a magic maximum duration ... this needs to be justified at least
• why are you making a new API call in is_short instead of just checking the playlist ID of current video starts with the magic string? We need to limit API calls at their strict minimum to avoid having throttling issues
• why do you differentiate lives from long videos? Are they really so special they need special care? does it make any sense in an offline scenario? Do live videos stays in the live playlist once the live is over?

[arjitcodes]

• I apologize for the confusion caused by my use of both 'normal videos' and 'long videos' – they actually refer to the same type of content. To clarify, 'normal/long videos' represent user-uploaded videos that are longer in format and don't include Shorts or live streams, which are already organized separately within the channel. The long_videos_playlist is specifically designed to help us render these videos in a style similar to YouTube's main video section, making it easier to access the user uploaded long-form content in a dedicated playlist. Thanks for pointing this out, and I've made the necessary adjustments to ensure consistency!

• The main reason for these additional calls to retrieve specific playlist IDs like get_long_videos_playlist_id is that get_channel_playlists_json gives us a full list of the channel's playlists, including the general 'user uploads' playlist, which contains all videos (shorts, long videos, and live streams). However, to mirror YouTube's organized experience, even offline, I’m specifically targeting separate playlists for long_videos_playlist, shorts_playlist, and lives_playlist. These allow us to categorize user-uploaded long videos, shorts, and live content separately, just as YouTube has distinct tabs for Videos (long videos), Shorts, and Live. This approach gives users a familiar, organized browsing experience similar to YouTube’s.
  I'm currently refining these naming conventions to make their purpose clearer, and I’m open to any suggestions you may have!

• I see where you're coming from! I initially added the is_short function to help identify short videos based on a cutoff date and maximum duration, but in retrospect, it’s unnecessary since now the frontend player can show the short video correctly without explicitly marking it. To simplify things and avoid adding extra logic, I’ll remove the is_short function for now. Thanks for catching this, and I apologize for the added complexity—I’ll be more cautious to avoid similar situations in the future!

• The main reason for differentiating live videos from long videos is to mirror the YouTube experience, where live streams have their own dedicated tab.

[benoit74]

Thank you !

I still don't get why you are making additional API calls. The playlists you need are already in the results of get_channel_playlists_json, why can't you simply use information already retrieved here? To me it looks like it is just a matter of finding the playlist with the ID matching the convention you are using.

And on top of that, regarding this convention (short playlist is "UUSH" + channel_id[2:], live playlist is "UULF" + channel_id[2:], video playlist is "UULV" + channel_id[2:]), where does it comes from? I have at least one example where this is purely not working. For CarbiDIY channel (ID UCuct2fNB1nSdS8Dl6XzdA1g), the short playlist ID is PLUo94RsjxOG82eLooMlRoCS2vWOqzB4_m, there are multiple long videos playlists (all these videos are visible in the "Videos" tab on youtube) and the live playlist is not listed in the playlists API endpoint. See below. Do I miss something?

Playlists of channel UCuct2fNB1nSdS8Dl6XzdA1g (CabriDIY)

{ "kind": "youtube#playlistListResponse", "etag": "RqabC8kHsOhhJm-tqxTMpmM5RoM", "pageInfo": { "totalResults": 22, "resultsPerPage": 25 }, "items": [ { "kind": "youtube#playlist", "etag": "vn_h-lgruI6dTb8FkcNMTKyxkfs", "id": "PLUo94RsjxOG82eLooMlRoCS2vWOqzB4_m", "snippet": { "publishedAt": "2023-10-18T13:49:02Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Short", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/U84iQbF7ob8/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/U84iQbF7ob8/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/U84iQbF7ob8/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/U84iQbF7ob8/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/U84iQbF7ob8/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Short", "description": "" } }, "contentDetails": { "itemCount": 7 } }, { "kind": "youtube#playlist", "etag": "G9NnDqOYdC61FlYVMF0NQPRRgBc", "id": "PLUo94RsjxOG9XYbTrSBwaUBRz_dSzC4am", "snippet": { "publishedAt": "2023-06-17T13:55:16Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "TikTok", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/tgfpj-Q1m80/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/tgfpj-Q1m80/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/tgfpj-Q1m80/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/tgfpj-Q1m80/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/tgfpj-Q1m80/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "TikTok", "description": "" } }, "contentDetails": { "itemCount": 6 } }, { "kind": "youtube#playlist", "etag": "4rPLddxHvgD-EVEOR3H2OOV0Mu0", "id": "PLUo94RsjxOG_C3ybIF7YxiRfhxwLYoMKq", "snippet": { "publishedAt": "2022-03-20T15:51:41Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Marathon", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/tyMbsYPd0zo/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/tyMbsYPd0zo/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/tyMbsYPd0zo/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/tyMbsYPd0zo/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/tyMbsYPd0zo/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Marathon", "description": "" } }, "contentDetails": { "itemCount": 3 } }, { "kind": "youtube#playlist", "etag": "AD5glo4vlrDp72umzslhUpFyH4s", "id": "PLUo94RsjxOG_Yt5kh1iklXpeH8zGMo5gB", "snippet": { "publishedAt": "2021-12-16T22:05:32Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "PlayStation 1 miroir", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/1TeRgE25fjo/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/1TeRgE25fjo/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/1TeRgE25fjo/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/1TeRgE25fjo/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/1TeRgE25fjo/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "PlayStation 1 miroir", "description": "" } }, "contentDetails": { "itemCount": 6 } }, { "kind": "youtube#playlist", "etag": "k-tihhlyFRaZwXxLFDDczkniMZQ", "id": "PLUo94RsjxOG8FEjZBeAcoM1u3j7b8rqJj", "snippet": { "publishedAt": "2021-07-28T11:14:15Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Réparation de l'extrême", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/BDjmFoSoqEM/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/BDjmFoSoqEM/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/BDjmFoSoqEM/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/BDjmFoSoqEM/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/BDjmFoSoqEM/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Réparation de l'extrême", "description": "" } }, "contentDetails": { "itemCount": 2 } }, { "kind": "youtube#playlist", "etag": "lfUx3NpnZfPx85tzXhAFM9XLeiM", "id": "PLUo94RsjxOG_QCmJOaSICFWmmcl3YdJ_k", "snippet": { "publishedAt": "2021-06-23T16:08:48Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Présentation PPT", "description": "Voici la playlist des différentes présentations de console réalisés en live sur Twitch. Certaines sont des VODs complètent mais commencent par la présentation de la console.", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/3JyY84OZQ5U/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/3JyY84OZQ5U/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/3JyY84OZQ5U/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/3JyY84OZQ5U/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/3JyY84OZQ5U/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Présentation PPT", "description": "Voici la playlist des différentes présentations de console réalisés en live sur Twitch. Certaines sont des VODs complètent mais commencent par la présentation de la console." } }, "contentDetails": { "itemCount": 8 } }, { "kind": "youtube#playlist", "etag": "Hq0KyqZZij6lSxObn5zqURiX5C8", "id": "PLUo94RsjxOG8e0lku3sOAY3rsg4N7o7CP", "snippet": { "publishedAt": "2020-11-04T20:43:01Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Free copyright music", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/kcUh5viJCAo/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/kcUh5viJCAo/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/kcUh5viJCAo/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/kcUh5viJCAo/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/kcUh5viJCAo/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Free copyright music", "description": "" } }, "contentDetails": { "itemCount": 4 } }, { "kind": "youtube#playlist", "etag": "SxVJiqpTVvoCKFwrerSJlXYIcwo", "id": "PLUo94RsjxOG8E7f5OaaX_iMPqC2UGizdT", "snippet": { "publishedAt": "2020-11-03T17:15:56Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Modernisation Radio 1950", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/00jJBwZoOC4/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/00jJBwZoOC4/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/00jJBwZoOC4/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/00jJBwZoOC4/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/00jJBwZoOC4/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Modernisation Radio 1950", "description": "" } }, "contentDetails": { "itemCount": 14 } }, { "kind": "youtube#playlist", "etag": "aygeqh8XE7uiYTu6krKoU1OfjZM", "id": "PLUo94RsjxOG_w4TLMGd-1wTNeCkuebmql", "snippet": { "publishedAt": "2020-10-02T20:41:08Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Soirée réact", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/SHlyzqfIh3o/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/SHlyzqfIh3o/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/SHlyzqfIh3o/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/SHlyzqfIh3o/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/SHlyzqfIh3o/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Soirée réact", "description": "" } }, "contentDetails": { "itemCount": 7 } }, { "kind": "youtube#playlist", "etag": "jwBimkhzpWwpccSffbqFSgM9Psw", "id": "PLUo94RsjxOG9EaKOdzy7q4D7ran5sA-nF", "snippet": { "publishedAt": "2020-09-14T18:38:11Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "PS4 portable V2", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/cuHzEORnE2g/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/cuHzEORnE2g/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/cuHzEORnE2g/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/cuHzEORnE2g/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/cuHzEORnE2g/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "PS4 portable V2", "description": "" } }, "contentDetails": { "itemCount": 20 } }, { "kind": "youtube#playlist", "etag": "-fEslU_EX5frfIe8EyQnHE3C048", "id": "PLUo94RsjxOG9wv2-UzGNgaEz2szI1_lpX", "snippet": { "publishedAt": "2020-09-11T07:00:11Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "GB Studio", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/omdEc4cE46o/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/omdEc4cE46o/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/omdEc4cE46o/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/omdEc4cE46o/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/omdEc4cE46o/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "GB Studio", "description": "" } }, "contentDetails": { "itemCount": 4 } }, { "kind": "youtube#playlist", "etag": "9M3m8iQCh67ApXf1YppOGDt6YWU", "id": "PLUo94RsjxOG_hgit0aNBiKFqm1E9Di6Xc", "snippet": { "publishedAt": "2020-07-23T04:57:38Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Forza", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/ULrOv_IRvJ0/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/ULrOv_IRvJ0/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/ULrOv_IRvJ0/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/ULrOv_IRvJ0/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/ULrOv_IRvJ0/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Forza", "description": "" } }, "contentDetails": { "itemCount": 1 } }, { "kind": "youtube#playlist", "etag": "NlK9rgt3E4dLiGAne4BJmSRBgGo", "id": "PLUo94RsjxOG8RkLZrgkY3vwZaqLWTuySa", "snippet": { "publishedAt": "2020-07-16T20:18:45Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Warzone", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/dwuM0LNJxOg/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/dwuM0LNJxOg/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/dwuM0LNJxOg/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/dwuM0LNJxOg/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/dwuM0LNJxOg/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Warzone", "description": "" } }, "contentDetails": { "itemCount": 7 } }, { "kind": "youtube#playlist", "etag": "EJ7Sv9i2EXYwkOKv9KWhD99IRQc", "id": "PLUo94RsjxOG9b9DvyewIF-9UwLCQX50mW", "snippet": { "publishedAt": "2020-07-06T20:59:55Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "PS4 portable", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/a_3tiiofN2k/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/a_3tiiofN2k/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/a_3tiiofN2k/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/a_3tiiofN2k/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/a_3tiiofN2k/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "PS4 portable", "description": "" } }, "contentDetails": { "itemCount": 7 } }, { "kind": "youtube#playlist", "etag": "-jinXOHlEOQt_m7Bqy16cPwEJ2s", "id": "PLUo94RsjxOG_9LT_F-L7diQfTdVjROpl-", "snippet": { "publishedAt": "2020-04-20T17:54:00Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "PlayStation 2 Minitel", "description": "Cette playlist contient les rediffusions de mes lives ou j'ai créé une Minitel Playstation 2. Donc mettre une vraie carte mère de PS2 dans un Minitel ainsi que rendre portable le tout avec une batterie tout en essayant de dénaturer le moins possible la forme originelle du Minitel.\n\nBon visionnage :)", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/0DW-fvakHto/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/0DW-fvakHto/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/0DW-fvakHto/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/0DW-fvakHto/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/0DW-fvakHto/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "PlayStation 2 Minitel", "description": "Cette playlist contient les rediffusions de mes lives ou j'ai créé une Minitel Playstation 2. Donc mettre une vraie carte mère de PS2 dans un Minitel ainsi que rendre portable le tout avec une batterie tout en essayant de dénaturer le moins possible la forme originelle du Minitel.\n\nBon visionnage :)" } }, "contentDetails": { "itemCount": 13 } }, { "kind": "youtube#playlist", "etag": "61bs7IQg4EwDv8LFgxBihsmZedo", "id": "PLUo94RsjxOG-pCzntzqcqSC0lFUqjBsHW", "snippet": { "publishedAt": "2020-03-31T10:07:28Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Unity", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/JaBpqoWtUlo/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/JaBpqoWtUlo/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/JaBpqoWtUlo/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/JaBpqoWtUlo/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/JaBpqoWtUlo/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Unity", "description": "" } }, "contentDetails": { "itemCount": 5 } }, { "kind": "youtube#playlist", "etag": "YpXl-nmnBGR1-SIkEmkolQAHS54", "id": "PLUo94RsjxOG83s148IMG7sWiQKKVYw9GO", "snippet": { "publishedAt": "2020-02-10T23:52:42Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Soirée Découverte", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/dPyVmhXx5zI/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/dPyVmhXx5zI/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/dPyVmhXx5zI/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/dPyVmhXx5zI/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/dPyVmhXx5zI/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Soirée Découverte", "description": "" } }, "contentDetails": { "itemCount": 38 } }, { "kind": "youtube#playlist", "etag": "OcFw5SvGmFG-zUhM7RzO3vMQr7o", "id": "PLUo94RsjxOG88Cdl01ZowabzfVfywKx17", "snippet": { "publishedAt": "2019-12-10T09:08:38Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Tutoriel", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/FEMwcdcnsHs/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/FEMwcdcnsHs/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/FEMwcdcnsHs/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/FEMwcdcnsHs/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/FEMwcdcnsHs/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Tutoriel", "description": "" } }, "contentDetails": { "itemCount": 4 } }, { "kind": "youtube#playlist", "etag": "qE17gYd5af5SmRmJuujH5Mr3pdE", "id": "PLUo94RsjxOG_ri33kAW5MrPhgrTGqkSxz", "snippet": { "publishedAt": "2019-09-12T10:46:28Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Vlog", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/X3gJKZnU1Xs/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/X3gJKZnU1Xs/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/X3gJKZnU1Xs/hqdefault.jpg", "width": 480, "height": 360 } }, "channelTitle": "CabriDIY", "localized": { "title": "Vlog", "description": "" } }, "contentDetails": { "itemCount": 2 } }, { "kind": "youtube#playlist", "etag": "flZzgqMsz3FdJ47xYc-4qvK2cRg", "id": "PLUo94RsjxOG_EUKSlM0VRocmsmkD2zYIn", "snippet": { "publishedAt": "2019-03-18T20:53:38Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Rétro Découverte", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/G_tNO82T2Pw/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/G_tNO82T2Pw/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/G_tNO82T2Pw/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/G_tNO82T2Pw/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/G_tNO82T2Pw/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Rétro Découverte", "description": "" } }, "contentDetails": { "itemCount": 25 } }, { "kind": "youtube#playlist", "etag": "cSZsHlZVokco2TEEyp0oKLmYA1o", "id": "PLUo94RsjxOG_MUINY63Xc7bZk5FK5iEfE", "snippet": { "publishedAt": "2018-06-20T18:00:02Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Montage", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/SCUpyAwtFlU/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/SCUpyAwtFlU/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/SCUpyAwtFlU/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/SCUpyAwtFlU/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/SCUpyAwtFlU/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Montage", "description": "" } }, "contentDetails": { "itemCount": 11 } }, { "kind": "youtube#playlist", "etag": "_xRRJyYLN5wYZBoN4zIheQUhdHk", "id": "PLUo94RsjxOG-SvUixo4LLRZX8Fy7xTO2E", "snippet": { "publishedAt": "2018-05-23T17:48:31Z", "channelId": "UCuct2fNB1nSdS8Dl6XzdA1g", "title": "Rediffusion", "description": "", "thumbnails": { "default": { "url": "https://i.ytimg.com/vi/3JiHpX1K-pg/default.jpg", "width": 120, "height": 90 }, "medium": { "url": "https://i.ytimg.com/vi/3JiHpX1K-pg/mqdefault.jpg", "width": 320, "height": 180 }, "high": { "url": "https://i.ytimg.com/vi/3JiHpX1K-pg/hqdefault.jpg", "width": 480, "height": 360 }, "standard": { "url": "https://i.ytimg.com/vi/3JiHpX1K-pg/sddefault.jpg", "width": 640, "height": 480 }, "maxres": { "url": "https://i.ytimg.com/vi/3JiHpX1K-pg/maxresdefault.jpg", "width": 1280, "height": 720 } }, "channelTitle": "CabriDIY", "localized": { "title": "Rediffusion", "description": "" } }, "contentDetails": { "itemCount": 943 } } ] }

[arjitcodes]

Thank you for your feedback!

The additional API calls are necessary because get_channel_playlists_json retrieves playlists manually created by the channel, but it doesn’t capture the distinct 'shorts' or 'long videos' separation we need. To provide a YouTube-like experience with separate tabs for shorts, long videos, and live streams, we differentiate user-uploaded content through specific playlist IDs.

In most cases, channels don't manually create a 'shorts' playlist, as YouTube generates this automatically by using an ID beginning with 'UUSH' (replacing 'UC' in the channel ID) for the 'Shorts' tab. This UUSH ID is the easiest and most consistent way to retrieve a channel's uploaded shorts; however, it’s not included in the get_channel_playlists_json results because its only contain the playlist creates manually by channel. In the CarbiDIY channel example, they manually created a playlist called 'shorts,' which includes all of their short videos. This is an exception rather than the rule, as most channels rely on YouTube’s auto-generated playlists.

user lives playlist and user long uploads playlist work the same way as shorts by adding UULV & UULF replacing UC in the channel ID.

[benoit74]

OK but then, if we run the scraper on CabriDIY channel, we will have no Lives tab, no Videos tab, and no Shorts tab. And we are far from moving close to Youtube UI. And #367 is not solved at all. Same for JamyEpicurieux channel which is the original target of the issue. Or am I missing something?

Do you have a sample channel where we have these "automatically created" playlists? It looks to me they are pretty rare (didn't found on in three channels I tested).

And regarding #366, it looks like your CSS changes are only targeting the player, and they not very aligned with Youtube UI. The ratio of preview pictures in "Shorts" tab is still 4:3 (or something like this) instead being 3:4 (or whatever ratio Youtube is using). And the player still exhibits black areas where it would be way better if it matches the video ratio.

[arjitcodes]

"Automatically created" playlists are not in the result of get_playlist_channel_json thats why you don't have any lives, shorts or any lives tab and that’s the only reason I need to create get_long_uploads_playlist... to check if long videos are available by making a request to /playlist with "UULF" + channel_id[2:]. The response includes all long videos uploaded by the channel . If the response's playlist_items is not None, it means long videos is available, and only then do we add user_long_upload_playlist to playlists.json; otherwise, we don’t add it.

get_user_short_uploads_playlist and get_user_lives_playlist work the same way."

[benoit74]

I think you didn't got me correctly. What I'm saying, is that by executing the code of this PR, I think that I will have a ZIM without any Videos, Live or Shorts tabs, despite your new code and the fact that these are displayed in Youtube online. I didn't tested from end-to-end, I just called your new method on my sample channel and got None, so I assume I won't have a tab. If so, this PR does not yet solve aforementioned issues at all. Do not hesitate to correct me if I'm wrong.

[arjitcodes]

I understand the concern, and I’ve thoroughly tested the code on the CarbiDIY channel and other sample channels, ensuring that the Videos, Live, and Shorts tabs appear correctly. When calling the new methods, they retrieve the expected playlists, so it should work as intended for displaying these tabs offline as well.

It’s possible there could be specific cases or conditions affecting playlist retrieval. I'd be happy to go through any additional channels or scenarios you’re concerned about to make sure the PR is fully robust. do you have any suggestions ?

For CabriDIY, Videos Tab: UULFuct2fNB1nSdS8Dl6XzdA1g Shorts Tab: UUSHuct2fNB1nSdS8Dl6XzdA1g Lives Tab:UULVuct2fNB1nSdS8Dl6XzdA1g all are includes at the end -->

For JamyEpicurieux,

[benoit74]

OK, got it, I've probably simply made a typo when testing your code, sorry for that.

See inline comments for required changes.

Other than that, you need to have properly run invoke lintall (inside scraper folder) to lint your code and invoke checkall to type check your code. This will run in the CI, but this would avoid the CI fails.

And please find a way to not duplicate all tabs and grid components in Vue.JS, I'm pretty sure 90% of the code is identical, and duplicating all these files is only going to be painful to maintain (I can already very hardly tell what the difference is between these tabs and grids codes).

[benoit74]

Why not just setting the properties? Looks like you worried setting them to None? Not a big deal to have few extra props in JSON, and would greatly enhance readability.

[benoit74]

Rather than having your own methods to check playlist exist, I think you should directly rely on get_playlist_json, this would greatly reduce the amount of code needed. Could directly be called from extract_playlists_details_from in fact, we can even slightly change signature of get_playlist_json to return None when playlist is just not found and raise an exception in from_id where we expect the playlist to not be None.

[benoit74]

You can replace this whole block by playlist_ids += filter(None, [user_long_uploads_playlist_id, user_short_uploads_playlist_id, user_lives_playlist_id]) and a comment Add special playlists if they exists

[benoit74]

As mentioned above, user_long_uploads_playlist_id = get_playlist_json("UULF" + main_channel_id[2:]") can do this trick (and so on for other playlists) if you slightly modify get_playlist_json to return None when playlist is not found

[benoit74]

Do we need this? Short tab is anyway not shown when there is not short, isn't it?

[benoit74]

Do we need this? Videos tab is anyway not shown when there is not short, isn't it?

[benoit74]

Do we need this? Lives tab is anyway not shown when there is not short, isn't it?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 95 lines in your changes missing coverage. Please review.

Project coverage is 1.38%. Comparing base (160ec2c) to head (713fc28).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
scraper/src/youtube2zim/youtube.py	0.00%	73 Missing ⚠️
scraper/src/youtube2zim/scraper.py	0.00%	19 Missing ⚠️
scraper/src/youtube2zim/schemas.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##            main    #371      +/-   ##
========================================
- Coverage   1.50%   1.38%   -0.12%     
========================================
  Files         11      11              
  Lines       1132    1227      +95     
  Branches     171     189      +18     
========================================
  Hits          17      17              
- Misses      1115    1210      +95     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.