Create recipes for TED by topics

[benoit74]

Rationale

• Automatically create one TED recipe per TED topic

This is a maintenance script. It is not expected to be used on a regular basis but still useful to keep / share.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.84%. Comparing base (818cc1a) to head (9fa9d71).

❗ Current head 9fa9d71 differs from pull request most recent head 557922d. Consider uploading reports for the commit 557922d to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
- Coverage   87.98%   87.84%   -0.14%     
==========================================
  Files          94       93       -1     
  Lines        5327     5307      -20     
==========================================
- Hits         4687     4662      -25     
- Misses        640      645       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rgaudin]

Can you explain what this is for, what will be the Content-Team actions following running this script?

• Will those be individually checked? What's the strategy regarding title/description metadata? Do you want to run and fail all those that will be above 30/80c?
• What about the mul and static list of languages? Do all topics have all those languages? In that order of importance? If not, how will this be acknowledged and fixed by content team?
• With enabled=True we are talking about hundreds of runs that will need to be checked. Alternative is to enabled=False and let Content team run/review at their own pace.
• How many recipes is this?

[benoit74]

Can you explain what this is for, what will be the Content-Team actions following running this script?

This is the script which has been used to generate all TED recipes by topics. The goal is only to not loose it so that it might be reused later on if needed.

See openzim/zim-requests#789 as well for answers to some of your questions.

Will those be individually checked? What's the strategy regarding title/description metadata?

They will be individually checked before being move to "prod" (library.kiwix.org)

Do you want to run and fail all those that will be above 30/80c?

I don't understand this question, sorry

What about the mul and static list of languages? Do all topics have all those languages? In that order of importance? If not, how will this be acknowledged and fixed by content team?

Good point, it reminded me I forgot to create some issues in TED scraper.

This is kinda of hack to retrieve videos in all languages (it is not possible to say "all" in languages, see openzim/ted#171).

Clearly not all topics have those languages, and order is not handled either (the scraper should filter and order them, see openzim/ted#172)

I did think this is going to be acknowledged and fixed by content team, I consider this should be fixed by dev team ; doing it it manually while it is mostly straightforward to automate with code is a bit sad

With enabled=True we are talking about hundreds of runs that will need to be checked. Alternative is to enabled=False and let Content team run/review at their own pace.

This has already ran and allowed to proceed quickly ^^

How many recipes is this?

355 recipes, 355 ZIMs

[rgaudin]

This is the script which has been used to generate all TED recipes by topics. The goal is only to not loose it so that it might be reused later on if needed.

Well if it's to be kept for reference, don't request a review!

Do you want to run and fail all those that will be above 30/80c?

I was wondering if you wanted to check the length of all those title/desc with the script or just let it run and have the scraper fail if it didn't fit. I understand from the last answer that all recipes created their ZIM so all metadata did fit.

the scraper should filter and order them, see openzim/ted#172

👍 I think this should be fixed before moving to prod if we are sending this to Languages metadata (readers trust this)