Add frequency number handlebars

[Kuuuube]

A simple frequency handlebar is something that I think yomitan has been in desperate need for. The current frequencies handlebar is useless for the vast majority of users. Many users are using default handlebars for everything else but have to muck around with custom handlebars to do frequency sorting.

This adds a handlebar that outputs frequency as an integer number. It combines frequency by taking the harmonic mean of all frequencies. This seems to be generally agreed on as the best way to merge frequencies together. When the same dictionary has multiple frequencies, only the first frequency is used.

Based on https://github.com/MarvNC/JP-Resources?tab=readme-ov-file#freq-handlebar. Modified to be more "general use".

I'm not sure if this is an acceptable way to implement this and there are some potential pitfalls but I think this handlebar (or another implementation of it) is very needed and wanted.

Opening this PR as a draft since I'm not familiar enough with the yomitan codebase to tell for sure if I need to touch anything else to properly implement a new handlebar.

[github-actions]

✔️ No visual differences introduced by this PR.

View Playwright Report (note: open the "playwright-report" artifact)

[toasted-nutbread]

test/utilities/anki.js => getFieldMarkers => add entry to terms and rebuild the test data. npm run test-code-write

[Kuuuube]

test/utilities/anki.js => getFieldMarkers => add entry to terms and rebuild the test data. npm run test-code-write

I've added this. However, there seems to be some issues with the pitch accent rendering in this test unrelated to my PR.

        "pitch-accents": "{pitch-accents-render-error}",
        "pitch-accent-graphs": "{pitch-accent-graphs-render-error}",
        "pitch-accent-positions": "{pitch-accent-positions-render-error}",

I've excluded these issues from my commit.

I bisected this issue to commit 7a0982b.

Removing the catch that writes {${marker}-render-error} in /ext/js/data/anki-note-builder.js throws the following error:

 FAIL  test/dictionary-data.write.js > Write dictionary data expected data
ReferenceError: document is not defined
 ❯ Module.createPronunciationText ext/js/display/sandbox/pronunciation-generator.js:31:23
     29|     const nasalPositionsSet = nasalPositions.length > 0 ? new Set(nasalPositions) : null;
     30|     const devoicePositionsSet = devoicePositions.length > 0 ? new Set(devoicePositions) : null;
     31|     const container = document.createElement('span');
       |                       ^
     32|     container.className = 'pronunciation-text';
     33|     for (let i = 0, ii = morae.length; i < ii; ++i) {

[toasted-nutbread]

Ooops, broke something again. Check #607, if you temporarily apply that change and then run the write file, it will work as expected. Just don't commit changes to that file in this PR.

[toasted-nutbread]

Although I guess you already did basically that. So that aside, feel free to mark the PR as ready for review, IMO it's fine to add afterwards.

[Kuuuube]

Testing #607 it generates exactly the same for anki-note-builder-test-results.json as what I put in that previous commit. Should I revert any changes there or is it okay to leave as is?

[StefanVukovic99]

Does this handle both rank-based and occurence-based frequency dicts?

[Kuuuube]

Does this handle both rank-based and occurence-based frequency dicts?

It does. But using the harmonic mean may not produce the most desirable results in that case. I can add a normal average handlebar as well. So we have something like frequency-harmonic and frequency-average.

[Kuuuube]

I've moved most of the logic to js now. I've added a handlebar for frequency average for users of occurance based frequency dicts. And since occurance based and rank based frequency have opposite bounds for what is considered "nonexistent frequency" I've made a separate handlebar for each there.

[Kuuuube]

Currently this adds 4 handlebars frequency-harmonic-rank,frequency-harmonic-occurance,frequency-average-rank,frequency-average-occurance.

[StefanVukovic99]

Hate to nitpick, but it's occurEnce.

[Kuuuube]

Hate to nitpick, but it's occurEnce.

Oops. All good, important to get things like that right.

[Kuuuube]

Got rid of most of the duplicated code. Having two different functions that do the exact same thing between the term and kanji cards seemed weird too so I merged them together.

Looking at getFrequencyAverage and getFrequencyHarmonic right next to eachother I don't love that they're almost the exact same thing but not sure if I should go and crunch those down into another helper function as well.

[toasted-nutbread]

IMO it's not worth trying to crush it even further. If you want smaller code, just collapse what you have currently.

    const frequencies = getFrequencyNumbers(dictionaryEntry);
    if (frequencies.length < 1) { return -1; }
    let total = 0;
    for (const frequency of frequencies) { total += 1 / frequency; }
    return Math.floor(frequencies.length / total);