@aignatovich thanks for your PR, let us know when your PR is ready for review 😊

I see there is current Rustfmt issue (cf CI)

Hello @aignatovich,
Thank you for your PR. Your PR seems good to me.
However, I think there is already a normalizer that covers the work of the Russian normalizer: the NonspacingMarkNormalizer, the goal of this normalizer is to remove all the nonspacing marks including the diacritics. The only missing change to activate this normalizer is to add the Cyrilic to the eligible scripts of the should_normalize function.

Let me know if this modification works.

See you!

Hi @ManyTheFish,

Solution that was proposed (adding Cyrilic to the predicate in NonspacingMarkNormalizer) did not reach expected result. The solution this PR proposes also does not impact search behavior as expected.

Conclusions made from debugging Case 2 scenario above: during indexing, normalization of "Ё" -> "e" (yes, that is a latin) takes place, but this does not happen for the input search term, instead "Е" (cyrillic) is normalized into "е" (cyrillic).

I will update as long as I know more.

Merge conflict.

bors merge

Already running a review

This PR was included in a batch that successfully built, but then failed to merge into main. It will not be retried.

Additional information:

{"message":"Changes must be made through a pull request.","documentation_url":"https://docs.github.com/articles/about-protected-branches","status":"422"}

Hi @ManyTheFish ,

There is still work to be done by me in this PR to solve the issue that is being described.

Would you be able to indicate what behavior is preferred in each of those scenarios?

Conclusions made from debugging Case 2 scenario above: during indexing, normalization of "Ё" -> "e" (yes, that is a latin) takes place, but this does not happen for the input search term, instead "Е" (cyrillic) is normalized into "е" (cyrillic).

This inconsistency between normalization of a query and the document, to the best of my understanding, is the cause of the issue.

- Should normalization of Cyrillic -> Latin take place for input query?
OR
- Should normalization of "Ё"(cyrillic) -> "e" (latin) not take place during document indexing?

Hello @aignatovich,

I don't understand why my suggestions in my comments don't work:

Thank you for your PR. Your PR seems good to me.
However, I think there is already a normalizer that covers the work of the Russian normalizer: the NonspacingMarkNormalizer, the goal of this normalizer is to remove all the nonspacing marks including the diacritics. The only missing change to activate this normalizer is to add the Cyrilic to the eligible scripts of the should_normalize function.

As I re-reading everything, I understand that doing normalizer that convert Cyrillic characters close to Latin into Latin characters should work if put after the Lowecase normalizer.

Something like:

static SPOOFING_VARIANTS: Lazy<HashMap<char, char>> = Lazy::new(|| {
    [
        ('е', 'e'),
    ].into_iter().collect()
});

pub struct CyrillicVariantsNormalizer;

impl CharNormalizer for CyrillicVariantsNormalizer {
    fn normalize_char(&self, c: char) -> Option<CharOrStr> {
        match SPOOFING_VARIANTS.get(&c) {
            Some(replacement) => Some(replacement.into()),
            None => Some(c.into()),
        }
    }

    fn should_normalize(&self, token: &Token) -> bool {
        token.script == Script::Cyrillic && token.lemma.chars().any(|c| SPOOFING_VARIANTS.contains_key(&c))
    }
}

Conclusions made from debugging Case 2 scenario above: during indexing, normalization of "Ё" -> "e" (yes, that is a latin) takes place, but this does not happen for the input search term, instead "Е" (cyrillic) is normalized into "е" (Cyrillic).

converting into Latin should be good to me.