implementing hashtags per UAX31 standard

[farooqkz]

closes #8, closes #32

Old

Hello. This is a draft. I am stuck with Rust stuff :)

I need some hints. Like how to pass the sets to hashtag and advice on implementing From<PropertiesError> for CustomError. Furthermore, I've upgraded the rust toolchain to 1.71.0 because icu_properties requires it.

[Simon-Laux]

icu_properties seems big, I would prefer if we could live without it. Like there is a difference between an emoji regex that checks for each emoji and one that checks for unicode ranges instead. Like if code point is in this range instead of comparing with all code points of that range - if possible that would be faster, less code and less binary size.

[farooqkz]

icu_properties seems big, I would prefer if we could live without it. Like there is a difference between an emoji regex that checks for each emoji and one that checks for unicode ranges instead. Like if code point is in this range instead of comparing with all code points of that range - if possible that would be faster, less code and less binary size.

It turns out we have to extract from here:

https://www.unicode.org/Public/14.0.0/ucdxml/

[Simon-Laux]

allowing only a subset and expanding later is better that allowing more and restricting later. It does not have to be 100% perfect on the first try

[farooqkz]

icu_properties seems big, I would prefer if we could live without it. Like there is a difference between an emoji regex that checks for each emoji and one that checks for unicode ranges instead. Like if code point is in this range instead of comparing with all code points of that range - if possible that would be faster, less code and less binary size.

Are you sure it's big? The binary size is only about 100 KB larger.

[farooqkz]

Okay we've got two ways:

• Create our own hashtag content character detector crate which I guess will be about 20-30 KB binary size and use it in this crate.
• Use icu_properties and make it optional. With this option, the hashtag detection will be done exactly as UAX31 has specified. Without it, we consider any non whitespace character a content character but ~100KB less binary size.

[farooqkz]

Okay. My new estimate is that if we implement our own solution, it'll be only about 10KB additional binary size.

[farooqkz]

Okay. My new estimate is that if we implement our own solution, it'll be only about 10KB additional binary size.

Okay I was wrong. See my latest commit. It's about 36KB :)

Also @Simon-Laux, for some reason I can't tell, the tests are not passing :/

[Simon-Laux]

Also @Simon-Laux, for some reason I can't tell, the tests are not passing :/

if you read the clip error it complains being run from a branch on your fork. But also litemap seems to use an unstable feature.

[farooqkz]

Okay. In these final commits:

• I've moved the function for finding range to its own file alongside with the ranges themselves.
• I've used RangeInclusive and its contains method instead of [u32; 2].
• A new enum has been defined as the result of the find range function.

But still the tests for German umlauts fail. Even though the left and right of the assertions seem to be equal:

---- text_to_ast::text_only::german_umlaut_hashtag stdout ----
thread 'text_to_ast::text_only::german_umlaut_hashtag' panicked at 'assertion failed: `(left != right)`
  left: `[Tag("#bücher"), Text(" "), Tag("#Ängste")]`,
 right: `[Tag("#bücher"), Text(" "), Tag("#Ängste")]`', tests/text_to_ast/text_only.rs:114:5

[Simon-Laux]

I still want a unit test that checks all ranges, and more example tests with different "real" use cases. But other than that the code looks good I'd say (didn't test nor benchmark it yet)

[farooqkz]

I still want a unit test that checks all ranges, and more example tests with different "real" use cases. But other than that the code looks good I'd say (didn't test nor benchmark it yet)

So you want test cases for the range function separately? I will add more real use case tests. But I also need hint from you why these tests fail :/

[Simon-Laux]

So you want test cases for the range function separately? I will add more real use case tests. But I also need hint from you why these tests fail :/

yes, you can also test for the german umlaute there: ö, ä, ü and the sharp s: ß

[farooqkz]

@Simon-Laux Ready to merge :D