Exclude not working for apostrofs

[drummerroma]

w = "L’Unesco a perdu tout crédit et Donald Trump en tire les conséquences"

tokeniser = WordsCounted::Tokeniser.new(w).tokenise(exclude: "’")

[ [ 0] "l", [ 1] "unesco", [ 2] "a", [ 3] "perdu", [ 4] "tout", [ 5] "crédit", [ 6] "et", [ 7] "donald", [ 8] "trump", [ 9] "en", [10] "tire", [11] "les", [12] "conséquences"

[abitdodgy]

@drummerroma interesting. Are you sure it's the same character in the text and exclude clause?

[drummerroma]

I used cut and paste in the rails console. So I am sure.
I have a french (or italian) text pasted from Word (or any other processor) and would like to have treated apostrofated frases counted as single token. Example:
l'appartamento
l'Unesco
l’abandon
Maybe there is a difference between "'" and "’"?

[abitdodgy]

@drummerroma yes, there is. These are different characters. The apostrophe, code is ’, while the single quote is '. The trouble is, many people don't use the right chars. For example, they use double quotes " instead “ and ”, the proper quoting chars. For this type of use case, I would exclude both because you can never be certain what the originally typed text contains, or whether it was transformed by an editor.

[drummerroma]

Yes, I agree. But the exclude instruction should work anyway.

[abitdodgy]

I just confirmed that this doesn't exclude the apostrophe but does exclude the single quote. Strange. I'll have to investigate why.