Use of UTF8 in program

[AlanoDel]

Hi!
I have scripts in R2, but they not accept UTF8.
I need UTF8 for some languages, so my question is "is it possible to show and analyse text in UTF8?"
Thanks in advance,
Alain

[Oldes]

Could you provide some example? I am not sure, if I understand your question.

[AlanoDel]

I modified a program written by Allen Kamp.
The script load a file which contains words, in ASCII one per line, then choose randomly a word of length 7 and use the letters of the word to find all words length 3 to 7 which contains these letters.
The script then display, by length, locations for the words which should be devined by the user.
As Rebol 2 was unable to use UTF8 coding I had to change accentd letters by a combination like cx for ĉ.

[AlanoDel]

How to send he script and the data files

[Oldes]

You can use gist: https://gist.github.com/AlanoDel

[Oldes]

If you need support for Rebol2, then I cannot help you. Rebol2 is out of my control. It was not Unicode aware. Cannot you use Rebol3 instead?

[AlanoDel]

Yes I want to use Rebols3 to have access to UTF8

[Oldes]

For text transcoding in Rebol3 there is iconv native:

>> bin: to-binary "ábč"
== #{C3A162C48D} ;= text as utf8

>> iconv/to bin 'utf8 'cp1250
== #{E162E8}

>> iconv/to bin 'utf8 'cp1252
== #{E16263} ;= note that the č is converted to c as there is no č in this codepage!

In case you have a source in some known codepage and want to load it in Rebol3, you can do it like:

>> to string! iconv/to #{E162E8} 'cp1250 'utf8
== "ábč"

where instead of #{E162E8} would be read %some.txt.

Actually it can be just:

>> iconv #{E162E8} 'cp1250
== "ábč"

As the utf8 target is the default codepage.

Supported codepages can be found in the source code.

[Oldes]

Unfortunately I don't have any script, which could guess, which encoding is used in an unknown text.

[Oldes]

There are also ascii? and latin1? natives which you may find useful...

>> latin1? "abí" ;; Returns TRUE if value or string is in Latin-1 character range (below 256). 
== true

>> ascii? "abí" ;; Returns TRUE if value or string is in ASCII character range (below 128). 
== false

But as these requires string, the use may be limited, because conversion from binary to string expects UTF8 by default (if not used iconv).