Support Unicode in chat & frag

[HenryQuan]

This PR addresses #6 to support Chinese, Japanese and Korean languages in Chat. Now, the font is determined based on the message language.

Two new dependencies are added, langdetect and hanzidentifier. Unfortunately, langdetect can detect Chinese as Korean, so the second library is required.

However, there are several issues with this implementation.

• Huge font sizes (11MB for Chinese, 5MB for Korean and 6MB for Japanese)
• More memory usage (23MB in total)
• Language detection can be inaccurate

[HenryQuan]

I have moved my function to resman instead of putting it under chat. There is also a minor dependency issue when I upgraded to Python 3.11. lxml==4.9.1 doesn't work somehow, I had to use lxml==4.9.4 locally. I didn't commit this change because I don't know if it breaks anything.

There is also one additional fix for this PR. I have supported rendering Chinese player names under frag. This is CN realm specific, so other servers aren't affected by this change.

For Chinese names to be rendered, I have to update battle_controller where I add the encoding under line 742 (12_11_1). I will also open an issue under replays_unpack.

name=player["name"].encode('ISO8859-1').decode('UTF-8'),

[wihn2021]

maybe no need to detect language, just let users add an argument, for example, "python -m render xxxx.wowsreplay --lang chinese", default argument is English. Then renderer loads different font files and ship infomation files

[HenryQuan]

@wihn2021 That is a good idea, but the chat may also contain Japanese or Korean at any moment. The Chinese font doesn't render all Japanese character or any Korean alphabet. Maybe, the flag can be something like --unicode to enable this feature or not. The renderer can determine the realm of the replay which is used to render frags correctly in this PR.

[MeowthTim]

For Chinese names to be rendered, I have to update battle_controller where I add the encoding under line 742 (12_11_1). I will also open an issue under replays_unpack.

name=player["name"].encode('ISO8859-1').decode('UTF-8'),

That is a very useful step for Chinese support. To use especilly for CN server without JP or KR, it just need a little change. The other thing I guess is translating the ship.json to Chinese which I would try to solve with the zh_lang.json in game. Thanks for your job.

[HenryQuan]

@MeowthTim I have the game language file in all supported languages. It is definitely possible to localise even further to support the Chinese server better. It should be available probably inside my custom lang.json.

[JustOneSummer]

when will this be merged into the main line?

[padtrack]

I put off addressing this because I was working on a rewrite that I burned out on. I'll merge this the next chance I have to work on personal projects.

[padtrack]

I am reviewing this on the train. Can you upload some test replays? I had some mixed language replays when I worked on this. The game does not load more than one font, but I think it would be nice to support all CJK characters and more together.

[padtrack]

@wihn2021 That is a good idea, but the chat may also contain Japanese or Korean at any moment. The Chinese font doesn't render all Japanese character or any Korean alphabet. Maybe, the flag can be something like --unicode to enable this feature or not. The renderer can determine the realm of the replay which is used to render frags correctly in this PR.

Do you have example of JP characters the Chinese font doesn't support? I checked my notes from May 2023 and I saw Chinese font supported all random JP phrases I generated.

[HenryQuan]

@padtrack While working on this, I found out that WoWs provided different fonts for Korean, Japanese and Chinese.

I also have a feeling that the Chinese Font supports most Japanese characters, but I am unsure if this is a valid assumption. However, the Japanese Font doesn't support all Chinese characters. If we want to remove a Font, it could be the Japanese one. On the other hand, the Korean Font should be kept.

The upstream has also fixed some encoding issues regarding the clan name in Monstrofil/replays_unpack#30. Maybe, @JustOneSummer dalao can provide some replays. I don't have any at hand now. We can have a training room replay at some time including all three languages.

I had an old Chinese server replay, 20231214_203306_PFSC210-Marseille_46_Estuary.zip, if this helps.

[padtrack]

It's alright, I think I can probably find the old replays I had when I'm home. If you had any on hand it would have been convenient.

Screenshots from when I discussed with a JP speaker:

[padtrack]

@HenryQuan Can you enable Allow edits from maintainers? I also sent a message to what I think is your Discord.