Word Segmentation Pipeline for SA-class scripts❤️

[Nicolas-Red]

Hello! The text rendering for most languages looks fantastic!
Something we still have to process for Thai text is correct space and break tokenization.

Adding something along the lines of this to UniText internally would be amazing. Thanks!

https://github.com/phanphantz/ThaiTextCare-for-Unity/blob/main/README.md#thaitextnurse

[LightSideMeowshop]

Thanks for the report! Thai word segmentation is on my radar

UniText already has a full UAX#14 line breaking implementation, but Thai (and other SE Asian scripts like Lao, Khmer, Myanmar) needs dictionary-based word segmentation. The Unicode spec itself says so for the SA (Complex Context Dependent) class

Here's what planning:

• A pluggable word segmentation system with a built-in Thai segmenter (Trie + longest match on a dictionary)
• Opt-in via UniText Settings, disabled by default, so projects that don't need it won't pay for it in build size
• Same system will cover other SE Asian scripts later, and you'll be able to plug in your own segmenter if needed

No ETA yet but it's planned for the 1.1.x cycle

[LightSideMeowshop]

Heeeeey!!! The word segmentation system has been implemented and will ship in v2.0.0 (I want to release on this weekend)

What's done

UniText now includes a complete dictionary-based word segmentation pipeline for SA-class scripts (Thai, Lao, Khmer, Myanmar). The system post-processes UAX#14 line break opportunities by injecting Optional breaks at word boundaries, so Thai text now wraps correctly at word boundaries instead of breaking mid-word ,

A pre-built Thai dictionary ScriptableObject (based on the ICU Thai word list, ~26K words) is included. To enable Thai word segmentation in your project, simply drag it into Project Settings > UniText > Word Segmentation > Dictionaries

How it works

The pipeline uses a best-path (maximal matching) algorithm, the same approach as ICU's Thai word breaker. It builds a DAG of all possible dictionary words at each position, then finds the segmentation with the fewest words via dynamic programming. This produces natural, linguistically correct word boundaries

Flexibility

The system is fully data-driven and pluggable:

• Multiple scripts - add dictionaries for Thai, Lao, Khmer, Myanmar (or any SA-class script) side by side. Each dictionary is a standalone ScriptableObject targeting a specific Unicode script
• Custom dictionaries - use the built-in Dictionary Builder (Tools > UniText Tools > Dictionary Builder tab) to create dictionaries from any .txt word list (one word per line, UTF-8). Just drag your word files, pick the target script, and hit Build. You can combine multiple word lists, so if the built-in dictionary is missing domain-specific words (names, slang, technical terms), just add your own .txt file alongside the base one
• Custom segmenters - implement the IWordSegmenter interface for completely custom segmentation logic (e.g. ML-based) and register it via WordSegmentationProcessor.Register()
• Zero overhead when unused - if no dictionaries are assigned in settings, the system early-exits with zero cost. No build size penalty either, dictionary assets are only included if referenced
• No code changes needed - adding support for a new script is purely a data operation: create a dictionary asset and add it to settings

20260306-0504-32.6801853.mp4

[Nicolas-Red]

This is great, thanks! This might need to be a separate topic, but asking it here.
Does UniText handle Korean, Japanese, etc with proper line breaking rules as well?

[LightSideMeowshop]

I’m awesome🥰 Yes! Korean, Japanese, and Chinese line breaking works out of the box with no extra setup

UniText has a full UAX#14 line breaking implementation which natively handles CJK:

• Chinese/Japanese ideographs (class ID) — break opportunities between any two characters
• Japanese kinsoku (禁則処理) - no break before closing punctuation (。、）), no break after opening brackets (（「), small kana (ぁぃぅ) can't start a line
• Korean Hangul - syllable blocks stay intact (Jamo rules LB26), break between syllables
• Mixed CJK + Latin - correct break behavior at script boundaries

Thai was a special case because UAX#14 classifies Thai characters as SA (Complex Context Dependent) meaning "the spec can't tell you where to break, you need a dictionary." That's why we built the WordSegmentationDictionary system specifically for Thai and other SE Asian scripts. CJK doesn't need it - UAX#14 handles everything directly

[LightSideMeowshop]

UniText already handled Korean and Japanese correctly even before the update. Why you ask? Did you face any problems with Korean and Japanese?

[Nicolas-Red]

Not any problems yet! Was just confirming. :)