Support fuzzy substring search for CJK (Chinese/Japanese/Korean) languages instead of word-based segmentation

[Aruelius]

Hi team,

First, thank you for building Pagefind — it's an excellent static search solution with great performance and low bandwidth usage.

However, the current search experience for languages without word boundaries (especially Chinese and Japanese) is quite limited. Pagefind appears to rely on word segmentation for indexing, which works well for space-separated languages like English, but performs poorly for CJK text.

Example:
Consider the following sentence in a document:
“这是一段简单的测试文本”
Expected behavior (common user expectation in Chinese search):

Searching “一段” → matches
Searching “简单” → matches
Searching “段简” → should match (substring within “一段简单”)
Searching “是一段” → should match

Current behavior:

“一段” and “简单” usually match
“段简” or “是一段” often return no results, even though the characters are clearly present in the text.

This happens because Chinese users typically expect substring / fuzzy containment matching (similar to Python's "keyword" in text or JavaScript's text.includes(keyword)), rather than word-based matching.

A great reference implementation is Fuse.js, which performs character-level fuzzy search by default and works perfectly for Chinese without any language-specific configuration. It reliably returns results for any substring within the text.

The extended version of Pagefind with npx already improves CJK support significantly, but it still falls short of the substring matching that native speakers expect.

It would be amazing if Pagefind could offer an optional fuzzy/character-level indexing mode (perhaps behind a flag like --fuzzy-cjk or automatically for lang="zh" / ja / ko) to better serve users of spaceless languages.

Thank you for considering this improvement — it would make Pagefind much more usable for a huge portion of the web.

[bglw]

Hi! On the list is to improve the handling here on the search side. The main current limitation, as I understand, is that Pagefind segments text when indexing but not when searching.

Browsers now ship with the Intl.Segmenter available, which means Pagefind will be able to segment the search term into words in the browser, which was not possible previously.

Both of these working together should be substantially better. The given sentence "这是一段简单的测试文本" currently indexes (through the extended Pagefind release) as ["这", "是", "一段", "简单", "的", "测试", "文本"], which lines up with what you're seeing regarding each word matching results but the other searches failing to.

Looking at the behaviour of Intl.Segmenter in the browser, “段简” segments into ["段", "简"] and “是一段” segments into [“是", "一段”]. So, in both of these cases, Pagefind would search for those as separate words, rather than for each as a single word which it does now. So both of these cases would also match the same way a whitespace-delimited language sentence would.

Since this improvement is entirely browser-side there isn't any need for a different indexing mode. This won't handle sub-word searching (i.e. searching for "段" won't match "一段") but this is true for all Pagefind languages and isn't going to change. But with the browser-side segmenting the search performance should be substantially better in most cases.

I'll endeavor to ensure this is in the next Pagefind release.

[Aruelius]

Looks great! Thanks for the detailed response and for putting this on the roadmap, really looking forward to the next release!

[bglw]

👋 @Aruelius — just letting you know this is available in the latest Pagefind Beta :)

[Aruelius]

@bglw Thank you for the update and for the notification!
Very grateful to you for considering my request and shipping it in the beta already — means a lot! ❤️