Outdate/broken Sentence tokenizer?

[parhammmm]

Came across an issue earlier, that highlighted the way the tokenizer works. I'm referring to the try/catch here

LlamaIndexTS/packages/core/src/node-parser/utils.ts

Lines 36 to 48 in 6f4549b

	export const splitBySentenceTokenizer = (): TextSplitterFn => {
	if (!sentenceTokenizer) {
	sentenceTokenizer = new SentenceTokenizerNew();
	}
	const tokenizer = sentenceTokenizer;
	return (text: string) => {
	try {
	return tokenizer.tokenize(text);
	} catch {
	return [text];
	}
	};
	};

I noticed natural is being shimmed in but it's the older version which is buggy and making the try/catch necessary. The new Sentence Splitter works quite well and doesn't have the same issues that would cause the tokenizer to fail.

Using a deprecated tokenizer and hiding the errors from it is quite messy which isn't ideal.

natural is CommonJS only and so that makes sense as why it's being shimmed in but could we update to the latest natural and do something like the following inside splitBySentenceTokenizer while we wait for NaturalNode/natural#744? would that break something in the packaging in LLamaindexTS @himself65 ?

  const {
    default: { SentenceTokenizer },
  } = await import("natural");

We can still try catch on the function but at least make it not silent... console.log at the minimum?

[petergoldstein]

@himself65 Any thoughts on this? Would there be support for a PR that addresses updating the natural version?

[himself65]

I used tsup to bundle the natural package, yeah let me do a upgrade later

[parhammmm]

@himself65 thanks!  🙏