Different ways of splitting document into nodes

[sumitbindra]

Hey @dosu,

What are different ways of splitting a PDF document?

By default for a pdf one node is one page. How can I sections or paragraphs? Are there other better options?

where can find the code for thoese?

Thanks

[dosubot[bot]]

To split a PDF document into sections or paragraphs, you can use several methods available in the LlamaIndex codebase:

1. split_into_paragraphs: This method splits the document into paragraphs based on line breaks using regular expressions.

2. semantic_chunking: This method performs semantic chunking on paragraphs, ensuring chunks do not exceed a specified maximum size and are semantically coherent.

3. SentenceSplitter: A text splitter that divides document texts into smaller chunks while preserving paragraphs/sentences as much as possible.

4. SmartPDFLoader: A fast PDF reader that understands the layout structure of PDFs, such as nested sections, lists, paragraphs, and tables, and chunks PDFs into optimal short contexts for LLMs.

These methods are part of the TopicNodeParser class and the LlamaIndex framework. You can find the code for these methods in the LlamaIndex repository, specifically in the files related to node parsing and PDF reading ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.