You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CHANGES.md
+15Lines changed: 15 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,19 @@
1
1
# Change Log
2
2
3
+
## Changes in version 0.2.5
4
+
5
+
### Fixes:
6
+
7
+
*[341](https://github.com/pymupdf/RAG/issues/341) - Broken markdown parsing for new line directly followed by 'o'...
8
+
9
+
### Other Changes:
10
+
11
+
* New parameter `table_format` in method `to_text()` (PyMuPDF-Layout only). This allows selecting the appearance of tables in plain text outputs. The possible values are defined in the list `tabulate.tabulate_formats`. Default is "grid".
12
+
* Installaing PyMuPDF4LLM now supports including all optional dependencies in the `pip` command: `pip install --update pymupdf4llm[ocr,layout]`. This will install pymupdf4llm, pymupdf, and pymupdf-layout. The "ocr" parameter - when needed - installs opencv-python for automatic OCR support in PyMuPDF-Layout mode. Combine this with parameters `--update`, `--force-reinstall` or `--no-cache-dir` as necessary.
13
+
* Major rework of the heuristics that determine whether a page should be OCR'd.
# Using PyMuPDF as Data Feeder in LLM / RAG Applications
1
+
# Using PyMuPDF as a Data Feeder in LLM / RAG Applications
2
2
3
3
This package converts the pages of a PDF to text in Markdown format using [PyMuPDF](https://pypi.org/project/PyMuPDF/).
4
4
@@ -8,42 +8,105 @@ Header lines are identified via the font size and appropriately prefixed with on
8
8
9
9
Bold, italic, mono-spaced text and code blocks are detected and formatted accordingly. Similar applies to ordered and unordered lists.
10
10
11
-
By default, all document pages are processed. If desired, a subset of pages can be specified by providing a list of 0-based page numbers.
11
+
By default, all document pages are processed. If desired, a subset of pages can be specified by providing a sequence of 0-based page numbers.
12
12
13
+
-----
14
+
15
+
[PyMuPDF-Layout](https://pypi.org/project/pymupdf-layout/) is an optional extension of PyMuPDF. It offers AI-based improved page layout analysis, for instance entailing a much higher table recognition.
16
+
17
+
Since version 0.2.0, pymupdf4llm fully supports pymupdf-layout. As part of this, output as plain text or a JSON string is also possible. In addition, every page is automatically OCR'd (based on a number of criteria) provided package [opencv-python](https://pypi.org/project/opencv-python/) is installed and Tesseract is available on the platform.
18
+
19
+
Layout mode is activated with a simple modification of the import statements - for details, please see below.
13
20
14
21
# Installation
15
22
16
23
```bash
17
24
$ pip install -U pymupdf4llm
18
25
```
19
26
20
-
> This command will automatically install [PyMuPDF](https://github.com/pymupdf/PyMuPDF) if required.
27
+
> This command will automatically install or upgrade [PyMuPDF](https://github.com/pymupdf/PyMuPDF) as required.
28
+
29
+
To install all Python packages for full support of the layout feature and automatic OCR, you can use the following command version:
30
+
31
+
```bash
32
+
$ pip install -U pymupdf4llm[ocr,layout]
33
+
```
34
+
35
+
This will install opencv-python and pymupdf-layout in addition to pymupdf4llm and pymupdf.
36
+
37
+
# Execution
38
+
## Legacy Mode
39
+
For **_standard (legacy) markdown extraction_**, use the following simple script
40
+
41
+
```python
42
+
import pymupdf4llm
43
+
44
+
md_text = pymupdf4llm.to_markdown("input.pdf")
45
+
46
+
# now work with the markdown text, e.g. store as a UTF8-encoded file
Instead of the filename string as above, one can also provide a PyMuPDF `Document`.
21
52
22
-
Then in your script do:
53
+
By default, all pages in the PDF will be processed. If desired, the parameter `pages=<sequence>` can be used to provide a sequence of zero-based page numbers to consider.
54
+
55
+
## Layout Mode
56
+
To **_activate layout mode_**, use the following
23
57
24
58
```python
59
+
import pymupdf.layout # activate PyMuPDF-Layout in pymupdf
25
60
import pymupdf4llm
26
61
62
+
# The remainder of the script is unchanged
27
63
md_text = pymupdf4llm.to_markdown("input.pdf")
28
64
29
65
# now work with the markdown text, e.g. store as a UTF8-encoded file
Instead of the filename string as above, one can also provide a PyMuPDF `Document`. By default, all pages in the PDF will be processed. If desired, the parameter `pages=[...]` can be used to provide a list of zero-based page numbers to consider.
70
+
Here are the JSON and plain text output versions.
71
+
72
+
### JSON
73
+
74
+
```python
75
+
import pymupdf.layout # activate PyMuPDF-Layout in pymupdf
76
+
import pymupdf4llm
77
+
78
+
json_text = pymupdf4llm.to_json("input.pdf")
79
+
80
+
# now work with the markdown text, e.g. store as a UTF8-encoded file
81
+
import pathlib
82
+
pathlib.Path("output.json").write_text(json_text)
83
+
```
84
+
85
+
### Plain Text
86
+
87
+
```python
88
+
import pymupdf.layout # activate PyMuPDF-Layout in pymupdf
89
+
import pymupdf4llm
90
+
91
+
plain_text = pymupdf4llm.to_text("input.pdf")
92
+
93
+
# now work with the markdown text, e.g. store as a UTF8-encoded file
* Support for pages with **_multiple text columns_**.
39
102
* Support for **_image and vector graphics extraction_**:
40
103
41
-
1. Specify `pymupdf4llm.to_markdown("input.pdf", write_images=True)`. Default is `False`.
42
-
2.Each image or vector graphic on the page will be extracted and stored as an image named `"input.pdf-pno-index.extension"` in a folder of your choice. The image `extension` can be chosen to represent a PyMuPDF-supported image format (for instance "png" or "jpg"), `pno` is the 0-based page number and `index` is some sequence number.
43
-
3. The image files will have width and height equal to the values on the page. The desired resolution can be chosen via parameter `dpi` (default: `dpi=150`).
44
-
4. Any text contained in the images or graphics will be extracted and **also become visible as part of the generated image**. This behavior can be changed via `force_text=False` (text only apears as part of the image).
104
+
1. Specify either `write_images=True` or `embed_images=True`. Default is `False`.
105
+
2.Images and vector graphics on the page will be stored as images named `"input.pdf-pno-index.extension"` in a folder of your choice or be embedded in the markdown text as base64-encoded strings. The image `extension` can be chosen to represent a PyMuPDF-supported image format (for instance "png" or "jpg"), `pno` is the 0-based page number and `index` is some sequence number.
106
+
3. The image files will have width and height equal to the values on the page. The desired resolution can be chosen via parameter `dpi` (default: `dpi=150`). So this is not an actual **_extraction_** but rather rendering of the respective page area.
107
+
4. Any standard text written in image areas will become a visible part of the generated image and otherwise be ignored. This behavior can be changed via `force_text=True` which causes the text to also become part of the output.
45
108
46
-
* Support for **page chunks**: Instead of returning one large string for the whole document, a list of dictionaries can be generated: one for each page. Specify `data = pymupdf4llm.to_markdown("input.pdf", page_chunks=True)`. Then, for instance the first item, `data[0]` will contain a dictionary for the first page with the text and some metadata.
109
+
* Support for **page chunks**: Instead of returning one large string for the whole document, a list of dictionaries can be generated: one for each page. Specify `data = pymupdf4llm.to_markdown("input.pdf", page_chunks=True)`. Then, for instance the first item, `data[0]` will contain a dictionary for the first page with its text and some metadata.
47
110
48
111
* As a first example for directly supporting LLM / RAG consumers, this version can output **LlamaIndex documents**:
49
112
@@ -57,6 +120,7 @@ Instead of the filename string as above, one can also provide a PyMuPDF `Documen
57
120
# Every list item contains metadata and the markdown text of 1 page.
58
121
```
59
122
60
-
* A LlamaIndex document essentially corresponds to Python dictionary, where the markdown text of the page is one of the dictionary values. For instance the text of the first page is the the value of `data[0].to_dict().["text"]`.
123
+
* A LlamaIndex document essentially corresponds to Python dictionary, where the markdown text of the page is one of the dictionary values. For instance the text of the first page is the value of `data[0].to_dict().["text"]`.
61
124
* For details, please consult LlamaIndex documentation.
62
-
* Upon creation of the `LlamaMarkdownReader`all necessary LlamaIndex-related imports are executed. Required related package installations must have been done independently and will not be checked during pymupdf4llm installation.
125
+
* Upon creation of the `LlamaMarkdownReader`all necessary LlamaIndex-related imports are executed. Required related package installations must have been done independently and will not be checked during pymupdf4llm installation.
0 commit comments