Detecting page breaks in markdown output

[AdityaMannu1709]

Is there any way to detect page breaks in markdown output of a PDF?
For example, in PyMuPDF4LLM library 'n\n-----\n\n' were used in markdown output to detect page breaks.....Can see one of that output below -

[dolfim-ibm]

This is an interesting feature request. Let us put it in the pipeline after the upcoming release.
We are already using comments <!-- --> for tagging images. I think we could easily do the same to signal page breaks. And potentially, having the user specifying the preferred placeholder for it.

[AdityaMannu1709]

yes it would be great then

[AdityaMannu1709]

Just a more adding to this feature request , just like doctags if its possible to identify code tags, table and images captions in markdown also bcz currently they are part of normal markdown text flow.

[laxas]

It would be usefull to have the page number in that "page break comment".

[rmenziejr]

I manually did this by looking at the page elements count and exporting to markdown from the beginning to end of each page element. you could probably implement this more elegantly on the backend but here is example:

from itertools import accumulate
...
result = converter.convert()
page_element_count = [1+len(i.assembled.elements) for i in result.pages]
element_page_cutoff = list(accumulate([1] + page_element_count))
pages = [
        (
            result
            .document
            .export_to_markdown(
                from_element=s,
                to_element=e
            )
        )  
        for s,e in zip(element_page_cutoff[:-1], element_page_cutoff[1:])
]
full_text = print("\n\n<!-- new page -->\n\n".join(pages))

[pat-ben]

Page break is really important indeed!
Is there any visibility to release this feature?

(In PyMuPDF4LLM, the page break looks like this: "n\n-----\n" (2 empty lines, then 5 hyphens, then 1 empty line).

[rmenziejr]

from itertools import accumulate
...
result = converter.convert()
page_element_count = [1+len(i.assembled.elements) for i in result.pages]
element_page_cutoff = list(accumulate([1] + page_element_count))
pages = [
(
result
.document
.export_to_markdown(
from_element=s,
to_element=e
)
)
for s,e in zip(element_page_cutoff[:-1], element_page_cutoff[1:])
]
full_text = print("\n\n\n\n".join(pages))

[Tonaras]

Awesome work, Rmenzie! This works quite well when the input file is a PDF. I couldn't make this work with docx files that have page breaks though. I guess it's a limitation with Docling's current docx parser. Any clues are highly appreciated until the team implements proper page-break detection in an upcoming version (hopefully soon, fingers crossed).

[rmenzie]

Haven’t worked with docx yet but will be shortly so I will let you know if I find the modification

[simjak]

Is this implemented in the markdown exporte or a custom-made solution like above is still needed?
@dolfim-ibm

[789AMAL]

@dolfim-ibm , Please take a look at #762.

[Tonaras]

I refused to believe that this awesome library cannot properly detect page break tags in docx input files. I've spent endless hours to figure out a way but nothing works. This is an absolute necessity for so many use cases and in desperation I'm begging this great team to implement pagination as soon as possible! Ideally, a custom defined "page break"-tag should be inserted in all exporters, not just in markdown.

Warmest regards to all!

[dKC-7000001]

Well to add page breaks in the markdown you can achieve it simply adding a argument of page_break_placeholder while exporting to markdown.

All docling dependency versions

docling==2.28.2
docling-core==2.24.0
docling-ibm-models==3.4.0
docling-parse==4.0.0

from docling.backend.docling_parse_v2_backend import DoclingParseV2DocumentBackend
from docling.datamodel.pipeline_options import PdfPipelineOptions

source = "your_document_path"

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            backend=DoclingParseV2DocumentBackend,  
            # pipeline_options=PdfPipelineOptions(artifacts_path=model_dir)
        )
    }
)
result = converter.convert(source)
markdown_text = result.document.export_to_markdown(page_break_placeholder="<!-- page break -->")
print(markdown_text )

Documentation link :- https://docling-project.github.io/docling/reference/docling_document/#docling_core.types.doc.DoclingDocument.export_to_markdown

[rmenziejr]

This appears to be an updated feature? I do not recall this on the original docs when this thread began but haven't confirmed.

[vagenas]

Page breaks in Markdown export (#309) was added in docling-project/docling-core#213, released with docling-core v2.24.0.

[jai-uparkar]

@vagenas I tried the "page_break_placeholder" feature last week and it was not working:
#1260

[vagenas]

@jai-uparkar, I just provided an answer: #1260 (comment)