Enhance GROBID Output to Include Image Feature Vectors for Multimodal KG Construction

[tekrajchhetri]

Description

To support multimodal Knowledge Graph (KG) construction, we need to extend the current GROBID-based extraction pipeline to also incorporate image-derived features.

Important: GROBID does not provide native capabilities for image understanding or feature extraction. It focuses on structured text extraction only. Therefore, an additional OCR / vision processing layer is required to handle figures, diagrams, and embedded images.

This update will enable documents to be represented with both:

• Textual features (from GROBID)
• Visual features (from OCR / vision models)

Proposed Approach

Integrate an image understanding component alongside GROBID, potentially using:

• PaddleOCR (text + layout + visual features):
  • https://github.com/PaddlePaddle/PaddleOCR
  • https://www.paddleocr.ai

This could enable:

• Extraction of text from images (OCR)
• Layout-aware features (tables, figures, regions)
• Visual embeddings for figures/diagrams

Goals

• Enable downstream consumers (KG pipeline) to reference both:
  • Textual nodes
  • Visual/multimodal nodes

[satra]

i was playing with paperqa the last few days and it has support for all of this. we should really dig into it before implementing new features with tools.

[tekrajchhetri]

I checked the documentation and it's more for the QA. Based on their documentation, I am little bit unsure if we can result like other extraction methods. Will check again.

[satra]

see the code in this comment: Future-House/paper-qa#1273 (comment)

[tekrajchhetri]

I still see pymupdf , docling + vlm in the backend with extracted text passed to llm. @puja-trivedi could you look into it?