Further "metadata" OCR scraping with OpenAI maintaining recipe integrity

[mavproductions]

First Check

• I used the GitHub search to find a similar requests and didn't find it.
• Checked the tasks tagged issues and verified my feature is not covered

Please provide a concise description of the problem that would be addressed by this feature.

Talking with @Kuchenpirat on discord about OpenAI compatibility as I want to start using LLM + mealie. Ollama eventually when something is supported

Jump to chase ---> When importing via image, OpenAI sometimes alters text and doesn't include all data. The main write-up at the top of the photo (see below) is completely ignored from the import.

What if the recipe at a feature image? What if I want the original photo saved? The discord link shows the recipe was altered from oregano -> orange. This was a 1 recipe test sample size. does that mean I scan 100 photos/recipes and I might yield 100 errors?

If so - the documentation for OpenAI MUST include tips on users improving accuracy.

My plan is to make mealie an xmas gift for culinary types in my family, that would be devastated by a lack of recipe integrity, or openAI having a "field day" on being "creative" with the wording of the recipe, or shortening things.

I think including necessary extra fields for this type of import metadata is vital for this type of use-case to gain appropriate and meaningful traction in the open-source community. This is also "peace-of-mind" for older folks that have had a system, aren't afraid of technology, but simply want something that is genuine to their original system.

Please provide a concise description of the feature that would resolve your issue.

Maybe adding in some hidden advanced settings that users can toggle to show/hide for openAI prompts, where you have in documentation on prompts that are tested and have yielded results for better metadata structuring and inclusions.

Checkboxes when importing (similar to enter edit mode after import) where users can choose how featured photos are included, does the original recipe photos get stored at the bottom (similar to URL imports), etc.

Please consider and list out some caveats or tradeoffs made in your design decision

Nothing off hand. Just might be confusing for users that want to click import > next > next and be done with it. Things might get chaotic with bulk imports where specific prompts assist specific recipe books or handwritten notes. Again something to add to the documentation on the website as this can perhaps get complex or more advance with the addition of bleeding edge, but much needed understandings of needed features.

I have checked "If this is accepted I'm willing to help maintain this feature". What I mean by this is, I don't mind helping with documentation, once I buy OpenAI and have a chance to crash course everything and add to documentation on the current MkDocs-Material pages. I would just want a core developer to provide appropriate insight to me about expectations and their known bottlenecks of updated systems before I rabbit hole myself into understanding the features for the purpose of not only using, but documenting.

Additional Information

• If this is accepted I'm willing to submit a PR to provide this feature
• If this is accepted I'm willing to help maintain this feature
• I'm willing to sponsor/pay a developer to do this work

[Kuchenpirat]

Ok since this is a bit unstructured i will ad some comments since i was part of the initial discord discussion.
Feature requests that make sense to me:

• Using the availlable recipe description used in the image source if availlable instead of generating one might be sensible.
• Using a photo representation of the finished dish that can often be found in cookbooks instead of the image that was supplied to OpenAI

Some comments on the error thing. We already have language in our prompts to prevent incorrect outputs, but as can not be said often enough. LLMs produce inaccurate information and will gladly make stuff up even when told otherwise. We might have to make it more clear that LLM output might be inaccurate and get tips to improve output (improve image quality ...)
For reference here is the prompt we use to import recipes from images: https://github.com/mealie-recipes/mealie/blob/mealie-next/mealie/services/openai/prompts/recipes/parse-recipe-image.txt

[mavproductions]

Thanks for the additions of info @Kuchenpirat. Yes AI LLMs are infamously inaccurate. I guess my point in all this is maybe eventually trialing and documenting with someone who has time with LLMs to see which OpenAI LLM or open-source LLMs can most accurately work with this project and provide the highest accuracy of results.

The errors made ultimately come down to the end-user's attention to detail, and fixing those errors, but also I'm not sure how batch imports work now, but perhaps how recipes can be piloted into the system then made public to groups/households after proper scanning for errors are made.
(tighter privacy settings ("Only Me") maybe, or special tag = certain privacy setting --> #4214)

[michael-genson]

The main write-up at the top of the photo

Can you include the photo here? Will help with testing

[mavproductions]

https://i.imgur.com/N4KitsX.jpeg

[mavproductions]

Please let me know if a retake is required. I can fix the lighting and create a more flush composition :P

[michael-genson]

Nope that's perfect, thanks!

[mavproductions]

I also forgot to include the discord message link along with the photo in the original post. I guess my morning was a field day in some regards :D

Discord message link here (convo with @Kuchenpirat) and of course you have the photo now.

Signed,
Dimwit -maV. :D