Image feature extraction.

[NOSCOPEdev]

mmproj model loading and image feature extraction update. You will need to load a vision model and its mmproj file. The settings are in the "LLM.cs" script under the "Advanced Options". You will also need llamalib 1.17 or higher.

[amakropoulos]

Thanks a lot for this PR!!
It needs some work before it is merged, I have left some comments.

[amakropoulos]

This file should be moved in a sample dir inside the Samples~ folder e.g. Samples~/ImageReceiver/ImageReceiver.cs
Also rename to ImageReceiver.cs :)

[amakropoulos]

The same with the AndroidLlava.unity above.
Also rename to Scene.unity similarly to the other samples.

[amakropoulos]

Please use a Text instead of TextMeshProUGUI element.
TextMeshProUGUI requires the TMP assets which vary between different Unity versions.

[amakropoulos]

Copy and paste the SimpleInteraction.cs code and modify it.
This ensures that samples are independent from each other and users can install whichever they want.

[amakropoulos]

I see what you do here, it's better to define a function in Runtime/LLMCharacter.cs that takes over this part and can be reused e.g.:

public async Task<string> ChatWithImage(string query, Uri url, Callback<string> callback = null, EmptyCallback completionCallback = null, bool addToHistory = true)
{
   URLContent urlText  = new URLContent(){url = url.ToString() };
   ImageURLContent urlContent = new ImageURLContent(type="image_url", image_url = urlText)
   TextContent message = new TextContent(){type = "text", text = query};

   string queryWithImage = "[" + JsonUtility.ToJson(message) + "," + JsonUtility.ToJson(urlContent) + "]";
   return await Chat(queryWithImage, callback, completionCallback, addToHistory);
}

public async Task<string> ChatWithImage(string query, Path path, Callback<string> callback = null, EmptyCallback completionCallback = null, bool addToHistory = true)
{
   string queryWithImage = ...
   return await Chat(queryWithImage, callback, completionCallback, addToHistory);
}

and inside the Runtime/LLMInterface.cs

    [Serializable]
    public struct TextContent
    {
        public string type;
        public string text;
    }

    [Serializable]
    public struct ImageURLContent
    {
        public string type;
        public URLContent image_url;
    }

    [Serializable]
    public struct URLContent
    {
        public string url;
    }

[amakropoulos]

Instead of manually defining the "What's in this image?" text, you can use the existing text box in the SimpleInteraction sample.

[amakropoulos]

Instead of copying a new LLM.cs file modify the Runtime/LLM.cs to add the MMPROJmodel.
The MMPROJmodel needs to be treated similar to e.g. the loras, rather than providing this as a text, it needs some additional functionality to load it and make sure it is added inside the builds.
I will take over this part because it is quite involved.