The VisionSample demonstrates the use of several vision-related models, from the ONNX Model Zoo collection, by a Xamarin.Forms app.
The sample enables you to take/pick a photo on the device or use a sample image, if one is provided, to explore the following models.
Classifies the major object in the image into 1,000 pre-defined classes.
Lightweight face detection model designed for edge computing devices providing detection boxes and scores for a given image. This model is included in the repository, but has been updated to remove unused initializers, and to make the initializers constant to enable more runtime optimizations to occur.
The sample also demonstrates how to switch between the default CPU EP (Execution Provider) and platform-specific options. In this case, NNAPI for Android and Core ML for iOS.
Important
There are some known issues that could impact aspects of the sample on specific devices or environments. See known issues section for workarounds.
The VisionSample should build and run as-is, and looks for model files in a folder in this directory called Models. The Ultraface model should already exist in that directory and to able to be used.
Additionally the ResNet model can be added if desired. With Models set as the current directory, you can use wget to download it.
From this directory:
> cd Models
> wget <model_url>
| MODEL | DOWNLOAD URL | Size |
|---|---|---|
| ResNet | https://github.com/onnx/models/raw/f064171f7dd8e962a8a5b34eac8e1bcf83cebbde/vision/classification/resnet/model/resnet50-v2-7.onnx | 97.7 MB |
Note
You may need to reload VisionSample.csproj before newly added model files will appear in Visual Studio Solution Explorer.
If you want to use a prerelease version of ONNX Runtime nuget packages from the integration repository, update the nuget.config
<add key="NuGetOrg" value="https://api.nuget.org/v3/index.json" />
+ <add key="INT NuGetOrg" value="https://apiint.nugettest.org/v3/index.json" />And choose the prerelease version of ONNX Runtime nuget packages
- Microsoft.ML.OnnxRuntime.Managed for VisionSample.csproj
- Microsoft.ML.OnnxRuntime for VisionSample.Forms.Android.csproj and VisionSample.Forms.iOS.csproj
The sample leverages Xamarin.Essetials MediaPicker APIs and Xam.Plugin.Media to handle taking and picking photos in a cross-platform manner. There are several open issues which may impact the ability to use these components on specific devices.
The take and capture photo options are provided as a convenience but are not directly related to the use of ONNX Runtime packages by a Xamarin.Forms app. If you're unable to use those options, you can explore use of the models using the sample image option instead.
MissingMethodException related to ReadOnlySpan<T>
In Visual Studio 2022, Hot Reload loads some additional dependencies including System.Memory and System.Buffers which may cause conflicts with packages such as ONNX Runtime. The workaround is to Disable Hot Reload until the issue has been addressed.