and a TF Lite Android example from https://github.com/tensorflow/exam…

…ples/tree/master/lite/examples/image_classification/android

.gitignore

@@ -0,0 +1,13 @@
+*.iml
+.gradle
+/local.properties
+/.idea/libraries
+/.idea/modules.xml
+/.idea/workspace.xml
+.DS_Store
+/build
+/captures
+.externalNativeBuild
+
+/.gradle/
+/.idea/

README.md

@@ -0,0 +1,141 @@
+# TensorFlow Lite image classification Android example application
+
+## Overview
+
+This is an example application for
+[TensorFlow Lite](https://tensorflow.org/lite) on Android. It uses
+[Image classification](https://www.tensorflow.org/lite/models/image_classification/overview)
+to continuously classify whatever it sees from the device's back camera.
+Inference is performed using the TensorFlow Lite Java API. The demo app
+classifies frames in real-time, displaying the top most probable
+classifications. It allows the user to choose between a floating point or
+[quantized](https://www.tensorflow.org/lite/performance/post_training_quantization)
+model, select the thread count, and decide whether to run on CPU, GPU, or via
+[NNAPI](https://developer.android.com/ndk/guides/neuralnetworks).
+
+These instructions walk you through building and running the demo on an Android
+device. For an explanation of the source, see
+[TensorFlow Lite Android image classification example](EXPLORE_THE_CODE.md).
+
+<!-- TODO(b/124116863): Add app screenshot. -->
+
+### Model
+
+We provide 4 models bundled in this App: MobileNetV1 (float), MobileNetV1
+(quantized), EfficientNetLite (float) and EfficientNetLite (quantized).
+Particularly, we chose "mobilenet_v1_1.0_224" and "efficientnet-lite0".
+MobileNets are classical models, while EfficientNets are the latest work. The
+chosen EfficientNet (lite0) has comparable speed with MobileNetV1, and on the
+ImageNet dataset, EfficientNet-lite0 out performs MobileNetV1 by ~4% in terms of
+top-1 accuracy.
+
+For details of the model used, visit
+[Image classification](https://www.tensorflow.org/lite/models/image_classification/overview).
+
+Downloading, extracting, and placing the model in the assets folder is managed
+automatically by download.gradle.
+
+## Requirements
+
+*   Android Studio 3.2 (installed on a Linux, Mac or Windows machine)
+
+*   Android device in
+    [developer mode](https://developer.android.com/studio/debug/dev-options)
+    with USB debugging enabled
+
+*   USB cable (to connect Android device to your computer)
+
+## Build and run
+
+### Step 1. Clone the TensorFlow examples source code
+
+Clone the TensorFlow examples GitHub repository to your computer to get the demo
+application.
+
+```
+git clone https://github.com/tensorflow/examples
+```
+
+Open the TensorFlow source code in Android Studio. To do this, open Android
+Studio and select `Open an existing project`, setting the folder to
+`examples/lite/examples/image_classification/android`
+
+<img src="images/classifydemo_img1.png?raw=true" />
+
+### Step 2. Build the Android Studio project
+
+Select `Build -> Make Project` and check that the project builds successfully.
+You will need Android SDK configured in the settings. You'll need at least SDK
+version 23. The `build.gradle` file will prompt you to download any missing
+libraries.
+
+#### Switch between inference solutions (Task library vs Support Library)
+
+This Image Classification Android reference app demonstrates two implementation
+solutions:
+
+(1)
+[`lib_task_api`](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android/lib_task_api)
+that leverages the out-of-box API from the
+[TensorFlow Lite Task Library](https://www.tensorflow.org/lite/inference_with_metadata/task_library/image_classifier);
+
+(2)
+[`lib_support`](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android/lib_support)
+that creates the custom inference pipleline using the
+[TensorFlow Lite Support Library](https://www.tensorflow.org/lite/inference_with_metadata/lite_support).
+
+The [`build.gradle`](app/build.gradle) inside `app` folder shows how to change
+`flavorDimensions "tfliteInference"` to switch between the two solutions.
+
+Inside **Android Studio**, you can change the build variant to whichever one you
+want to build and run—just go to `Build > Select Build Variant` and select one
+from the drop-down menu. See
+[configure product flavors in Android Studio](https://developer.android.com/studio/build/build-variants#product-flavors)
+for more details.
+
+For gradle CLI, running `./gradlew build` can create APKs under
+`app/build/outputs/apk` for both solutions.
+
+*Note: If you simply want the out-of-box API to run the app, we recommend
+`lib_task_api` for inference. If you want to customize your own models and
+control the detail of inputs and outputs, it might be easier to adapt your model
+inputs and outputs by using `lib_support`.*
+
+The file `download.gradle` directs gradle to download the two models used in the
+example, placing them into `assets`.
+
+<img src="images/classifydemo_img4.png?raw=true" style="width: 40%" />
+
+<img src="images/classifydemo_img2.png?raw=true" style="width: 60%" />
+
+<aside class="note"><b>Note:</b><p>`build.gradle` is configured to use
+TensorFlow Lite's nightly build.</p><p>If you see a build error related to
+compatibility with Tensorflow Lite's Java API (for example, `method X is
+undefined for type Interpreter`), there has likely been a backwards compatible
+change to the API. You will need to run `git pull` in the examples repo to
+obtain a version that is compatible with the nightly build.</p></aside>
+
+### Step 3. Install and run the app
+
+Connect the Android device to the computer and be sure to approve any ADB
+permission prompts that appear on your phone. Select `Run -> Run app.` Select
+the deployment target in the connected devices to the device on which the app
+will be installed. This will install the app on the device.
+
+<img src="images/classifydemo_img5.png?raw=true" style="width: 60%" />
+
+<img src="images/classifydemo_img6.png?raw=true" style="width: 70%" />
+
+<img src="images/classifydemo_img7.png?raw=true" style="width: 40%" />
+
+<img src="images/classifydemo_img8.png?raw=true" style="width: 80%" />
+
+To test the app, open the app called `TFL Classify` on your device. When you run
+the app the first time, the app will request permission to access the camera.
+Re-installing the app may require you to uninstall the previous installations.
+
+## Assets folder
+
+_Do not delete the assets folder content_. If you explicitly deleted the files,
+choose `Build -> Rebuild` to re-download the deleted model files into the assets
+folder.

app/.gitignore

@@ -0,0 +1,3 @@
+/build
+
+/build/

app/build.gradle

@@ -0,0 +1,57 @@
+apply plugin: 'com.android.application'
+
+android {
+    compileSdkVersion 28
+    defaultConfig {
+        applicationId "org.tensorflow.lite.examples.classification"
+        minSdkVersion 21
+        targetSdkVersion 28
+        versionCode 1
+        versionName "1.0"
+
+        testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
+    }
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
+        }
+    }
+    aaptOptions {
+        noCompress "tflite"
+    }
+    compileOptions {
+        sourceCompatibility = '1.8'
+        targetCompatibility = '1.8'
+    }
+    lintOptions {
+        abortOnError false
+    }
+    flavorDimensions "tfliteInference"
+    productFlavors {
+       // The TFLite inference is built using the TFLite Support library.
+       support {
+           dimension "tfliteInference"
+       }
+       // Default: The TFLite inference is built using the TFLite Task library (high-level API).
+       taskApi {
+           getIsDefault().set(true)
+           dimension "tfliteInference"
+       }
+   }
+
+}
+
+dependencies {
+    implementation fileTree(dir: 'libs', include: ['*.jar'])
+    supportImplementation project(":lib_support")
+    taskApiImplementation project(":lib_task_api")
+    implementation 'androidx.appcompat:appcompat:1.0.0'
+    implementation 'androidx.coordinatorlayout:coordinatorlayout:1.0.0'
+    implementation 'com.google.android.material:material:1.0.0'
+
+    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
+    androidTestImplementation 'com.google.truth:truth:1.0.1'
+    androidTestImplementation 'androidx.test:runner:1.2.0'
+    androidTestImplementation 'androidx.test:rules:1.1.0'
+}

app/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

app/src/androidTest/assets/fox-mobilenet_v1_1.0_224_support.txt

@@ -0,0 +1,3 @@
+red_fox 0.79403335
+kit_fox 0.16753247
+grey_fox 0.03619214

app/src/androidTest/assets/fox-mobilenet_v1_1.0_224_task_api.txt

@@ -0,0 +1,3 @@
+red_fox 0.85
+kit_fox 0.13
+grey_fox 0.02

app/src/androidTest/java/AndroidManifest.xml

@@ -0,0 +1,5 @@
+<?xml version="1.0" encoding="utf-8"?>
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="org.tensorflow.lite.examples.classification">
+  <uses-sdk />
+</manifest>

app/src/androidTest/java/org/tensorflow/lite/examples/classification/ClassifierTest.java

@@ -0,0 +1,121 @@
+/*
+ * Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *       http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.tensorflow.lite.examples.classification;
+
+import static com.google.common.truth.Truth.assertThat;
+
+import android.content.res.AssetManager;
+import android.graphics.Bitmap;
+import android.graphics.BitmapFactory;
+import android.util.Log;
+import androidx.test.ext.junit.runners.AndroidJUnit4;
+import androidx.test.platform.app.InstrumentationRegistry;
+import androidx.test.rule.ActivityTestRule;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Scanner;
+import org.junit.Assert;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.tensorflow.lite.examples.classification.tflite.Classifier;
+import org.tensorflow.lite.examples.classification.tflite.Classifier.Device;
+import org.tensorflow.lite.examples.classification.tflite.Classifier.Model;
+import org.tensorflow.lite.examples.classification.tflite.Classifier.Recognition;
+
+/** Golden test for Image Classification Reference app. */
+@RunWith(AndroidJUnit4.class)
+public class ClassifierTest {
+
+  @Rule
+  public ActivityTestRule<ClassifierActivity> rule =
+      new ActivityTestRule<>(ClassifierActivity.class);
+
+  private static final String[] INPUTS = {"fox.jpg"};
+  private static final String[] GOLDEN_OUTPUTS_SUPPORT = {"fox-mobilenet_v1_1.0_224_support.txt"};
+  private static final String[] GOLDEN_OUTPUTS_TASK = {"fox-mobilenet_v1_1.0_224_task_api.txt"};
+
+  @Test
+  public void classificationResultsShouldNotChange() throws IOException {
+    ClassifierActivity activity = rule.getActivity();
+    Classifier classifier = Classifier.create(activity, Model.FLOAT_MOBILENET, Device.CPU, 1);
+    for (int i = 0; i < INPUTS.length; i++) {
+      String imageFileName = INPUTS[i];
+      String goldenOutputFileName;
+      // TODO(b/169379396): investigate the impact of the resize algorithm on accuracy.
+      // This is a temporary workaround to set different golden rest results as the preprocessing
+      // of lib_support and lib_task_api are different. Will merge them once the above TODO is
+      // resolved.
+      if (Classifier.TAG.equals("ClassifierWithSupport")) {
+        goldenOutputFileName = GOLDEN_OUTPUTS_SUPPORT[i];
+      } else {
+        goldenOutputFileName = GOLDEN_OUTPUTS_TASK[i];
+      }
+      Bitmap input = loadImage(imageFileName);
+      List<Recognition> goldenOutput = loadRecognitions(goldenOutputFileName);
+
+      List<Recognition> result = classifier.recognizeImage(input, 0);
+      Iterator<Recognition> goldenOutputIterator = goldenOutput.iterator();
+
+      for (Recognition actual : result) {
+        Assert.assertTrue(goldenOutputIterator.hasNext());
+        Recognition expected = goldenOutputIterator.next();
+        assertThat(actual.getTitle()).isEqualTo(expected.getTitle());
+        assertThat(actual.getConfidence()).isWithin(0.01f).of(expected.getConfidence());
+      }
+    }
+  }
+
+  private static Bitmap loadImage(String fileName) {
+    AssetManager assetManager =
+        InstrumentationRegistry.getInstrumentation().getContext().getAssets();
+    InputStream inputStream = null;
+    try {
+      inputStream = assetManager.open(fileName);
+    } catch (IOException e) {
+      Log.e("Test", "Cannot load image from assets");
+    }
+    return BitmapFactory.decodeStream(inputStream);
+  }
+
+  private static List<Recognition> loadRecognitions(String fileName) {
+    AssetManager assetManager =
+        InstrumentationRegistry.getInstrumentation().getContext().getAssets();
+    InputStream inputStream = null;
+    try {
+      inputStream = assetManager.open(fileName);
+    } catch (IOException e) {
+      Log.e("Test", "Cannot load probability results from assets");
+    }
+    Scanner scanner = new Scanner(inputStream);
+    List<Recognition> result = new ArrayList<>();
+    while (scanner.hasNext()) {
+      String category = scanner.next();
+      category = category.replace('_', ' ');
+      if (!scanner.hasNextFloat()) {
+        break;
+      }
+      float probability = scanner.nextFloat();
+      Recognition recognition = new Recognition(null, category, probability, null);
+      result.add(recognition);
+    }
+    return result;
+  }
+}