Android-blockly-updates (#394)

1. **UI Fixes and Enhancements**:
    - Fixed bugs in webview (alert box, white theme, Google sign-in, profile picture editing).
    - Improved navigation bar in Bluetooth table and edit profile.
    - Added animations and fixed UI bugs in projects, scanner, and model management fragments.
    - Applied app-wide styling and updated the README.md.
    - Updated various screen elements (playground, bottom sheet, main screen, point goal navigation).
    - Added text-to-speech and other Blockly functions.
    - Improved sensor reading display and vehicle control synchronization.

2. **Feature Additions**:
    - Added new functionalities: stop car button, delete functionality, follow object block, multiple object detection, server setup, and autosync.
    - Implemented dynamic speed, object tracking, and autopilot features.
    - Enhanced the playground and run robot screens with new blocks and UI updates.
    - Integrated text-to-speech and new AI model handling blocks.

3. **Bug Fixes**:
    - Fixed issues with sound mode, QR scanner, barcode scanner, and profile edits.
    - Resolved multiple bugs related to Blockly, project fragments, and object tracking.
    - Improved app stability by fixing crash bugs and optimizing code.

4. **Documentation and Code Cleanup**:
    - Added comments and removed unused code.
    - Updated and optimized the README.md and other documentation files.
    - Enhanced code readability and functionality across various fragments and features.

android/robot/README.md

@@ -10,7 +10,7 @@
 
 ## DISCLAIMERS
 
-1. **Safety:** Always make sure you operate in a safe environment. Keep in mind, that your phone could be damaged in a collision! Special care is neccessary when using automated control (e.g. person following or driving policy). Make sure you always have a game controller connected and are familiar with the key mapping so you can stop the vehicle at any time. Use at your own risk!
+1. **Safety:** Always make sure you operate in a safe environment. Keep in mind, that your phone could be damaged in a collision! Special care is necessary when using automated control (e.g. person following or driving policy). Make sure you always have a game controller connected and are familiar with the key mapping, so you can stop the vehicle at any time. Use at your own risk!
 2. **App under development:** The application is under development and may crash or exhibit unexpected behaviour depending on your phone model and version of the operating system. Make sure to test all functionalities with no wheels connected. Use at your own risk!
 
 ## App Screens
@@ -20,17 +20,17 @@
 The app starts with a menu screen that shows all available screens. The settings screen can be opened with a click on the icon at the top right corner. By clicking on the other icons the user can access various screens whose functionalities are explained in the following.
 
 <p align="left">
-<img style="padding-right: 2%;" src="../../docs/images/screen_main.png" alt="Main Menu" width="24.5%"/>
-<img src="../../docs/images/screen_settings.png" alt="Settings Menu" width="24.5%"/>
-<img src="../../docs/images/dialog_stream_mode.png" alt="Settings Menu" width="24.5%"/>
-<img src="../../docs/images/dialog_connectivity_mode.png" alt="Settings Menu" width="24.5%"/>
+<img src="../../docs/images/screen_main.jpg" alt="Main Menu" width="21.6%"/>
+<img src="../../docs/images/screen_settings.jpg" alt="Settings Menu" width="20%"/>
+<img src="../../docs/images/dialog_stream_mode.jpg" alt="Settings Menu" width="20%"/>
+<img src="../../docs/images/dialog_connectivity_mode.jpg" alt="Settings Menu" width="20%"/>
 </p>
 
 ### Settings Menu
 
 #### USB Connection
 
-Tap the USB icon to open the USB options. The drop-down menu is used to set the baud rate. The default is 115200 and you should not need to change this unless you mess with the Arduino firmware. The app will attempt to connect automatically, but in case you encounter issues you can use this switch to disconnect/connect.
+Tap the USB icon to open the USB options. The drop-down menu is used to set the baud rate. The default is 115200, and you should not need to change this unless you mess with the Arduino firmware. The app will attempt to connect automatically, but in case you encounter issues you can use this switch to disconnect/connect.
 
 <p align="left">
 <img src="../../docs/images/usb_disconnected.jpg" alt="Connecting device" width="25%"/>
@@ -133,7 +133,7 @@ Simple UI for tracking objects of 80 different classes. A short description of t
 </p>
 
 - **Dynamic Speed**: reduces the robot speed in "Auto Mode" if it gets closer to the tracked object.
-  The speed is scaled based on the area of the bouding box (works best in landscape orientation).
+  The speed is scaled based on the area of the bounding box (works best in landscape orientation).
 - **Model**: Choose an object detector based on your phone performance (see below for [benchmarking results](#benchmark)).
 - **Object**: Pick the object you want to track. The models can detect the 80 COCO [object classes](https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/).
 - **Confidence**: Confidence threshold to determine if detections are accepted. Increase if you get false detections, decrease if the object of interest it not detected.
@@ -144,7 +144,7 @@ Simple UI for tracking objects of 80 different classes. A short description of t
 
 ### Point Goal Navigation
 
-Note that this fragment requires ARCore and camera permission. If your device does not support ARCore and you continue anyways, the app will crash. In this screen you can specify a goal via a 2D vector with respect to the current position and orientation of the robot. The 2D vector contains the distance to the front and left of the robot in meters. Both values can also be negative and correspond to back and right of the robot in that case. After specifying the goal and pressing `Start` the robot will exectue an AI policy that attempts to reach the goal while avoiding obstacles.
+Note that this fragment requires ARCore and camera permission. If your device does not support ARCore and you continue anyway, the app will crash. In this screen you can specify a goal via a 2D vector with respect to the current position and orientation of the robot. The 2D vector contains the distance to the front and left of the robot in meters. Both values can also be negative and correspond to back and right of the robot in that case. After specifying the goal and pressing `Start` the robot will exectue an AI policy that attempts to reach the goal while avoiding obstacles.
 
 <p align="left">
 <img src="../../docs/images/screen_point_goal_nav.gif" alt="Alt text" width="50%" />
@@ -365,7 +365,7 @@ There are two models that come with the app:
 - **MobileNetV1-300**: This model is used for person following. It uses a SSD object detector with MobileNet V1 backbone. The model is quantized for better performance on embedded devices. It comes with the app.
 - **CIL-Mobile**: This model is used for autonomous navigation. It will predict controls directly from the camera input. Chances are that it will not work in your environment. You should follow our instructions to train your own [Driving Policy](../../policy) and replace it.
 
-Additonal models can be downloaded from the Model Management screen.
+Additional models can be downloaded from the Model Management screen.
 
 The switch on the right is used to turn the network on and off. When the network is running, it produces the controls for the robot and the game controller is disabled. However, you may still use the buttons on the game controller, for example to toggle this switch with the R1 trigger button to regain control of the robot.
 
@@ -379,6 +379,63 @@ Use the drop-down menu to select the device on which the neural network should b
 
 If a model is active, the inference speed in [ms] will be displayed next to the device which is running the model.
 
+### Projects Screen
+
+The Projects Screen displays a list of your OpenBot Playground projects if you are signed in with your Google account. You can execute these projects to connect with your OpenBot, or scan their QR codes by clicking the scanner icon in the top right corner. If you are not signed in, the screen will display a Google Sign-In button, but you can still scan your project's QR code without signing in. If you get the message `Oops, no project found` on the screen after signing in, make sure that the account has projects stored on Google Drive.
+
+If you don't see your latest projects in the project list, you can reload them by pulling down on the project screen.
+<p align="left">
+<img src="../../docs/images/projects_tab_screen.gif" alt="Project Screen" width="24.7%"/>
+<img src="../../docs/images/no_projects_found.jpg" alt="No project screen" width="25%"/>
+<img src="../../docs/images/reload_projects.gif" altq="Reload project screen" width="24.5%"/>
+</p>
+
+- **Google Drive projects**: To run a Google Drive project, tap on the project you want to execute and wait for the contents of the project file to be read. If the file is successfully retrieved without any errors, a pop-up will appear with two buttons: `Start` and `Cancel`. The pop-up will also display the name of the project you are about to run. To execute the project, click on the Start button. If you want to stop the activity, click on the Cancel button. If you receive a pop-up message stating `Something went wrong`, there may be an error with the Google Drive file. To resolve this issue, refresh the project screen by pulling down and then repeating the same process.
+
+
+- **Qr code scanner**: To scan the QR code of a Playground project, click on the QR code icon located in the top right corner of the screen. Grant camera access to the app so that it can scan the QR code. Once the code is scanned, wait for the contents of the file to be read. If the file is retrieved successfully without any errors, a pop-up will appear with two buttons: `Start` and `Cancel`. The pop-up will also display the name of the project you are about to run. To execute the project, click on the Start button. If you want to stop the activity, click on the Cancel button. If you receive a pop-up message stating `Something went wrong`, there may be an error with the Google Drive file. To resolve this issue, generate a new QR code in Playground and repeat the process.
+
+
+- **Executing Project**: If your OpenBot Playground project runs successfully, the screen will display the names of code blocks along with a stop button that can be used to stop the execution of playground block commands.
+
+
+- **Delete Project**: To delete a project, long-press on the project you wish to delete. This will bring up a popup screen asking to confirm the deletion. Tap on 'Yes' to delete the project.
+
+<p align="left">
+<img src="../../docs/images/android_google_drive_projects_execute.gif" alt="Google Drive project execute" width="25%"/>
+<img src="../../docs/images/android_qr_code_scanning.gif" alt="Qr code scanner project execute" width="25%"/>
+<img src="../../docs/images/android_delete_project.jpg" alt="Delete Project" width="24.7%"/>
+</p>
+
+### Profile Screen
+The Profile Screen in the app provides different options based on whether the user is signed in or not.
+If the user is not signed in, a `Google Sign-in` button will appear, prompting the user to sign in their Google account. Once signed in, the user will be able to access their profile and other features.
+If the user is signed in, two buttons will be listed in the  `Profile` tab: `Edit Profile` and `Logout`.
+
+<p align="left">
+<img src="../../docs/images/android_logged_out_profile_screen.jpg" alt="Logged out profile screen" width="25%"/>
+<img src="../../docs/images/android_logged_in_profile_screen.jpg" alt="Logged in profile screen" width="24.9%"/>
+</p>
+
+- **Edit Profile**: Tapping on this button will open a new screen where the user can update their profile information, such as their name and profile picture.
+
+
+- **Logout**: This button allows the user to log out of their account. Tapping on this button will log the user out and return them to the login screen.
+
+  <p align="left">
+  <img src="../../docs/images/android_edit_profile.jpg" alt="Edit profile screen" width="25%"/>
+  <img src="../../docs/images/android_logout_dialog_box.jpg" alt="Logout dialog box" width="24.4%"/>
+  </p>
+
+### OpenBot PlayGround Screen
+
+To access OpenBot Playground services, click on the OpenBot Playground icon located at the top of the screen in the toolbar options. If you want to learn more about OpenBot Playground, [click here](../../open-code/README.md).
+
+<p align="left">
+<img src="../../docs/images/android_playground_sign-in.gif" alt="openBot playground Sign-in" width="25%"/>
+<img src="../../docs/images/android_playground_services.gif" alt="openBot playground Services" width="25%"/>
+</p>
+
 ## Add your own fragment
 
 Please refer to the [ContributionGuide](ContributionGuide.md) to learn how to add your own fragments to the OpenBot app.

android/robot/build.gradle

@@ -1,13 +1,15 @@
 plugins {
     id 'io.github.reactivecircus.app-versioning' version "1.1.2"
+    id 'com.google.gms.google-services'
 }
 
 appVersioning {
     /**
      * Git root directory used for fetching git tags.
      * Use this to explicitly set the git root directory when the root Gradle project is not the git root directory.
      */
-    gitRootDirectory.set(rootProject.file("../")) // if the .git directory is in the root Gradle project's parent directory.
+    gitRootDirectory.set(rootProject.file("../"))
+    // if the .git directory is in the root Gradle project's parent directory.
 }
 
 apply plugin: 'com.android.application'
@@ -51,16 +53,19 @@ android {
         checkReleaseBuilds false
     }
     namespace 'org.openbot'
+    packagingOptions {
+        resources.excludes.add("META-INF/*")
+    }
 }
 
 // Download default models
 project.ext.ASSET_DIR = projectDir.toString() + '/src/main/assets'
-apply from:'download.gradle'
+apply from: 'download.gradle'
 apply plugin: 'kotlin-android'
 apply plugin: 'com.google.gms.google-services'
 
 dependencies {
-    implementation fileTree(dir: 'libs', include: ['*.jar','*.aar'])
+    implementation fileTree(dir: 'libs', include: ['*.jar', '*.aar'])
     implementation 'com.google.android.material:material:1.4.0'
 
     // Build off of stable TensorFlow Lite
@@ -91,6 +96,9 @@ dependencies {
     implementation 'androidx.lifecycle:lifecycle-viewmodel:2.3.1'
     implementation 'androidx.legacy:legacy-support-v4:1.0.0'
     implementation 'androidx.preference:preference:1.1.1'
+    implementation 'com.google.firebase:firebase-auth'
+    implementation 'com.google.firebase:firebase-firestore:24.5.0'
+    implementation 'com.google.firebase:firebase-storage:20.1.0'
 
     def nav_version = "2.3.5"
     // Java language implementation
@@ -143,8 +151,8 @@ dependencies {
     // classpath.
     androidTestImplementation 'androidx.test.espresso:espresso-idling-resource:3.3.0'
 
-    implementation platform('com.google.firebase:firebase-bom:26.4.0')
-
+    implementation platform('com.google.firebase:firebase-bom:31.5.0')
+    implementation 'com.google.android.gms:play-services-auth:20.5.0'
     // Declare the dependencies for the Crashlytics and Analytics libraries
     // When using the BoM, you don't specify versions in Firebase library dependencies
     implementation 'com.google.firebase:firebase-crashlytics'
@@ -166,4 +174,15 @@ dependencies {
     // BLE
     implementation 'com.github.Ficat:EasyBle:v2.0.2'
     implementation 'com.github.Ficat:EasyPermissions:v2.1.0'
+
+    //Scan QR
+    implementation 'com.google.android.gms:play-services-vision:20.1.3'
+    implementation('com.google.api-client:google-api-client-android:1.31.0') {
+        exclude group: 'org.apache.httpcomponents'
+        exclude module: 'guava-jdk5'
+    }
+    implementation('com.google.apis:google-api-services-drive:v3-rev136-1.25.0') {
+        exclude group: 'org.apache.httpcomponents'
+        exclude module: 'guava-jdk5'
+    }
 }

android/robot/google-services.json

@@ -1,34 +1,65 @@
 {
   "project_info": {
-    "project_number": "1062508888733",
-    "project_id": "openbot-public",
-    "storage_bucket": "openbot-public.appspot.com"
+    "project_number": "955078484078",
+    "project_id": "opencode-openbot",
+    "storage_bucket": "opencode-openbot.appspot.com"
   },
   "client": [
     {
       "client_info": {
-        "mobilesdk_app_id": "1:1062508888733:android:0fc8c6a8842e1ff5510a74",
+        "mobilesdk_app_id": "1:955078484078:android:0ed69dbcaad0793a65867f",
         "android_client_info": {
           "package_name": "org.openbot"
         }
       },
       "oauth_client": [
         {
-          "client_id": "1062508888733-jlrr946jdlv59vgpt7h1oa91n9gs0hd2.apps.googleusercontent.com",
+          "client_id": "955078484078-385d4dqq0l8rr1ksge8gbe4q8794b96b.apps.googleusercontent.com",
+          "client_type": 1,
+          "android_info": {
+            "package_name": "org.openbot",
+            "certificate_hash": "b7e97790753c13482e22d0c1ea899ae17b1b4f88"
+          }
+        },
+        {
+          "client_id": "955078484078-610055je567ae4i8p7q75et34e7ssvkm.apps.googleusercontent.com",
+          "client_type": 1,
+          "android_info": {
+            "package_name": "org.openbot",
+            "certificate_hash": "3ee1786f8f6f7fb3a93e593d0c1ccdb6a334cb0d"
+          }
+        },
+        {
+          "client_id": "955078484078-jaoouon63eq00dqkc8l1rdl1qphpfmv2.apps.googleusercontent.com",
+          "client_type": 1,
+          "android_info": {
+            "package_name": "org.openbot",
+            "certificate_hash": "f397f5f4db44bef0e43eb61f529a7532dbf62483"
+          }
+        },
+        {
+          "client_id": "955078484078-kbadvp54dljcpk0g0j5bnf5c17q63ir2.apps.googleusercontent.com",
           "client_type": 3
         }
       ],
       "api_key": [
         {
-          "current_key": "AIzaSyCD59-IMWo1ZKzp-gMoTG6eKIkza_8XVYg"
+          "current_key": "AIzaSyA7AhVQcAsZJtYIrJqcvmlxNlcPgmt5bGw"
         }
       ],
       "services": {
         "appinvite_service": {
           "other_platform_oauth_client": [
             {
-              "client_id": "1062508888733-jlrr946jdlv59vgpt7h1oa91n9gs0hd2.apps.googleusercontent.com",
+              "client_id": "955078484078-kbadvp54dljcpk0g0j5bnf5c17q63ir2.apps.googleusercontent.com",
               "client_type": 3
+            },
+            {
+              "client_id": "955078484078-40uecrlb9cdan3fkckisjlv8uthaj9dd.apps.googleusercontent.com",
+              "client_type": 2,
+              "ios_info": {
+                "bundle_id": "com.emboidedai.openbot"
+              }
             }
           ]
         }

android/robot/src/main/AndroidManifest.xml

@@ -16,7 +16,8 @@
     <uses-permission android:name="android.permission.CHANGE_WIFI_STATE" />
 
     <!-- RTSP Video streaming -->
-    <uses-permission android:name="android.permission.RECORD_AUDIO"/>
+    <uses-permission android:name="android.permission.RECORD_AUDIO" />
+
 
     <uses-feature android:name="android.hardware.camera" android:required="true"/>
     <uses-feature android:name="android.hardware.camera.autofocus" android:required="true"/>
@@ -28,13 +29,11 @@
     <uses-feature android:name="android.hardware.bluetooth_le" android:required="true"/>
     <uses-feature android:name="android.hardware.location.gps" android:required="true"/>
 
-
-
     <application
+        android:name=".OpenBotApplication"
         android:allowBackup="false"
         android:icon="@drawable/ic_openbot"
         android:label="@string/app_name"
-        android:name=".OpenBotApplication"
         android:requestLegacyExternalStorage="true"
         android:roundIcon="@drawable/ic_openbot"
         android:supportsRtl="true"
@@ -69,24 +68,41 @@
 
         </activity>
 
+        <activity
+            android:name=".modelManagement.BackHandlingFilePickerActivity"
+            android:exported="true"
+            android:label="@string/app_name"
+            android:theme="@style/FilePickerTheme">
+            <intent-filter>
+                <action android:name="android.intent.action.GET_CONTENT" />
+                <category android:name="android.intent.category.DEFAULT" />
+            </intent-filter>
+        </activity>
+
         <service
             android:name=".logging.SensorService"
             android:enabled="true"
             android:launchMode="singleTask" />
 
-        <service android:name="net.majorkernelpanic.streaming.rtsp.RtspServer"/>
+        <service android:name="net.majorkernelpanic.streaming.rtsp.RtspServer" />
 
         <provider
             android:name="androidx.core.content.FileProvider"
             android:authorities="${applicationId}.provider"
             android:exported="false"
-            android:grantUriPermissions="true" >
+            android:grantUriPermissions="true">
             <meta-data
                 android:name="android.support.FILE_PROVIDER_PATHS"
                 android:resource="@xml/provider_paths" />
         </provider>
 
-        <meta-data android:name="com.google.ar.core" android:value="optional" />
+        <meta-data
+            android:name="com.google.ar.core"
+            android:value="optional" />
+
+        <meta-data
+            android:name="com.google.android.gms.vision.DEPENDENCIES"
+            android:value="barcode" />
 
     </application>
 

android/robot/src/main/java/org/openbot/OpenBotApplication.java

@@ -26,6 +26,7 @@ public void onCreate() {
     SharedPreferences sharedPreferences = PreferenceManager.getDefaultSharedPreferences(this);
     int baudRate = Integer.parseInt(sharedPreferences.getString("baud_rate", "115200"));
     vehicle = new Vehicle(this, baudRate);
+    vehicle.initBle();
     vehicle.connectUsb();
     vehicle.initBle();
     if (BuildConfig.DEBUG) {