Merge branch 'audio_image'

CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2023-03-22
+### Added
+* Functional application with support for TeachableMachine Audio/Image.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Dominik Rzepka
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,7 +1,38 @@
-# Lego AI
+# ![](ai-brick-ic.png) aiBrick
 
-Simple project for controlling Lego Bluetooth Hubs using Artificial Inteligence algorithms on smarthone. 
-* Processing is performed in web browser using [TensorFlow.js](https://www.tensorflow.org/js) and [speech-commands.js](https://github.com/tensorflow/tfjs-models/tree/master/speech-commands) library for speech recognition. 
-* Communication with Lego Bluetooth Hubs uses [WebBluetoothAPI](https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API) to establish Nordic UART connection (code based on [web-device-cli](https://github.com/makerdiary/web-device-cli)
-* Communication processing on Lego Hub was implemented in [Pybrick](https://pybricks.com/)
-
+aiBrick is [browser application](https://repkovsky.github.io/aibrick) which enables using machine learning and artificial intelligence algorithms with Lego Mindstorms/Spike/Technic hubs programmed in [Pybricks](https://pybricks.com/). Processing of camera/microphone input is performed in web browser using [TensorFlow.js](https://www.tensorflow.org/js) and results are sent to hub over [WebBluetoothAPI](https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API). Transmission on the hub's side is handled by `aibrick.py` module. Currently supported AI/ML models are:
+* [TeachableMachine Audio](https://teachablemachine.withgoogle.com/train/audio)
+* [TeachableMachine Image](https://teachablemachine.withgoogle.com/train/image)
+
+## Running TeachableMachine models
+
+### Creating AI/ML model and development of the Python program
+
+1. Teach your AI/ML model on [TeachableMachine](https://teachablemachine.withgoogle.com/). After gathering samples and training, click "Export Model" and in the tab "Tensorflow.js" choose option "Upload (shareable link)" and click "Upload my model". Below you will find the shareable link with you audio/image recognition model, which will be used to configure aiBrick.
+
+2. Install [Pybricks firmware](https://code.pybricks.com/) (v3.2.2 or higher) if you don't have it already on your hub. Download [aibrick.py](aibrick.py) module and upload it into [Pybricks code editor](https://code.pybricks.com/) using option "Import a file".
+
+3. Create new file in Pybricks code editor (next to `aibrick.py`) and develop your code (see example of implementation in [aibrick_brick_classifier.py](aibrick_brick_classifier.py)).
+
+4. Turn on the hub and connect Bluetooth in [Pybricks code editor](https://code.pybricks.com/). Run the program (it is going to be present in hub also when the Pybricks code editor is disconnected).
+
+5. Open the [aiBrick website](https://repkovsky.github.io/aibrick) at the same device in Chrome Browser, where the code editor is running. Press the Bluetooth button in aiBrick, find hub on the list of available devices and connect.
+
+### Running aiBrick with programmed hub
+
+1. Turn on the hub.
+
+2. Open the [aiBrick website](https://repkovsky.github.io/aibrick) in Chrome Browser on smartphone, tablet or PC. Press the Bluetooth button in aiBrick, find hub on the list of available devices and connect.
+
+3. Push the button on hub to start the program.
+
+## How does it work?
+
+Currently supported AI/ML models are based on neural networks - you can find [nice introduction on 3Blue1Brown])https://www.youtube.com/watch?v=aircAruvnKk) YouTube channel. Some nice answers can be also found in [TeachableMachine FAQ](https://teachablemachine.withgoogle.com/faq). Here is short description how aiBrick works:
+* aiBrick web application and Lego hub running Pybricks with `AiBrickteachableMachine` class communicate over Bluetooth using simple named messages.
+* aiBrick web application sends `setup` messages to hub until hub responds with `setup` message with JSON-formatted configuration of AI/ML model to be loaded in aiBrick.
+* aiBrick web application loads requested model downloading it from the provided address and sends `labels` message to hub with list of classes, which can be recognized by the AI model.
+* aiBrick web application starts to process audio/image. Depending on the configuration it will send two types of frames to Lego hub:
+  * `p` (_probability_) frame containing probabilities of each class (after each processing of input chunk), with frequency limited to 10Hz
+  * `d` (_detected_) frame, notyfing about class whose probability just exceeded 0.5, so it is considered to be detected.
+* No audio/image is recorded, stored or send over network - all processing is performed locally, on the device running web application. Network connection is necessary only for downloading AI/ML model from Google storage, after that step you can safely disable network connection at all, if you wish.

aibrick.py

@@ -0,0 +1,109 @@
+from usys import stdin
+from uselect import poll
+
+SEP = chr(183)
+EOF = '\n'
+
+class AiBrick:
+    """Base class for AiBrick communication."""
+    frames = {}
+
+    def __init__(self):
+        self.buffer = ''
+        self.nuart = poll()
+        self.nuart.register(stdin)
+        self._flush()
+
+    def _flush(self):
+        while self.nuart.poll(0):
+            stdin.read(1)
+
+    def _send(self, cmd: str, message: str):
+        """Sends the frame with given command's name and message."""
+        print(cmd, SEP, message, '\n', sep='')
+
+    def _process(self, cmd: str, message: str):
+        """This method should be implemented in the drived class."""
+        ...
+
+    def receive(self):
+        """
+        Listens to Blueooth transmission and pushes the received bytes into 
+        buffer until and of frame (EOF) byte is received. Then calls _process()
+        to interpret contents of the frame and returns frame's name (command).
+        If EOF was not yet received, None is returned.
+        """
+        while self.nuart.poll(0):
+            char = stdin.read(1)  
+            if char == EOF:
+                received, self.buffer = self.buffer, ''
+                if SEP in received:
+                    cmd, message = received.split(SEP, 1)
+                else:
+                    cmd, message = '', received
+                self._process(cmd, message)
+                return self.frames.get(cmd, cmd)
+            else: 
+                self.buffer += char
+        return None
+
+
+class AiBrickTeachableMachine(AiBrick):
+    """
+    Implements communication with aiBrick web application, running TeachableMachine AI/ML model.
+    """
+    frames = {"d": "detection", "p": "probability"}
+
+    def __init__(self, model='', detection=True, probability=False):
+        super().__init__()
+        self.config = {
+            "type": "teachablemachine",
+            "model": model,
+            "detection": detection,
+            "probability": probability
+        }
+        self.labels = []
+        self.detection = ""
+        self.probability = {}
+
+    def _process(self, cmd: str, message: str):
+        """
+        Processes the frame content to respond with setup configuration
+        or assign `labels`, `detection` or `probability` properties.
+        """
+        if cmd == 'setup':
+            self._send('setup', json_dumps(self.config))
+        elif cmd == 'labels':
+            self.labels = message.split('"')[1::2]
+        if cmd == 'd' and self.config['detection']:
+            self.detection = self.labels[int(message)]
+        elif cmd == 'p' and self.config['probability']:
+            probabilities = message.split(',')
+            self.probability = {label: int(p) if p.isdigit() else 0 
+                                for label, p in zip(self.labels, probabilities)}
+        else:
+            # for debug purposes
+            print("cmd", cmd, len(cmd), "message", message)
+
+def json_dumps(dictionary: dict) -> str:
+    """Serializes dict to a JSON formatted str."""
+    def json_format(value):
+        if type(value) is dict:
+            return '{%s}' % ','.join(['"%s":%s' % (str(key), json_format(val))
+                                      for key, val in value.items()])
+        elif type(value) in [list, tuple]:
+            return '[%s]' % ','.join([json_format(elem) for elem in value])
+        elif type(value) in [int, float]:
+            return str(value)
+        elif type(value) is str:
+            for char, escaped in [('\\', '\\'), ('"', '"'), ('\n', 'n'), 
+                                  ('\r', 'r'), ('\t', 't')]:
+                value = value.replace(char, '\\' + escaped)
+            return '"%s"' % value
+        elif type(value) is bool:
+            return str(value).lower()
+        elif value is None:
+            return 'null'
+
+    assert type(dictionary) is dict
+    return json_format(dictionary)

aibrick_brick_classifier.py

@@ -0,0 +1,87 @@
+from pybricks.hubs import PrimeHub
+from pybricks.parameters import Color, Icon
+from pybricks.geometry import Matrix
+from aibrick import AiBrickTeachableMachine
+
+"""
+This is Pybricks code for classification of bricks using aiBrick, 
+TeachableMachine AI/ML model and camera in PC/Laptop/Smartphone. 
+6 classes are recognized:
+* Technic Beam 1 x 3 Thick
+* Technic Beam 1 x 5 Thick
+* Technic Beam 2 x 4 L-Shape Thick
+* Technic Beam 3 x 5 L-Shape Thick
+* Technic Beam 3 x 3 T-Shape Thick
+* No brick
+"""
+
+# define images corresponding to detection of bricks
+BRICK_ICONS = {
+    "3": Matrix([
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0],
+        [0, 1, 1, 1, 0],
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0]
+        ]),
+    "5": Matrix([
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0],
+        [1, 1, 1, 1, 1],
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0]
+        ]),
+    "4L": Matrix([
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 1, 0],
+        [1, 1, 1, 1, 0],
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0]
+        ]),
+    "5L": Matrix([
+        [0, 0, 0, 0, 1],
+        [0, 0, 0, 0, 1],
+        [1, 1, 1, 1, 1],
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0]
+        ]),
+    "T": Matrix([
+        [0, 0, 1, 0, 0],
+        [0, 0, 1, 0, 0],
+        [0, 1, 1, 1, 0],
+        [0, 0, 0, 0, 0],
+        [0, 0, 0, 0, 0]
+        ])
+}
+
+# Initialize the hub.
+hub = PrimeHub()
+
+# Initialize the aiBrick
+MODEL = 'https://teachablemachine.withgoogle.com/models/ztCFxnUmJ/'
+aibrick = AiBrickTeachableMachine(MODEL,            # provide link to TeachableMachine model
+                                  detection=True,   # enable notification on detection
+                                  probability=True) # enable updates about each class' probability
+
+# continuous listening to Bluetooth connection and processing received frames
+while True:
+    received = aibrick.receive()
+    # handle all types of received frames
+    if received == 'setup':
+        # aiBrick app requested for setup
+        hub.light.on(Color.ORANGE)
+    elif received == 'labels':
+        # the setup of aiBrick app is finished
+        hub.light.on(Color.WHITE)
+    elif received in ['detection', 'probability']:
+        # display detected brick shape
+        if aibrick.detection in BRICK_ICONS:
+            hub.display.icon(BRICK_ICONS[aibrick.detection]*100)
+        else:
+            hub.display.icon(Icon.EMPTY)
+        # display probabilities of each class as brightness of last row ('no brick' class probability is not included)
+        col = 0
+        for class_label, class_probability in aibrick.probability.items():
+            if class_label in BRICK_ICONS:
+                hub.display.pixel(4, col, brightness=class_probability)
+                col += 1