From d1b92a02a5f69450825508b2a8b459b8d8aecd92 Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 21:49:36 +0200 Subject: [PATCH 1/4] add reference to launch files --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 9ba95fa3..335dc487 100644 --- a/README.md +++ b/README.md @@ -145,7 +145,7 @@ Additionally to the UR5, the following devices have to be connected and configur 2. Scarlett 2i2 USB Audio Interface (package marimbabot_audio) #### Logitech StreamCam (required for packages marimbabot_vision and marimbabot_speech): -Change the parameter device of the node audio_capture in the launch file +Change the parameter device of the node audio_capture in the [launch file](marimbabot_speech/launch/command_recognition.launch) of the package marimbabot_speech: ```bash marimbabot_speech/launch/command_recognition.launch @@ -159,7 +159,7 @@ marimbabot_vision/config/cv_camera.yaml #### Scarlett 2i2 USB Audio Interface (required for package marimbabot_audio): -Adjust the device parameter for the note_audio_capture node in the launch file: +Adjust the device parameter for the note_audio_capture node in the [launch file](marimbabot_audio/launch/audio_feedback.launch) of the package marimbabot_audio: ```bash marimbabot_audio/launch/audio_feedback.launch From bedafdfcc6733a83e71a055f76fe32ce88b8c4fd Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Thu, 12 Oct 2023 21:50:08 +0200 Subject: [PATCH 2/4] add ref to vision config file --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 335dc487..f0c9b458 100644 --- a/README.md +++ b/README.md @@ -151,7 +151,7 @@ Change the parameter device of the node audio_capture in the [laun marimbabot_speech/launch/command_recognition.launch ``` -and modify the device_id parameter in the configuration file: +and modify the device_id parameter in the [configuration file](marimbabot_vision/config/cv_camera.yaml) of the package marimbabot_vision: ```bash marimbabot_vision/config/cv_camera.yaml From 9dc45e6b92df45587252a00a9d85e215f68766d6 Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Fri, 13 Oct 2023 17:50:50 +0200 Subject: [PATCH 3/4] add information on 3 bars --- marimbabot_vision/scripts/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/marimbabot_vision/scripts/README.md b/marimbabot_vision/scripts/README.md index 341d5123..57f768bf 100644 --- a/marimbabot_vision/scripts/README.md +++ b/marimbabot_vision/scripts/README.md @@ -11,7 +11,7 @@ This script generates a separate dataset including one sample folder for each ca Executes all data generating scripts (generate_data.py, generate_hw_data.py, generate_augmented_data.py) in order to generate a full dataset. The dataset is saved in the `data`, `data_augmented`, `data_hw` and `data_hw_augmented` folders. ### `generate_data.py` -Generates a dataset of images of the random note sheets withing given a note-specific duration restriction (e.g. use a 1/16th note as a minimum duration). The dataset is saved in the `data` folder. +Generates a dataset of images of the random note sheets withing given a note-specific duration restriction (e.g. use a 1/16th note as a minimum duration). The dataset is saved in the `data` folder. In the current configuration, also because of limited computational resources while training, the dataset is generated with 3 bars of music. Arguments: - num_samples: Amount of data to be generated. From 7f4cebef4284a35a0625a3f3964a95c77ae5b84d Mon Sep 17 00:00:00 2001 From: Juphex <20795522+Juphex@users.noreply.github.com> Date: Fri, 13 Oct 2023 19:16:12 +0200 Subject: [PATCH 4/4] add more documentation --- README.md | 5 +++++ marimbabot_msgs/README.md | 3 +++ marimbabot_vision/README.md | 13 +++++++++++++ 3 files changed, 21 insertions(+) create mode 100644 marimbabot_vision/README.md diff --git a/README.md b/README.md index f0c9b458..a433a93b 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,7 @@ source devel/setup.bash Additionally to the UR5, the following devices have to be connected and configured before launching the project: 1. Logitech StreamCam (packages marimbabot_vision and marimbabot_speech) 2. Scarlett 2i2 USB Audio Interface (package marimbabot_audio) +3. Wifi connection to the MalletHolder (package marimbabot_hardware) #### Logitech StreamCam (required for packages marimbabot_vision and marimbabot_speech): Change the parameter device of the node audio_capture in the [launch file](marimbabot_speech/launch/command_recognition.launch) of the package marimbabot_speech: @@ -165,6 +166,10 @@ Adjust the device parameter for the note_audio_capture node in the marimbabot_audio/launch/audio_feedback.launch ``` +#### Wifi connection to the MalletHolder +You should be connected to its Wifi. Please see the [README](marimbabot_hardware/README.md) of the package marimbabot_hardware and [README](marimbabot_bringup/README.md) of the package marimbabot_bringup for further information. + + #### Launch the whole project In order to run the whole project on the real robot, one has to run two launch files. First, the launch file that sets up the robot and its hardware: diff --git a/marimbabot_msgs/README.md b/marimbabot_msgs/README.md index 46ebcb9d..05bdfe81 100644 --- a/marimbabot_msgs/README.md +++ b/marimbabot_msgs/README.md @@ -50,6 +50,7 @@ Have a look at [README](../marimbabot_speech/README.md#5-command-examples) for m ### [CQTStamped.msg](msg/CQTStamped.msg) This message contains information of the Constant-Q Transform(CQT). See [README](../marimbabot_audio/README.md#4-pipeline-of-music-note-detection) for more information. + ### [HitSequence.msg](msg/HitSequence.msg) This message contains an array of HitSequenceElement messages. @@ -58,7 +59,9 @@ This message contains the information single element of a HitSequence message. I ### [NoteOnset.msg](msg/NoteOnset.msg) Used for publish the detect music note. + ### [SequenceMatchResult.msg](msg/SequenceMatchResult.msg) For published the final evaluation of the robot performance + ### [Speech.msg](msg/Speech.msg) For published the transcribed test \ No newline at end of file diff --git a/marimbabot_vision/README.md b/marimbabot_vision/README.md new file mode 100644 index 00000000..bf0d1384 --- /dev/null +++ b/marimbabot_vision/README.md @@ -0,0 +1,13 @@ +# TAMS Master Project 2022/2023 - Vision + +## Scripts +For more information on the usage of the scripts, please refer [README](../marimbabot_vision/scripts/README.md). + +## Src +### [vision_node.py](src/vision_node.py) + +This ROS node is responsible for processing images from a camera source and recognizing notes in the images using a pre-trained model. It converts the image data into a textual representation of recognized musical notes and publishes them as ROS messages. + + +### [visualization_node.py](src/visualization_node.py) +The ROS node receives recognized notes from the vision_node and generates visual representations of the musical notations. It uses the LilyPond library to create musical staff notation and publishes the resulting images as ROS messages for visualization. \ No newline at end of file