Research Update Meeting 09/12/2024

[SimoneMic]

Update meeting about the current work in progres and the short-term goals for the current month

Chair:

• @SimoneMic

Attendees:

• @vigisushrutha23
• @andrearosasco
• @enyen
• @lornat75
• @PasMarra

Absent:

• @steb6

---

@andrearosasco

TODAY

• Data collection for ergoCub: Changed server structure on metaCub for RL and Behavior Cloning. Added observations for these modes.
• Added data compression for dataset collection in metaCub: zar is more performing in disk space savings.
• Improved algorithm on how the actions are clustered and how to deal with rotations: the averaging of the rotation was not providing any significant information. So now we are evaluating the distance of the population to the average. It's still important to find the average and @PasMarra proposed a solution which is being working on. For the time being we use keypoints and averaging them wuth a registration algorithm.

OLD MEETING

• Tried to improve the diffusion policy using a methon from the literature, although it works well on the real robot, it has really poor performance in simulation.
• To improve the ICRA work it has been tested a variable number of bins in the Diffusion Policy, using an automatic method based on statistics, however there's no significant improvement (in some tasks it works, in other it degradates the performance)
• Red papers (https://sites.google.com/stanford.edu/sentinel https://arxiv.org/abs/2410.10088 https://arxiv.org/abs/2410.12557 )
• Trajectory Visualizer almost ready, polishing few final things (like color palette). After this is finished, will work on compressing data from the server for loading datasets and figuring (using the visualizer) how the trajectories could be improved.

@steb6

ABSENT
OLD MEETING

• On metaCub, bypassed the issue of the pointcloud bug by having the default mode in RGB only, then it's possible to switch to pintcloud by interacting with the oculus using a gesture.
• Literature review (for Action Recognition) done
• Created a dataloader to interface with open online datasets
• Implemented a paper idea (https://arxiv.org/abs/2311.06218) and fixed some bugs. TODO: replicating the paper results
• Will collect a dataset with @andrearosasco on metaCub to start evaluating Behavior cloning

@vigisushrutha23

TODAY

• ergoCub on Mujoco: figured out where the bug was in the port data formats. But the implementation of the remote-motorcontrolboard-remapper needs to implement some further methods, which are missing in python. Vig will ask Silvio a hand on this and how to figure out the mujoco-yarp integration.
• Probably there is a bug on bimanual which is mixing the encoder values with the references. This could generate the drift error. This issue will be investigated.
• After the mujoco part is done, will work on MPC controller implementation in ergoCub.

OLD MEETING

• No updates on ergoCub-bimanual drift/bug
• Found a bug on interfacing ergocub-bimanual with mujoco, bug related to invalid command received by yarp polydriver from bimanual code. Working on solving it.
• (MPC) After John's improvements, will work in these weeks to evaluate how close to the path can an obstacle be placed. The goal is to see how the controller will handle and replan the path under a wide range of situations.

@PasMarra

TODAY

• Remade Xela setup: will start data collection in these days
• bimanual controller made with grasping baselines: tested in simulation and uses both hands and torso. Will test this in these days on the real robot.
• Verify that the new updates to YARP and firmware are not conflicting with the grasping baselines

OLD MEETING

• Added the Gaze controller (from grasping-baselines) to metaCub for compensating the torso motion. However the controller works in image space, and will not take into account roll orientation changes by the user.
• Redesigned the Force sensor placement for the fingertip sensors data acquisition. Since the sensor is mounted on the gripper base, the momentum generated by the force applied on the EE was saturating the sensor. It was placed then on the sensor support base, reducing this momentum lever by 10 mm, avoiding saturation.
• Working on comparing the grasping-baselines with bimanual-controller: mostly on speed, repeatability, singularity handling (mostly equal), QP solvers (main difference is here)

@enyen

TODAY

• Achieved first result with using a single policy for all manipulators, using DPPO and the new anchor points refinements (for consistent ref frame).
• Working on adding depth and vision for training the policy.

OLD MEETING

• Implemented and trained the automatic hand frame placement (Iterative Palm Frame Alignment) on different manipulators. In this way we have a consistently oriented frame for each end effector.
• Fine-tuning the diffusion policy on PChands with this new frame alignement in place.
• Converting a new online open dataset into a format that could be used for PChands.

@SimoneMic

TODAY

• Implemented LLM querying and tool handling, wrapped in ros2 services.
• Tried ollama with llama31-8b model, but the model was failing in reason tasks. So the code was based on chatgpt4.
• ChatGPT was able to properly call tool handling, and the reason was correct, however was lazy and was not considering all the information available (was cutting the lenght of vector of objects in the environment)

OLD MEETING

• Refactored VLN code to use only ros2 nodes and avoid using VLMaps classes, to further differentiate the code from the original paper.
• Implemented an automatic goal generator for a text query. The generator will cluster the objects in the map and pick the closest cluster to the robot, then finds the closest free cell of the costmap to generate a viable goal.
• Removed embeddings feature fusion, and changed the indexing strategy: instead of using an argmax between the query and "other", I used cosine similarity. With these strategies in place the quality of the indexing has been improved, by removing a lot of false positives.
• Working on how to use an LLM to do geometric reasoning based on queries from the map.