Keywords: tactile sensing, gelsight, isaaclab, vision-based-tactile-sensor, vbts, reinforcement learning

Note

Preview Release:

The framework is under active development and currently in its beta phase.
If you encounter bugs or have suggestions on how the framework can be improved, please tell us about them (e.g. via Issues/Discussions).

TacEx - Tactile Extension for Isaac Sim/Isaac Lab

TacEx brings Vision-Based Tactile Sensor (VBTS) into Isaac Sim/Lab.

The framework integrates multiple simulation approaches for VBTS's and aims to be modular and extendable. Components can be easily switched out, added and modified.

Currently, only the GelSight Mini is supported, but you can also easily add your own sensor (guide coming soon). We also plan to add more VBTS types later.

Main features:

• GPU accelerated Tactile RGB simulation via Taxim's simulation approach
• Marker Motion Simulation via FOTS
• Integration of UIPC for GPU accelerated incremental potential contact to simulate FEM soft bodies, rigid bodies, cloth, etc. in a penetration-free and robust manner
• Marker Motion Simulation with FEM soft body based on the simulator used by the ManiSkill-ViTac challenge that leverages UIPC

Checkout the website for showcases and the documentation for details, guides and tutorials.

Installation

Note

TacEx currently works with Isaac Sim 4.5 and IsaacLab 2.1.0. The installation was tested on Ubuntu 22.04 with a 4090 GPU and Driver Version 550.163.01 + Cuda 12.4.

0. Make sure that you have git-lfs:

# Need it for the USD assets
git lfs install

1. Clone this repository and its submodules:

git clone --recurse-submodules https://github.com/DH-Ng/TacEx
cd TacEx

Then install TacEx locally or build a Docker Container.

Contributing

Contributions of any kind are, of course, very welcome. Be it suggestions, feedback, bug reports or pull requests.

Let's work together to advance tactile sensing in robotics!!!

Citation

@article{nguyen2024tacexgelsighttactilesimulation,
      title={TacEx: GelSight Tactile Simulation in Isaac Sim -- Combining Soft-Body and Visuotactile Simulators},
      author={Duc Huy Nguyen and Tim Schneider and Guillaume Duret and Alap Kshirsagar and Boris Belousov and Jan Peters},
      year={2024},
      eprint={2411.04776},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2411.04776},
}

Acknowledgements

TacEx is built upon code from

• Isaac Lab
• Taxim
• FOTS
• UIPC
• ManiSkill-ViTac challenge