A digital twin of the UR5 robot that replicates its movements in real-time, enabling users to both monitor and control the robot remotely.
I contributed to this project as part of the Horizon EU-funded initiative, IntellIoT. This use case was developed in collaboration with Siemens and several other companies and institutions. The application enables real-time remote control with a Stylus of a UR5 robot within an Augmented Reality (AR) environment using Microsoft's Hololens 2 AR glasses, where users can view the robot's camera streams directly within their space.
When I started this project, I was completely new to the field of Robotics and figuring out how to set this up took me a long time but was a fun learning journey. That's why I developed a simplified desktop version of the application without the camera streams, including links that allow anyone with a Windows machine to control a simulated UR5 robot.
My motivation for this project is to assist anyone trying to integrate Robotics within Unity and help them with the learning curve that comes with it. That is why I will include a thourough documentation within this readme.
If you still require any help getting this project to work, please contact me on my socials:
This project contains a 3D environment with a UR5 robot model. It allows the user to monitor and control the UR5 robot accurately in real-time by providing the IP address of the robot's controller.
The robot is a Unified Robotics Description Format (URDF) model that was created using the ROS framework within a Ubuntu environment. I added the model to Unity using the URDF-Importer Package. I then used Unity Robotics Hub Package and attached the necessary scripts to control the robots movement by assigning values to the different robot joints. Once this was setup, I linked the digital and real robots using a bilateral TCP/IP interface. I extract the information I need (robot angles) from the updates received from the connection and apply them to the digital robot. The same interface is also used to send commands to the robot in the form of program snippets written using URScript programming language.
- URDF file of the UR5 model with a Robotiq gripper.
- Real-time monitoring of the real robot's movement in simulated environment.
- Capabililty to control the real robot from the application through digital one using a keyboard. (Controls below)
- Download and extract the Application Build.
- Download and extract URSim 3.15 VMWare Image.
- Download and install VMWare Player 17.5
- Run VMWare Player
- Add URSIM's image (.vmx) to VMWare
- Click on Edit virtual machine settings
- Configure the VM specifications as shown
- Memory: 4 GB
- Processors: 4 vCPUs
- Disk: 10 GB
- Network: Bridged (Automatic)
- Click on Play virtual machine
- Install VMWare Tools (A pop-up will appear when you run it for the first time)
- Run UXTerm
- Type ifconfig to get the IP of the machine
- Run UR5 Sim and make sure the robot is running
- Within the UR5 Sim home page, click on Program Robot
- Select the Move tab
- Run the Unity Application
- Type in the IP address from earlier
- Press "Connect"
- Use the robot interface from the move tab with UR5 Sim to see the Digital Robot mimic its movement. Or, use the controls mentioned below to move the UR5 robot using the digital one.
- Right Arrow: X+
- Left Arrow: X-
- Up Arrow: Y+
- Down Arrow: Y-
- K: Z-
- L: Z+