The Environment and Robot Models
PyLIS [15], the 3D simulation environment, was implemented for research in Human-Level Artificial Intelligence (HLAI) and Artificial General Intelligence (AGI), based on the request for research "RFR: 3D Agent Test Suite" [4] from the Whole Brain Architecture Initiative. The software library reads a robot model into a virtual 3D space equipped with a physics engine, acquires color and depth images from a camera on the robot, and detects collisions. This chapter describes its elements.
PyLIS uses PyBullet as the physics engine [1][2][3]. PyBullet is a software framework developed by the Bullet Physics team for applying the laws of classical physics in a virtual 3D space, robot control, sensory information acquisition, and collision detection. It can be easily used with Python, a programming language used for many deep learning software libraries such as Tensorflow [12] and PyTorch [14]. So, we chose PyBullet, which is one of the most promising physics engines to be used with deep learning frameworks [4].
PyLIS supports the following robot models. Other models can be used if they are described in URDF (the robot model format).
HSR is a real robot created by Toyota Research Institute (TRI) and is equipped with wheels, cameras, and an arm. As a model for simulating it in virtual 3D space, URDF model files [5] and 3D mesh files [6] for HSR are also provided by TRI. Fig. 1 shows a screenshot of the HSR model loaded onto PyBullet.
Fig.2: HSR model loaded into PyBullet
The HSR URDF model provided by TRI has problems with PyBullet so that the manipulator does not work and that the robot falls over due to abnormal inertial force acting on the manipulator in some occasions. For PyLIS, the manipulator of another robot model (R2D2) has been transplanted to the HSR model.
R2D2 is a virtual robot model provided by PyBullet [2], which has wheels, a camera, and an arm. PyBullet installation comes with its URDF files.
Fig.3: R2D2 model loaded into PyBullet