[Question] QuadCopter training with camera

[JulienHansen]

Hi,

Apologies if this question has been asked before or if the answer is obvious. I’m still relatively new to reinforcement learning, robotics, and simulation, having started just a month ago.

I’m currently working on a reinforcement learning task for a drone that needs to navigate through a level to reach a desired position on the opposite side. To achieve this, I’ve added a camera to the drone for perception.

My question is: how should I approach training for such an application? In your opinion, what would be the fastest and most effective way to implement this?

Would it be better to:

1. Build my own library and custom wrappers?
2. Modify an existing library like RSL-RL?
3. Consider another approach entirely?

Thanks in advance for your responses

[shaoxiang]

I also really want to know that currently the quadcopter example only has a direct version, not a manage_based version, and I don't know if it's possible to implement a manage_based version because it doesn't seem to be possible to implement it through ActionsCfg's EffortActionCfg, etc. Therefore, I personally think that it may be difficult to implement a ready-made visual quadcopter.

[RandomOakForest]

This is a great discussion. Will move it into our Discussions section for the team to follow upon. Thanks for posting this.

[Toni-SM]

@JulienHansen in order to provide an appropriate solution there is necessary to know in details what is the expected observation and action space you want to use

[JulienHansen]

For the moment i'm using a Dict, here is my get_observation function in my application :

def _get_observations(self) -> dict:
    cam_images = self.scene["camera"].data.output["rgb"][..., :3]
    desired_pos_b, _ = subtract_frame_transforms(
        self._robot.data.root_state_w[:, :3], self._robot.data.root_state_w[:, 3:7], self._desired_pos_w
    )

    state = torch.cat([
        self._robot.data.root_lin_vel_b,        
        self._robot.data.root_ang_vel_b,        
        self._robot.data.projected_gravity_b,   
        desired_pos_b,
    ], dim=-1)

    return {
        "policy": state,
        "Camera": cam_images
    }

If i remember correctly it is explain in the doc that this function needs to return the observations under the "policy" key, so to extend this function for my camera purpose i extended the dictionnary and i was thinking about modifying the rsl_rl wrapper
and actor_critic file as explained above to gain time and not having to do to much code.
I hope i'm answering your reply and not forgetting or misunderstanding your demand has english is not my first language.

[Toni-SM]

In the Direct workflow the observation space value (fundamental or composite) needs, and it can, to be defined under the policy key. E.g.:

observation = {
    "policy": {
        "robot-state": state,
        "camera": camera,
    }
}

Now, the second part, the RL library.
Currently, only skrl (version 1.4.0: develop branch, not released yet) support learning using any space (fundamental or composite). See https://skrl.readthedocs.io/en/develop/api/utils/model_instantiators.html#inputs

You can use it... or modify the rsl_rl or rl_games Isaac Lab wrappers and libraries source code to support it.

Do you have any initial network structure? If so, I can help you defining it in the skrl's agent config file.

[JulienHansen]

Ohhhhh okay i get it now i didn't know we could implement composite observation space like that it's awesome !
Regarding the initial network structure except if you are referring to the skrl_config file like this one :

IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/locomotion/velocity/config/go1/agents/skrl_flat_ppo_cfg.yaml

Line 4 in d603b67

# Models are instantiated using skrl's model instantiator utility

i don't have one and i could clearly benefit from you help. 🙂

[Toni-SM]

Well, I asked for an initial network structure to get an idea of a possible skrl configuration definition.
Again, if you are going to use skrl you need to use the version 1.4.0 (not released yet): develop branch.

./isaaclab.sh -p -m pip install git+https://github.com/Toni-SM/skrl.git@develop

---

Here is an example:

• left network:

  network:
    - name: features_extractor
      input: permute(STATES["camera"], (0, 3, 1, 2))  # PyTorch NHWC -> NCHW
      layers:
        - conv2d: {out_channels: 32, kernel_size: 8, stride: 4, padding: 0}
        - conv2d: {out_channels: 64, kernel_size: 4, stride: 2, padding: 0}
        - conv2d: {out_channels: 64, kernel_size: 3, stride: 1, padding: 0}
        - flatten
      activations: relu
    - name: net_state
      input: STATES["robot-state"]
      layers: [128, 128, 64]
      activations: elu
    - name: net
      input: concatenate([features_extractor, net_state])
      layers: [512, 256, 128]
      activations: elu
  output: ACTIONS

• right network:

  network:
    - name: features_extractor
      input: permute(STATES["camera"], (0, 3, 1, 2))  # PyTorch NHWC -> NCHW
      layers:
        - conv2d: {out_channels: 32, kernel_size: 8, stride: 4, padding: 0}
        - conv2d: {out_channels: 64, kernel_size: 4, stride: 2, padding: 0}
        - conv2d: {out_channels: 64, kernel_size: 3, stride: 1, padding: 0}
        - flatten
      activations: relu
    - name: net
      input: concatenate([features_extractor, STATES["robot-state"]])
      layers: [512, 256, 128]
      activations: elu
  output: ACTIONS

Refer to https://skrl.readthedocs.io/en/develop/api/utils/model_instantiators.html#inputs for more details about model definitions in skrl using Isaac Lab .yaml configs

Also, see the skrl skrl_camera_ppo_cfg.yaml file for the Isaac-Cartpole-RGB-Camera-Direct-v0 example (CNN)

[JulienHansen]

Thank you so much it much more clear now how i can trainmy drone !