Remotely Detectable Robot Policy Watermarking

This is the official repository for Remotely Detectable Robot Policy Watermarking.

Requirements: This repository requires Python 3.10.12 and all dependencies listed in requirements.txt.

💧 1. Watermarking a trained policy

This repository already contains pre-trained policies for the environments used in the paper. Thus, this section explains how to load these base policies and watermark them. To train your own policy, see Section 2.

💧 1.1. Evaluating a watermarking strategy: watermark_policy.py

The watermark_policy.py script is the core tool for evaluating watermarking strategies. It uses Hydra for configuration management, with corresponding configuration files in config_watermark_policy/ (detailed in Section 1.2).

To run the script with default parameters (which performs 10 replications of CoNoCo on VMAS Navigation):

python3 watermark_policy.py

This would do the following:

1. Loads the model specified by the model_folder argument (e.g., models/navigation/). This loads both the policy and the environment it was trained on.
2. Builds the watermarking strategy specified by the watermark argument (e.g., conoco). This uses the corresponding definition in the watermarking_strategies/ folder.
3. For each requested glimpse modality (specified by ground_truth_actions_glimpses, onboard_sensors_glimpses, and remote_camera_feed_glimpses arguments), performs the analyses requested by the different analysis arguments (see Section 1.3 for details).

💧 1.2. Understanding the configuration in config_watermark_policy/

The main configuration file for watermark_policy.py is located at config_watermark_policy/config.yaml, which references two sub-configurations: the glimpse config in config_watermark_policy/glimpses/ and the watermark config in config_watermark_policy/watermark/.

Currently, there is only one glimpse configuration file: config_watermark_policy/glimpses/glimpses.yaml. However, there is one watermark configuration file per watermarking strategy. Hydra automatically loads the corresponding sub-configuration based on the chosen watermarking strategy. For example, when using CoNoCo, it loads config_watermark_policy/watermark/conoco.yaml.

The main configuration file config_watermark_policy/config.yaml defines common arguments such as model_folder (specifying where to load the policy from), ground_truth_actions_glimpses, onboard_sensors_glimpses, and remote_camera_feed_glimpses. It also defines per-analysis arguments (see Section 1.3).

Each watermark configuration in config_watermark_policy/watermark/ defines one mandatory argument name (the name of the watermarking strategy) and multiple strategy-specific arguments within an args section.

💧 1.3. Available analyses in watermark_policy.py

watermark_policy.py currently implements the following analyses. Each analysis runs for all requested glimpse modalities, specified in the configuration by the ground_truth_actions_glimpses, onboard_sensors_glimpses, and remote_camera_feed_glimpses arguments:

• analysis_episodes: Main analysis. Performs analysis_episodes.num_replications independent policy episodes, each of length analysis_episodes.length, both with and without watermarking. Records detectability, final return, and other metrics.
• analysis_length: Studies the impact of evaluation length on watermarking performance.
• analysis_sensitivity: Studies the impact of a given parameter on watermarking performance.
• analysis_glimpses: Studies the impact of glimpse parameters on watermarking performance.

💧 1.4. Comparing the results of watermarking strategies: plot_watermark_policy.py

The plot_watermark_policy.py script is a plotting tool that aggregates and compares results from multiple runs of watermark_policy.py. It recursively searches for all results in the outputs folder, automatically detects parameter discrepancies, and generates comparison plots.

🏋️ 2. Training a policy

This section provides instructions for training a base policy. Note that this does not include any watermarking and should not be needed for most use cases of this repository.

🏋️ 2.1. Training an RL policy: train_policy.py

The train_policy.py script trains an RL policy using PPO on a chosen environment. This script uses Hydra for configuration, with configuration files located in the config_train_policy/ folder. It can run PPO on any environment defined in this repository, using the default implementation from TorchRL.

To use a trained policy with watermark_policy.py, the outputs must be copied to the models/ directory so they can be accessed by the watermarking script.

🏋️ 2.2. Defining a new environment: environments/

The environments/ repository defines Environment Factories for each type of environment. These are interfaces that expose all necessary functions for watermarking. All factories follow the structure defined in environments/base_environment_factory.py. Most environments in this repository are not used directly from their corresponding libraries but are modified first, either to use velocity control (as in Navigation) or to adjust their delta_t parameter (as in MuJoCo environments).

An important distinction in this repository compared to other RL repositories is that multiple policies are trained within variants of the same environment that differ only in their frequencies, given by parameters:

• env_frequency: how often the environment is stepped, corresponds to the glimpses frequency.
• policy_frequency: how often the policy is queried and the action is updated, corresponds to the policy frequency.

Citation

If you use this code, please cite:

@misc{amir2025remotelydetectablerobotpolicy,
      title={Remotely Detectable Robot Policy Watermarking},
      author={Michael Amir and Manon Flageat and Amanda Prorok},
      year={2025},
      eprint={2512.15379},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2512.15379},
}