Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning

Yunchao Zhang, Zonglin Di, Kaiwen Zhou, Cihang Xie, Xin Eric Wang

University of California, Santa Cruz, USA 

Accepted to the Main Conference of NAACL 2024

Paper | Project Page

We release the reproducible code here.

Environment Installation

Python version: Need python 3.8

pip install -r python_requirements.txt

Please refer to this link to install Matterport3D simulator:

Pre-Computed Features

ImageNet ResNet152

Download image features for environments for Envdrop model:

mkdir img_features
wget https://www.dropbox.com/s/o57kxh2mn5rkx4o/ResNet-152-imagenet.zip -P img_features/
cd img_features
unzip ResNet-152-imagenet.zip

CLIP Features

Please download the CLIP-ViT features for CLIP-ViL models with this link:

wget https://nlp.cs.unc.edu/data/vln_clip/features/CLIP-ViT-B-32-views.tsv -P img_features

Training and Testing On RxR

Data

Please download the pre-processed data with link:

wget https://nlp.cs.unc.edu/data/vln_clip/RxR.zip -P tasks
unzip tasks/RxR.zip -d tasks/

Testing NAW and PBA

For testing the performance of PBA on RxR dataset with model FedEnvDrop, please run

bash run/agent_rxr_envdrop_attack.bash

For testing the performance of PBA on RxR dataset with model FedCLIP-ViL, please run

bash run/agent_rxr_clip_vit_attack.bash

If you want to simply test NAW without any defense, please change the param defense_method in the bash file from PBA to mean.

Training and Testing on R2R

Download the Data

Download Room-to-Room navigation data:

bash ./tasks/R2R/data/download.sh

Testing NAW and PBA

For testing the performance of PBA on R2R dataset with model FedEnvDrop, please run

bash run/agent_envdrop_attack.bash

For testing the performance of PBA on R2R dataset with model FedCLIP-ViL, please run

bash run/agent_clip_vit_attack.bash

If you want to simply test NAW without any defense, please change the param defense_method in the bash file from PBA to mean.

Related Links

• FedVLN: paper, code
• R2R Dataset: paper, code
• RxR Dataset: paper, code

Reference

If you use our work in your research or wish to refer to the baseline results published here, please use the following BibTeX entry.

@inproceedings{zhang-etal-2024-NAW,
    title = "Navigation as Attackers Wish? Towards Building Robust Embodied Agents under Federated Learning",
    author = "Zhang, Yunchao and Di, Zonglin and Zhou, Kaiwen and Xie, Cihang and Wang, Xin Eric",
    booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)",
    year = "2024",
}