DroidRun Evaluation with AndroidWorld

This module provides tools for benchmarking DroidRun using the AndroidWorld (upstream) task suite - a collection of 116 diverse tasks across 20 Android applications.

---

1. Manual Environment Setup

Reference: android_world official documentation.

1.1 Install Android Emulator

• Download Android Studio
• Create an Android Virtual Device (AVD):
  • Hardware: Pixel 6
  • System Image: Tiramisu, API Level 33
  • AVD Name: AndroidWorldAvd
• Setup video tutorial

1.2 Launch Android Emulator

Launch the emulator from command line (not Android Studio UI) with the -grpc 8554 flag, which is required for accessibility forwarding.

# Emulator path is usually located at:
# ~/Android/Sdk/emulator/emulator
# ~/Library/Android/sdk/emulator/emulator

EMULATOR_NAME=AndroidWorldAvd
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

---

2. Clone and Install

2.1 Clone repository and initialize submodules

git clone https://github.com/droidrun/droidrun-android-world && \
cd droidrun-android-world

# initialize submodules (android_world + droidrun)
git submodule update --init

2.2 Install dependencies

# install dependencies using uv
uv sync
# build cli
uv build

---

3. Configuration

3.1 Set API Key

export GEMINI_API_KEY=your-key  # Or other provider keys

3.2 Start AndroidWorld Environment

Must be executed in droidrun-android-world/android_world directory, and keep running:

cd android_world
python -m server.android_server

3.3 Common Issues

1. adb not found
   Locate adb path:

   which adb

   Then update adb_path in android_server.py:

   adb_path="/opt/android/platform-tools/adb"

---

4. Running Tasks

4.1 Verify Environment

Run in another terminal (keep server.android_server running):

droidworld check

4.2 Run a Task

Execute from droidrun-android-world directory:

# Example: add contact task
droidworld run --tasks ContactsAddContact

---

5. Summary

• Android Emulator must be launched via command line and kept running
• server.android_server must be kept alive in background
• Run tasks only under droidrun-android-world directory

Usage

Basic Usage

Run a specific task by index:

droidworld run --min-task-idx 0 --max-task-idx 1

Run a specific task by name:

droidworld run --task ContactsAddContact

List Available Tasks

View all available tasks with their IDs:

droidworld list-tasks

Customizing the Benchmark

# Run with a different LLM provider and model
droidworld run --llm-provider Anthropic --llm-model claude-3-sonnet-20240229

# Set maximum steps per task: multiplier * task complexity
droidworld run --max-step-multiplier 15

# Run multiple parameter combinations per task
droidworld run --n-task-combinations 3

# Check all available configuration options with
droidworld run --help

Accessibility Service Notes

The benchmark script automatically manages the DroidRun accessibility service, which is required for proper interaction with Android UI elements: If you encounter issues with UI interaction:

• Verify the DroidRun Portal app is installed correctly
• Make sure both Droidrun Portal and Google Accessibility Forwarder are configured and enabled as accessiblity service

To diagnose run droidworld check