A simple API for sampling symbolic planning tasks in large-scale 3D scene graphs.
This repository corresponds to Taskography-API as described in Taskography: Evaluating robot task planning over large 3D scene graphs, presented at CoRL2021: project page, paper link. We provide support for the following:
- Hierarchical-Symbolic Graph Construction.
The raw Gibson scene graph data is loaded from its
.npzfile format encoding before heuristically determining the scene's inter- and intra-layer connectivity structure. - Task Sampling. Problem samplers for auto-generating PDDLGym domains of increasing complexity:
Rearrangement(k),Courier(n,k),Lifted Rearrangement(k),Lifted Courier(n,k), as described in our paper. Task samplers are modifiable to the degree literal goal conjuctions (k) and stow capacity (n). - Trajectory Sampling. Scripts for generating state-action trajectory datasets atop PDDLGym environments with your choice of PDDL planner; for purposes of training neurosymbolic learning-to-plan algorithms.
- Environment Wrappers. Gym environment wrappers enabling symbolic interaction with 3D scene graphs - in support of training online decision making algorithms and reinforcement learning methods.
We refer to the Instructions section for details on the basic and extended usage of our framework.
This repository has been primarily tested on Ubuntu 16.04, 18.04 and macOS Monterey with Python 3.7.
Install Docker as per these steps. We also recommend creating a virtual environment, e.g., with venv or anaconda3 before proceeding with the following installation commands.
# if on macOS
brew install coreutils
git clone https://github.com/taskography/taskography-api.git --recurse-submodules
cd ./taskography_api && pip install .
pip install -r requirements.txtPlease follow the download instructions for the Gibson 3D Scene Graph database. The testing of our repository is limited to the tiny (388 Mb) and medium (389.5 Mb) data splits.
Set an environment variable $DATA to parent directory of the unpacked dataset organized as follows:
📦 3dscenegraph
┣ 📂 medium
┃ ┗ 📂 automated_graph
┃ ┃ ┣ 📜 3DSceneGraph_Adairsville.npz
┃ ┃ ┣ ...
┃ ┃ ┗ 📜 3DSceneGraph_Wyldwood.npz
┣ 📂 tiny
┃ ┣ 📂 automated_graph
┃ ┃ ┣ 📜 3DSceneGraph_Allensville.npz
┃ ┃ ┣ ...
┃ ┃ ┗ 📜 3DSceneGraph_Woodbine.npz
┃ ┗ 📂 verified_graph
┃ ┃ ┣ 📜 3DSceneGraph_Allensville.npz
┃ ┃ ┣ ...
┃ ┃ ┗ 📜 3DSceneGraph_Woodbine.npz
┣ 📜 3DSceneGraph_medium.zip
┗ 📜 3DSceneGraph_tiny.zipYou may validate your dataset setup with:
python scripts/validate/loader.py --data-path $DATA/3dscenegraphTaskography-API is designed to enable rapid prototyping of 3D scene graph symbolic planning domains. The API's functionality can be partitioned into three main code groups: samplers, datasets and envs.
Task samplers allow the user to generate an arbitrary number of PDDL planning problems to any one 3D scene graph instance. The example samplers are written on a per-domain basis; i.e., each sampler has a corresponding hand-defined PDDL domain file that specifies the object types, predicate relations, the action set, and the symbolic transition model of a given task category.
Task category versions are often referred to interchangebly by their domain alias. For instance, taskographyv2 corresponds to the rearrangement task category. Please see the table below for a comprehensive list.
| Task Category | Task Sampler | PDDL Domain File | Short Description |
|---|---|---|---|
Rearrangement(k) |
TaskSamplerV2 | taskographyv2.pddl | Pick-and-place k items at uniform randomly sampled goal locations |
Courier(n,k) |
TaskSamplerV3 | taskographyv3.pddl | Rearrangement(k) + bagslot capacity n to stow and retrieve items |
Lifted Rearrangement(k) |
TaskSamplerV4 | taskographyv4.pddl | Satisfy k item-receptacle class relations (e.g., cup in sink) sampled at random |
Lifted Courier(n,k) |
TaskSamplerV5 | taskographyv5.pddl | Lifted Rearrangement(k) + bagslot capacity n to stow and retrieve items |
Task samplers must subclass ProblemSamplerBase: an abstract base class defining several must-implement methods that interface heavily with PDDLGym's domain parser to create object-oriented references to scene graph entities and relations before the problem parser writes them out as PDDL problem files.
ProblemSamplerBase itself inherets from TaskSamplerBase, which is responsible for determining the hierarchical, spatial, and symbolic relationships between nodes in the loaded 3D scene graph.
Should you require a custom sampler for new task categories, we recommend following these steps:
- Manually write a PDDL domain file and include it in the domains directory;
- If the domain file follows the
domains/taskographyv<i>.pddlformat, provide a domain name alias for use in naming auto-generated PDDLGym domains and problem files; - Code-up a sampler in
taskography_api/taskography/samplers/domainssubclassingProblemSamplerBaseand make it accessible by exposing the class in these modules:
The task samplers can be used for various downstream applications.
We currently provide scripts for (a) creating PDDLGym domains and (b) generating expert demonstrations provided by classical planners.
Dataset specifics can be configured in .yaml files passed to the appropriate entry points.
The PDDLGymDataset class gives users the ability to auto-generate PDDLGym environments comprising of train and test problems across an entire data split of 3D scene graphs, e.g., Gibson tiny or medium.
Task and dataset details are configured as per the provided .yaml file - please see examples for each of our task categories.
With this complete, creating a novel PDDLGym environment is as simple as running:
python scripts/generate_domain.py --config <path/to/problem_config.yaml>Our instructions installs PDDLGym at third_party/pddlgym as an editable package.
This allows new environments to be incorporated without rebuilding Taskography-API.
The script dynamically modifies pddlgym/pddlgym/__init__.py so that new environments are registered the next time import pddlgym is run. If you did not follow our installation instructions, you may need to modify the configuration file's problem_dir parameter to reflect your PDDLGym package location.
If parameter save_samplers is set true, the task samplers will be saved in pickle file format to the default location of datasets/samplers/<domain_alias>/<sampler_name>.
This circumvents 3D scene graph pre-processing when generating additional datasets of previously specified tasks.
The TrajectoryGymDataset class loads a newly generated PDDLGym environment, iterates through all train (and optionally test) problems, and saves the state-action trajectories of problems solved by a specified classical planner. Please refer to our fork of pddlgym_planners for a full list of supported planners.
The trajectories are by default saved to datasets/trajectories/<domain_name> in pickle file format.
We provide an example configuration file to complement the launch command:
python scripts/generate_trajectories.py --config <path/to/trajectory_config.yaml>To support training of futuristic learning-to-plan algorithms, we provide a Taskography gym class that can be utilized as per the familiar gym API. Instantiating this environment requires specifying the task category and Gibson split, along with a few other parameters - please see our example.
You may validate your environment configuration as so:
python scripts/validate/taskography_env.py --config <path/to/env_config.yaml>A scene graph and task are sampled at uniform random with each env.reset() call.
For efficiency, there's the option to pre-compute episodes_per_scene tasks per sampled 3D scene graph, i.e., agents learn in a new scene every episodes_per_scene episodes.
Tasks are sampled with replacement.
We provide two additional benchmarking scripts for extracting metrics of interest against newly generated PDDLGym environments. To benchmark the performance of any supported planner on a particular domain, scripts/benchmark/plan.py is the appropriate entry point:
python scripts/benchmark/plan.py --domain-name <pddlgym_domain_name> --planner <planner_name> [--log-dir <log_dir>] [--expid <expid>] [--timeout <timeout>]To analyze key properties of the PDDLGym environment, e.g., the number of ground actions or the mean branch fator across all task instances, use scripts/benchmark/domain_analysis.py as:
python scripts/benchmark/domain_analysis.py --domain-name <pddlgym_domain_name> --exp-name <exp_name>Statistics are by default saved to the ./exps.
Please refer to the code for optional arguments not included above.
The official benchmark domains are pre-registered in our PDDLGym fork and follow a different naming convention than those produced by Taskography-API.
A list of the domain names can be found in constants.py with associated benchmark results made available in benchmark.
Taskography-API has an MIT License. If you find this package helpful, please consider citing our work:
@inproceedings{agia2022taskography,
title={Taskography: Evaluating robot task planning over large 3D scene graphs},
author={Agia, Christopher and Jatavallabhula, {Krishna Murthy} and Khodeir, Mohamed and Miksik, Ondrej and Vineet, Vibhav and Mukadam, Mustafa and Paull, Liam and Shkurti, Florian},
booktitle={Conference on Robot Learning},
pages={46--58},
year={2022},
organization={PMLR}
}
We would like to credit the developers of several very useful packages.