WIP: long-horizon with LLMs

.codespell-ignore-words.txt

@@ -1 +1,2 @@
 empy
+crate

.gitignore

@@ -135,4 +135,6 @@ data/
 log/
 
 # install folder
-install/
+install/
+
+debug*/

docs/apps.md

@@ -226,3 +226,13 @@ You can add the `--reset` flag to make it go back to the start position. The def
 ```
 python -m stretch.app.pickup --reset
 ```
+
+## Experimental
+
+### VLM Planning
+
+This is an experimental app that uses the voxel map to plan a path to a goal. It is not yet fully functional.
+
+```bash
+python -m stretch.app.vlm_planning
+```

docs/docker.md

@@ -2,6 +2,8 @@
 
 This is a guide to setting up a Docker container for running the Stretch software. This is useful for running the software on a computer that doesn't have the correct dependencies installed, or for running the software in a controlled environment.
 
+Hello Robot has a set of scripts for [building robot-side docker images](https://github.com/hello-robot/stretch_docker), which can be conconsidered separately. Here, we will go through what docker is, why we use it, and how to set it up on your client side to run AI code.
+
 ## What is Docker and Why Would I Use It?
 
 Docker is a tool that allows you to run software in a container. A container is a lightweight, standalone, executable package of software that includes everything needed to run the software, including the code, runtime, system tools, libraries, and settings. Containers are isolated from each other and from the host system, so they are a great way to run software in a controlled environment.

docs/simulation.md

@@ -0,0 +1,49 @@
+# Robocasa Installation
+
+You can install Robocasa by following the instructions below, or you can try the [install script](scripts/install_robocasa.sh).
+
+## Install Robosuite
+
+```bash
+git clone https://github.com/ARISE-Initiative/robosuite -b robocasa_v0.1
+cd robosuite
+pip install -e .
+```
+
+## Install Robocasa
+
+```bash
+cd ..
+git clone https://github.com/robocasa/robocasa
+cd robocasa
+pip install -e .
+```
+
+## Install Stretch Mujoco
+
+```bash
+cd ..
+git clone git@github.com:hello-robot/stretch_mujoco.git
+cd stretch_mujoco
+pip install -e .
+```
+
+## Download assets
+
+```bash
+python robocasa/scripts/download_kitchen_assets.py   # Caution: Assets to be downloaded are around 5GB.
+python robocasa/scripts/setup_macros.py              # Set up system variables.
+```
+
+## Run Robocasa
+
+In one terminal start the server:
+
+```bash
+python -m stretch.simulation.mujoco_server --use-robocasa
+```
+
+In another run an app, like mapping:
+
+```bash
+```

install.sh

@@ -11,8 +11,6 @@ VERSION=`python $script_dir/src/stretch/version.py`
 CPU_ONLY="false"
 NO_REMOVE="false"
 NO_SUBMODULES="false"
-INSTALL_PYTORCH3D="false"
-INSTALL_TORCH_GEOMETRIC="false"
 MAMBA=mamba
 NO_VERSION="false"
 # Two cases: -y for yes, --cpu for cpu only
@@ -41,10 +39,6 @@ do
             NO_SUBMODULES="true"
             shift
             ;;
-        --torch-geometric)
-            INSTALL_TORCH_GEOMETRIC="true"
-            shift
-            ;;
         --no-version)
             NO_VERSION="true"
             shift
@@ -157,19 +151,6 @@ echo "---------------------------------------------"
 echo "---- INSTALLING STRETCH AI DEPENDENCIES  ----"
 echo "Will be installed via pip into env: $ENV_NAME"
 
-# This is no longer necessary but might be useful for some checks
-if [ "$INSTALL_PYTORCH3D" == "true" ]; then
-    echo "Installing pytorch3d from source"
-    pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
-fi
-
-if [ "$INSTALL_TORCH_GEOMETRIC" == "true" ]; then
-    echo "Installing torch-geometric"
-    # If not using cpu only, install the following
-    # It is important to use --no-cache-dir to avoid issues different versions of pytorch and cuda
-    pip install torch_cluster torch_scatter torch_geometric -f https://pytorch-geometric.com/whl/torch-${PYTORCH_VERSION}+${CUDA_VERSION_NODOT}.html --no-cache-dir
-fi
-
 pip install -e ./src[dev]
 
 echo ""

scripts/copy_calibrated_stretch_urdf.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+# Default user
+USER="hello-robot"
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --robot_ip)
+            ROBOT_IP="$2"
+            shift 2
+            ;;
+        --user)
+            USER="$2"
+            shift 2
+            ;;
+        *)
+            echo "Unknown option: $1"
+            exit 1
+            ;;
+    esac
+done
+
+# If ROBOT_IP is not set, try to load it from the file
+if [ -z "$ROBOT_IP" ]; then
+    if [ -f ~/.stretch/robot_ip.txt ]; then
+        ROBOT_IP=$(cat ~/.stretch/robot_ip.txt)
+    else
+        echo "Error: Robot IP not provided and ~/.stretch/robot_ip.txt does not exist."
+        exit 1
+    fi
+fi
+
+# Check if ROBOT_IP is set
+if [ -z "$ROBOT_IP" ]; then
+    echo "Error: Robot IP is not set."
+    exit 1
+fi
+
+# Run the scp command
+scp "$USER@$ROBOT_IP:~/ament_ws/src/stretch_ros2/stretch_description/urdf/stretch.urdf" $HOME/.stretch/stretch.urdf
+
+# Check if the scp command was successful
+if [ $? -eq 0 ]; then
+    echo "File successfully copied."
+else
+    echo "Error: Failed to copy the file."
+    exit 1
+fi

scripts/install_robocasa.sh

@@ -0,0 +1,92 @@
+#!/bin/bash
+
+set -e
+
+# Initialize variables
+yes_flag=false
+conda_env=""
+
+# Parse command-line options
+while getopts ":y" opt; do
+    case ${opt} in
+        y )
+            yes_flag=true
+            ;;
+        \? )
+            echo "Invalid option: $OPTARG" 1>&2
+            exit 1
+            ;;
+    esac
+done
+
+# Check if running in a conda/mamba environment
+if [[ -n $CONDA_DEFAULT_ENV ]]; then
+    conda_env=$CONDA_DEFAULT_ENV
+elif [[ -n $MAMBA_ROOT_PREFIX ]]; then
+    conda_env=$(basename "$MAMBA_ROOT_PREFIX")
+else
+    echo "Error: No conda/mamba environment detected. Please activate an environment before running this script."
+    exit 1
+fi
+
+echo "Using conda/mamba environment: $conda_env"
+
+# Function to ask for confirmation
+confirm() {
+    if [ "$yes_flag" = true ]; then
+        return 0
+    fi
+    read -r -p "${1:-Are you sure?} [y/N] " response
+    case "$response" in
+        [yY][eE][sS]|[yY])
+            true
+            ;;
+        *)
+            false
+            ;;
+    esac
+}
+
+if confirm "Do you want to install Robosuite and Robocasa?"; then
+    # Install Robosuite
+    git clone https://github.com/ARISE-Initiative/robosuite -b robocasa_v0.1
+    cd robosuite
+    pip install -e .
+    cd ..
+
+    # Install Robocasa
+    git clone https://github.com/robocasa/robocasa
+    cd robocasa
+    pip install -e .
+    cd ..
+else
+    echo "Skipping Robosuite and Robocasa installation"
+    # Quit
+    exit 1
+fi
+
+# Download assets
+python robocasa/scripts/download_kitchen_assets.py
+
+# Setup macros
+if confirm "Do you want to set up system variables and macros for Robocasa?"; then
+    python robocasa/scripts/setup_macros.py
+fi
+
+# Clone, cd, and install stretch_mujoco repository
+if git clone git@github.com:hello-robot/stretch_mujoco.git; then
+    echo "Successfully cloned stretch_mujoco repository"
+    cd stretch_mujoco
+    if pip install -e .; then
+        echo "Successfully installed stretch_mujoco"
+    else
+        echo "Failed to install stretch_mujoco"
+        exit 1
+    fi
+    cd ..
+else
+    echo "Failed to clone stretch_mujoco repository"
+    exit 1
+fi
+
+echo "Installation complete!"

src/setup.py

@@ -49,6 +49,7 @@
         "pyusb",
         "schema",
         "overrides",
+        "wget",
         # From openai
         "openai",
         "openai-clip",

src/stretch/agent/manipulation/dinobot.py

@@ -8,12 +8,17 @@
 # license information maybe found below, if so.
 
 import glob
+import sys
 
 import click
 import matplotlib.pyplot as plt
 import numpy as np
 import torch
 
+# sys.path.append("/home/hello-robot/repos/dino-vit-features")
+sys.path.append("/home/cpaxton/src/dino-vit-features")
+import sys
+
 # Install this DINO repo to extract correspondences: https://github.com/ShirAmir/dino-vit-features
 from correspondences import draw_correspondences, find_correspondences
 from PIL import Image
@@ -61,6 +66,8 @@ def load_demo(path_to_demo_folder):
         demo_deltas.append(delta)
 
     # pull out the first frame in the videos for the bottleneck images
+    # rgb_bn = Image.open("ee_rgb_0.jpg")
+    # depth_bn = Image.open("ee_rgb_1.jpg")
     rgb_bn = Image.open("bottleneck_rgb.png")
     depth_bn = Image.open("bottleneck_depth.png")
 
@@ -138,6 +145,7 @@ def extract_3d_coordinates(points, xyz):
         x, y = point
         points_3d.append([x, y, depths[i]])
 
+    breakpoint()
     return np.array(points_3d)
 
 
@@ -233,3 +241,7 @@ def main(robot_ip, local, path_to_demo_folder):
 
     # Start the demo
     dinobot_demo(robot, path_to_demo_folder)
+
+
+if __name__ == "__main__":
+    main()