Vision Tracking · Teleoperation · Raspberry Pi

LeRobot SO101: Recreating Advanced Robotic Control with Raspberry Pi 5, ROS2, and Hand Tracking

This blog post outlines the implementation of the LeRobot SO101 project. The goal of this project is to recreate the sophisticated control system which originally utilizes a wireless remote using a resource-conscious setup. It requires the lerobot setup provided here.

This implementation successfully translates precise human movements into robot coordinates using a Raspberry Pi 5, Raspberry Pi Camera Module 3 (IMX708), an ArUco marker cube, and a 10-axis HiWonder IMU.

Three distinct control methodologies have been developed and implemented:

01Robot Operating System 2 (ROS2) integration via WSL2/Ubuntu.
02Direct Hand Tracking using MediaPipe on a Raspberry Pi 5.
03ArUco Marker Cube Tracking with camera calibration.

Raspberry Pi 5 ROS2 Jazzy MediaPipe ArUco / ChArUco HiWonder IMU

METHOD 1

Robot Operating System 2 (ROS2) Implementation

The ROS2 framework uses an Ubuntu operating system environment (either dual-boot or Windows Subsystem for Linux (WSL)) to manage camera tracking and command mapping for the robotic arm.

Camera & Device Integration via WSL

To pass the laptop or external web camera feed into the WSL terminal, the host machine uses usbipd-win.

Install the tool via Windows PowerShell:
PowerShell

winget install --interactive --exact dorssel.usbipd-win
Identify the target camera's bus ID using the listing command:
PowerShell

usbipd list
Attach the camera (and the port connected to LeRobot) to the running WSL distribution:
PowerShell

usbipd attach --wsl --busid <BUSID>

Dependencies & Environment Setup

Inside the designated ROS2 virtual environment, execute the following commands to install the required Python libraries and ros-jazzy system packages:

Bash

pip install catkin_pkg empy==3.3.4 lark

pip install mediapipe opencv-python numpy scipy transforms3d

pip install rclpy geometry-msgs tf2-ros

sudo apt install ros-jazzy-pymoveit2 ros-jazzy-tf2-ros-py ros-jazzy-tf2-ros ros-jazzy-geometry-msgs ros-jazzy-rclpy ros-jazzy-moveit-py ros-jazzy-moveit-ros-planning-interface ros-jazzy-moveit-configs-utils

sudo apt update

Build the workspace by changing to your workspace directory and building the target packages:

Bash

cd ~/lerobot_ws

colcon build --symlink-install --packages-select pymoveit2

(Note: If the package fails, clean broken packages using rm -rf build/pymoveit2 install/pymoveit2 log/ and rebuild.)

ROS2 Core Code Implementation

1. Camera Teleoperation Node (camera-teleop.py)

This node captures video frames, processes hand joints via MediaPipe, mitigates jitter via a 1-Euro Filter, and publishes standard PoseStamped messages. The code for camera-teleop.py can be found here.

2. Serial Mapping Bridge (moveit-bridge.py)

This script subscribes to /fingertip_target, interprets changes in spatial coordinates, maps them to physical steps (0-4095), builds structural packages matching Dynamixel/Feetech serial guidelines, and executes actions. The code for moveit-bridge.py can be found here.

Initializing and Executing ROS2 Controls

Before starting execution, clean background processes and apply proper hardware definitions:

Bash

# Terminate conflicting routines
killall -9 python3
ros2 daemon stop
pkill -9 -f controller_manager
pkill -9 -f spawner
pkill -9 -f ros2

# Grant peripheral reading permissions
sudo chmod 777 /dev/video0
sudo chmod 777 /dev/video1
sudo chmod a+rw /dev/ttyACM0
sudo usermod -a -G dialout $USER
sudo usermod -a -G video $USER

# Terminal 1: Launch Controller Framework
ros2 launch lerobot_controller so101_controller.launch.py use_sim_time:=false

# Terminal 2: Run Hand Joint Teleoperation Tracking
python3 camera-teleop.py

# Terminal 3: Run Hardware Actuation Serial Bridge
python3 moveit-bridge.py

METHOD 2

Direct Hand Tracking using Raspberry Pi 5

This method targets lightweight execution locally on the Raspberry Pi 5, running Python 3.12.x and MediaPipe to handle coordinate mapping directly without a bulky ROS2 pipeline.

Python 3.12.13 Installation on Raspberry Pi OS

Ensure the board dependencies are initialized, install pyenv, and configure paths:
Bash

sudo apt update sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python3-openssl git curl https://pyenv.run | bash echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc echo 'eval "$(pyenv init -)"' >> ~/.bashrc exec "$SHELL"
Build and target Python v3.12.13 inside your project space:
Bash

pyenv install 3.12.13 cd /path/to/your/project/folder pyenv local 3.12.13
Initialize and trigger the local isolated workspace:
Bash

python3 -m venv myenv source myenv/bin/activate

Peripheral & System Optimizations

MediaPipe requires specific driver declarations to read system frames cleanly. Edit the main hardware configuration file:

Bash

sudo nano /boot/firmware/config.txt

Append the following parameters to ensure optimal functionality:

Plaintext

# Enable the IMX708 (Camera Module 3)
camera_auto_detect=1
dtoverlay=imx708

# Enable I2C and SPI buses for sensors/servo drivers
dtparam=i2c=on
dtparam=i2c_vc=on
dtparam=spi=on

# Allocate Memory for High-Res Camera Frames (Critical Fix for CMA Buffer Error)
dtoverlay=vc4-kms-v3d,cma-512

Save (Ctrl+O, Enter, Ctrl+X) and issue a machine restart command (reboot).

Next, pass required system permissions, set up symbolic links for libcamera, and export system paths:

Bash

echo 'SUBSYSTEM=="dma_heap", GROUP="video", MODE="0660"' | sudo tee /etc/udev/rules.d/99-raspberrypi-dma-heap.rules

sudo udevadm control --reload-rules && sudo udevadm trigger

sudo usermod -aG video,render,i2c $USER

echo "i2c-dev" | sudo tee -a /etc/modules
echo "i2c-bcm2835" | sudo tee -a /etc/modules

sudo apt install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxext6 libxrender-dev libcamera-tools libcamera-v4l2 gstreamer1.0-libcamera gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad python3-gst-1.0 python3-opencv i2c-tools v4l2loopback-utils cmake

# Resolve library variant mismatches (e.g., v0.5.2 vs v0.7)
sudo ln -sf /usr/lib/aarch64-linux-gnu/libcamera.so.0.5.2 /usr/lib/aarch64-linux-gnu/libcamera.so.0.7
sudo ln -sf /usr/lib/aarch64-linux-gnu/libcamera-base.so.0.5.2 /usr/lib/aarch64-linux-gnu/libcamera-base.so.0.7

sudo ldconfig

echo 'export GST_PLUGIN_PATH=/usr/lib/aarch64-linux-gnu/gstreamer-1.0' >> ~/.bashrc
source ~/.bashrc

pip install opencv-contrib-python flask mediapipe numpy

Local Camera Stream Server (camera-livestream.py)

Because local graphical display can be taxing, the camera pipeline exposes a Flask server interface running on a specific host IP address. Run code tracking scripts using libcamerify python3 script.py. The code for testing the camera-livestream.py can be found here.

Run the following command to run the camera-livestream.py:

Bash

python3 camera-livestream.py

HiWonder 10-Axis IMU Reading Node (imu-data-stream.py)

To monitor spatial orientation alongside linear coordinates, the HiWonder IMU captures live changes over serial (/dev/ttyUSB0). The code for imu-data-stream.py can be found here.

Run the following command to run the imu-data-stream.py:

Bash

python3 imu-data-stream.py

Direct 3D Spatial Hand Tracking Pipeline (hand-tracking.py)

This script activates low-overhead skeletal mapping (model_complexity=0) to extract spatial metrics, calculate depth proxy variables, and print results over web streaming links. The code for hand-tracking.py can be found here.

Run the following command to run the hand-tracking.py:

Bash

python3 hand-tracking.py

METHOD 3

ArUco Marker Cube Tracking & Vision Calibration

This method uses an ArUco marker tracking approach, translating the motion of a physical 3D-printed tracking cube (featuring distinct markers on 5 faces) into actionable robot instructions.

Hardware Warning: Ensure you calibrate your camera module using a ChArUco sheet before running tracking programs to isolate spatial distortion matrices.

1. ChArUco Calibration Pattern Generator (ChArUco-board-generator.py)

Run this script to build a high-resolution grid pattern. Print this pattern onto a flat, non-warping cardboard backing. The code for ChArUco-board-generator.py can be found here.

Run the following command to run the ChArUco pattern generator:

Bash

python3 ChArUco-board-generator.py

2. Interactive Vision Calibration Dashboard (camera-calibration.py)

This tool hosts an interactive web environment. Capture around 20 frames from varied vantage points and angles, then compute the intrinsic optical matrices to output camera_matrix.npz. The code for camera-calibration.py can be found here.

Run the following command to run the camera-calibration.py:

Bash

python3 camera-calibration.py

3. Cubical Target Marker Generation (marker-generator.py)

This routine produces tracking indicators across a small scale context. The code for marker-generator.py can be found here.

Assembly Note: Once printed, paste markers ID 0 through 4 onto a mathematically square physical reference cube (5 cm × 5 cm × 5 cm). Discard the 6th layout boundary section entirely, leaving 5 tracked faces exposed. The 3D cube file can be found here.

Save this code and run the following command to start the teleoperation using ArUco marker cube:

Bash

python3 ArUco-marker-teleoperation.py

As the cube with the ArUco marker is moved, the Lerobot starts moving like a puppet. Ensure to check and update the port to which the Lerobot is connected (ACM0 for this code).