CRAVES provides a vision-based system for controlling a toy robotic arm (OWI-535) using a single RGB camera. It targets researchers and developers in robotics and computer vision, enabling robotic arm manipulation without additional sensors through its pose estimation module.

How It Works

The system employs a deep learning-based 2D pose estimation module, built upon the pytorch-pose framework, to identify keypoints on the robotic arm. This module is trained on synthetic, lab, and YouTube datasets. The estimated poses are then used by a separate control module (hosted on a different repository) to command the robotic arm's movements.

Quick Start & Requirements

• Install dependencies: pip install pytorch opencv-python imageio
• Download pretrained model checkpoint: [link provided in README]
• Download datasets: synthetic, lab, youtube [links provided in README]
• Run evaluation: sh scripts/val_arm_reall.sh (ensure paths in the script match your setup)
• For data generation: Download Unreal Engine binary, then run python data_generation/demo_capture.py after installing unrealcv and imageio.

Highlighted Details

• Achieves 99.46% accuracy on the lab dataset using the PCK@0.2 metric.
• Offers scripts for training from scratch, evaluating on synthetic and YouTube datasets, and visualizing results.
• Includes a data generation pipeline using Unreal Engine for creating synthetic training data.

Maintenance & Community

• The project is associated with a CVPR 2019 publication.
• Issues can be raised in the repository for questions or suggestions.

Licensing & Compatibility

• The README does not explicitly state a license. The project is presented as open-source code for research purposes.

Limitations & Caveats

• Requires PyTorch version 0.4.1 or higher.
• The control module for hardware interaction is hosted in a separate repository.
• The system is demonstrated with a specific toy robotic arm (OWI-535).