PhysTwin offers a framework for physics-informed reconstruction and simulation of deformable objects from video data. It is designed for researchers and developers in XR, VR, and robotics, aiming to enable realistic object dynamics, interactive manipulation, and bridge the gap between real-world data and simulation environments.

How It Works

The system reconstructs deformable object dynamics and appearance by leveraging a differentiable Spring-Mass simulator, significantly accelerated by Nvidia Warp. This approach enables rapid first-order optimization, achieving results in minutes rather than hours, and allows for interactive exploration and simulation of object behavior based on video input.

Quick Start & Requirements

• Installation: Supports Linux (Python 3.10, CUDA 12.1/12.8), Windows (via windows_setup branch), and Docker. Specific setup scripts (env_install.sh, 5090_env_install.sh) are provided.
• Prerequisites: CUDA (12.1+ recommended), Python 3.10. Optional dependencies (Trellis, Grounding-SAM-2, etc.) can be skipped for interactive playground use.
• Data: Requires downloading pretrained models and project data (original, processed, experiments).
• Resource Footprint: GPU memory optimized to ~2GB for the interactive playground. First-order optimization completes in ~5 minutes.
• Links: Mentions Website, Paper, and Arxiv, but direct URLs are not provided in the README.

Highlighted Details

• Optimization Speed: First-order optimization achieves ~5-minute runtimes using Nvidia Warp, a substantial improvement over prior methods.
• Memory Efficiency: Interactive playground GPU memory usage reduced to ~2GB through CPU offloading of LBS initialization.
• Advanced Features: Includes experimental capabilities for material visualization, force visualization, and handling collisions among multiple PhysTwins.
• Self-Collision Acceleration: A pre-released feature significantly speeds up cloth-like simulations by optimizing collision checks.
• Remote Interaction: Interactive playground supports virtual keyboard input from remote devices.

Maintenance & Community

The repository is actively maintained by the authors, with plans for continuous updates and the long-term development of a comprehensive physics simulator for real-to-sim applications. Collaboration is welcomed via email (hanxiao.jiang@columbia.edu). No specific community channels (e.g., Discord, Slack) are listed.

Licensing & Compatibility

The license type is not explicitly stated in the README. Compatibility for commercial use or integration with closed-source projects is not detailed.

Limitations & Caveats

Several features, including multi-object collision handling and self-collision acceleration, are experimental or pre-released. Potential library conflicts exist between gaussian-splatting and Trellis during data processing. The fully accelerated self-collision system is pending further development.