MineBench addresses the challenge of evaluating AI models' spatial reasoning capabilities, moving beyond traditional text-based benchmarks. It targets researchers and developers seeking to assess an AI's ability to understand and generate 3D geometry, offering a direct, visual method to gauge raw intelligence and identify practical limitations often masked by theoretical performance metrics.

How It Works

MineBench evaluates AI models by tasking them with generating Minecraft-style voxel constructions from natural-language prompts. Models output raw 3D coordinates as JSON, or utilize a voxel.exec tool with minimal primitives for complex builds exceeding token limits. This approach uniquely tests spatial logic and mathematical reasoning without relying on vision models, providing a direct assessment of a model's geometric understanding. Outputs are visualized, and models are ranked via head-to-head comparisons using a confidence-aware Glicko-style system, highlighting practical intelligence over benchmark scores.

Quick Start & Requirements

For local development and comparison of existing builds, prerequisites include Node.js 18+, pnpm, and Docker. Installation involves running pnpm install, copying .env.example to .env, and executing pnpm dev:setup. A second terminal requires pnpm prompt --import. The application is accessible via http://localhost:3000/ (Arena), http://localhost:3000/sandbox, and http://localhost:3000/leaderboard. Detailed setup, environment variables, and deployment instructions are available in the project's documentation.

Highlighted Details

• Arena: Facilitates blind, head-to-head model comparisons with a confidence-aware ranking system.
• Sandbox: Allows live generation and comparison of builds using personal API keys.
• Local Lab: Enables users to run prompts locally, paste JSON output for rendering, and test models independently.
• Leaderboard: Displays live rankings based on win/loss/draw statistics across all benchmarked models.
• Exports: Supports saving generated builds in GLB, STL, or WorldEdit .schem formats for use in Blender, 3D printing, and Minecraft.
• Supported Models: Integrates with major providers including OpenAI, Anthropic, Google, Meta, and models via OpenRouter.

Maintenance & Community

The project is sponsored by 3D-Agent, an AI assistant for 3D workflows, with an affiliate commission disclosure. Contributions are welcomed for adding new models, prompts, UI improvements, or bug fixes, guided by CONTRIBUTING.md. Running the benchmark incurs significant operational costs, with direct support encouraged via Buy Me a Coffee.

Licensing & Compatibility

MineBench is released under the MIT license, permitting broad usage and modification. No specific compatibility restrictions for commercial use or closed-source linking are detailed beyond the standard terms of the MIT license.

Limitations & Caveats

The project's documentation is noted as being almost entirely AI-created. Significant operational costs associated with model inference, storage, and hosting are incurred by running the benchmark, requiring financial support or sponsorship.