Summary

This repository provides a specialized benchmark, MingLi-Bench, designed to evaluate the capabilities of Large Language Models (LLMs) in understanding and reasoning about complex Chinese traditional fortune-telling systems: Bazi (八字) and Ziwei Doushu (紫微斗数). It targets researchers and developers seeking to assess LLM performance in niche, culturally specific domains, offering a structured method to measure analytical and predictive reasoning beyond standard NLP tasks. The primary benefit is isolating LLM reasoning from the complexities of astrological chart derivation.

How It Works

MingLi-Bench employs a corpus of 160 normalized, multiple-choice questions sourced from the Global Fortune Teller Competition (2022–2025). Evaluation is performed via exact match against ground-truth answers. A key design choice is the --astro flag, which injects pre-computed Bazi and Ziwei charts into the prompt. This isolates the LLM's pure reasoning ability, decoupling it from the task of deriving astrological charts from birth data. The --cot (Chain-of-Thought) flag further encourages methodical reasoning.

Quick Start & Requirements

• Installation: Clone the repository (git clone https://github.com/DestinyLinker/MingLi-Bench.git), navigate into the directory (cd MingLi-Bench), and install dependencies (pip install -r requirements.txt).
• Prerequisites: Python 3.9+ is required. API keys for supported LLM providers (OpenAI, Anthropic, Google, Deepseek, Doubao/Volcengine) must be configured in a .env file.
• Setup: Requires cloning, dependency installation, and API key configuration.
• Links: Repository: https://github.com/DestinyLinker/MingLi-Bench

Highlighted Details

• Dataset: 160 multiple-choice questions covering twelve life aspects (career, health, marriage, wealth, etc.), sourced from 2022–2025 competitions.
• Evaluation Modes: Recommends using --cot for reasoning and --astro to inject pre-computed charts, isolating LLM reasoning.
• Model Routing: Supports direct API calls to providers or routing through OpenRouter for access to a wider range of models.
• CLI Interface: Offers extensive command-line options for filtering by year, category, sampling, shuffling, and controlling output.

Maintenance & Community

• Contact: Issues can be reported via GitHub Issues (https://github.com/DestinyLinker/MingLi-Bench/issues). Direct email contact is available at help@destinylinker.com. No community forums like Discord or Slack are explicitly mentioned.

Licensing & Compatibility

• License: MIT License.
• Compatibility: The MIT license is highly permissive, allowing for commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

The benchmark is highly specialized to Chinese fortune-telling, limiting its generalizability to other domains. Running evaluations requires access to and configuration of third-party LLM APIs, which may incur costs and are subject to rate limits. The exact-match scoring mechanism might be overly strict for LLM outputs that demonstrate understanding but not precise phrasing.