<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This repository offers a straightforward Python script designed to evaluate the performance of local Codex Command Line Interface (CLI) models. It targets users who need to quickly assess model capabilities on a specific mathematical reasoning task, providing metrics on reasoning token consumption and accuracy. The primary benefit is a simple, dependency-free method for batch testing Codex CLI models against a defined problem set.

How It Works

<2-4 sentences on core approach / design (key algorithms, models, data flow, or architectural choices) and why this approach is advantageous or novel.> The core approach involves executing a predefined "candy math problem" through the Codex CLI. Users can configure the specific model to test, the desired reasoning effort level (ranging from low to xhigh), and the number of test iterations to perform. Upon execution, the script quantifies the reasoning tokens consumed by the model for each test and determines correctness by checking if the exact numerical answer '21' is present within the model's generated output. This allows for a quantitative comparison of different model configurations.

Quick Start & Requirements

• Primary install / run command (pip, Docker, binary, etc.).
• Non-default prerequisites and dependencies (GPU, CUDA >= 12, Python 3.12, large dataset, API keys, OS, hardware, etc.).
• Estimated setup time or resource footprint.
• If they are present, include links to official quick-start, docs, demo, or other relevant pages.

• Primary install/run command: No installation is required beyond having the Codex CLI installed and authenticated. The script can be run directly from a local file or fetched and executed on-the-fly using wget or curl.
  • Example: wget -qO- "https://raw.githubusercontent.com/haowang02/codex-candy-eval/main/codex_candy_eval.py" | python3 - -m gpt-5.5 -r high -n 5
• Prerequisites: Codex CLI installed and logged in.
• Parameters:
  • -m, --model: Specify the Codex model name (e.g., gpt-5.5).
  • -r, --reasoning-effort: Set effort level (low, medium, high, xhigh; default: medium).
  • -n, --tests: Number of test runs (default: 1).

Highlighted Details

• Bullet 1 (benchmarks, performance claims, novel integration, etc.)
• Bullet 2
• Bullet 3
• Bullet 4 (optional)

• Facilitates batch testing of local Codex CLI models.
• Provides key metrics: reasoning token count and accuracy.
• Features zero third-party Python dependencies for ease of use.

Maintenance & Community

• Notable contributors, sponsorships, partnerships, deprecations, migrations, or other health signals if notable.
• Links to Discord/Slack, social handles, roadmap, etc.

• No specific details regarding contributors, sponsorships, or community channels (e.g., Discord, Slack) are present in the provided README.

Licensing & Compatibility

• License type and notable restrictions (GPL -> copyleft, SSPL, etc.).
• Compatibility notes for commercial use or closed-source linking.

• The license for this project is not explicitly stated in the provided README, which may require further investigation for commercial or distribution use.

Limitations & Caveats

<1-3 sentences on caveats: unsupported platforms, missing features, alpha status, known bugs, breaking changes, bus factor, deprecation, etc. Avoid vague non-statements and judgments.> The evaluation scope is narrow, focusing on a single, specific math problem. The correctness check is simplistic, relying solely on the presence of the digit '21' in the output, which may not accurately reflect the model's understanding or problem-solving capabilities. Requires a functional and authenticated Codex CLI setup, which could be a barrier to entry for users without prior configuration.