This Python framework accelerates structured output generation from LLMs by decomposing target schemas into atomic, independently generatable components. It targets developers and researchers needing to extract structured data from unstructured text, offering up to 10x speed improvements over naive methods and enhanced determinism.

How It Works

Super JSON Mode leverages the inherent parallelism of LLMs by treating each key-value pair in a target schema as a separate, independent generation task. Instead of prompting the LLM to generate an entire JSON object at once, it queries the model for each field individually, significantly reducing token usage and inference time. This approach exploits the fact that many schema fields do not depend on each other for their extraction.

Quick Start & Requirements

• Install via PyPI: pip install super-json-mode
• Requires Python 3.10+.
• Supports OpenAI legacy completions API, Hugging Face Transformers, and vLLM.
• Examples and detailed usage for OpenAI and Hugging Face Transformers are available in the repository.

Highlighted Details

• Achieves up to 10x faster structured output generation compared to naive prompting.
• More deterministic and less prone to parsing errors than standard methods.
• Supports parallel generation of schema components for improved LLM throughput.
• Integrates with popular LLM frameworks like OpenAI, Hugging Face Transformers, and vLLM.

Maintenance & Community

• Developed as part of CS 229: Systems for Machine Learning.
• Citation details are provided for academic use.
• Roadmap includes qualitative analysis, structured sampling, dependency graph support, local model integration (Ollama, Llama.cpp), and TRT-LLM support.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.

Limitations & Caveats

The framework currently does not support schemas with inter-field dependencies (e.g., chain-of-thought where a response depends on a prior thought). Future work aims to address this by incorporating dependency graph support.