UltraFeedback provides a large-scale, fine-grained, and diverse preference dataset for training reward and critique models, targeting researchers in Reinforcement Learning from Human Feedback (RLHF). It aims to improve LLM alignment by offering detailed annotations and enabling the development of more capable feedback models.

How It Works

The dataset comprises 64k prompts from diverse sources, generating 256k responses from 17 different LLMs. Responses are annotated by GPT-4 across four dimensions: instruction-following, truthfulness, honesty, and helpfulness, providing both numerical ratings and textual rationales. This fine-grained approach allows for more nuanced reward model training compared to simpler preference datasets.

Quick Start & Requirements

• Dataset access via HuggingFace: 🤗 UltraFeedback
• Reward model (UltraRM) and critique model (UltraCM) also available on HuggingFace.
• Requires Python and standard data science libraries for processing.

Highlighted Details

• Features 256k responses with fine-grained annotations across 4 aspects.
• Includes UltraRM and UltraCM, reward and critique models trained on the dataset, achieving SOTA performance.
• Utilizes a diverse set of 17 LLMs (including GPT-4, LLaMA family, Falcon, etc.) for response generation.
• Incorporates 6 diverse data sources for prompt collection (UltraChat, ShareGPT, Evol-Instruct, etc.).

Maintenance & Community

• Actively maintained with recent updates addressing data quality issues.
• Associated with OpenBMB research group.

Licensing & Compatibility

• Dataset is released under a permissive license, suitable for commercial use.
• Models (UltraRM, UltraCM) are typically released under Apache 2.0.

Limitations & Caveats

While GPT-4 annotations are used, the project acknowledges that GPT-4 can still make mistakes, potentially impacting data quality. The dataset is primarily focused on single-turn interactions, with multi-round dialogue extensions planned.