CBLUE provides a comprehensive benchmark for Chinese biomedical language understanding, targeting researchers and developers in the medical AI field. It offers datasets, baseline models, and an evaluation platform for eight distinct Natural Language Understanding (NLU) tasks, aiming to accelerate AI research and application in healthcare by establishing standardized evaluation metrics and facilitating model comparison.

How It Works

The benchmark comprises eight Chinese biomedical NLU tasks: Named Entity Recognition (NER), Relation Extraction (RE), Diagnosis Normalization, Sentence Classification (two tasks), Sentence Similarity, Natural Language Inference (two tasks). It supports various Chinese pre-trained language models (e.g., BERT, RoBERTa, ALBERT, MacBERT) and provides task-specific data processors, datasets, and trainers built on PyTorch and Huggingface Transformers.

Quick Start & Requirements

• Installation: Clone the repository and install dependencies: pip install -r requirements.txt.
• Prerequisites: Python 3, PyTorch (>= 1.7), Huggingface Transformers (>= 4.5.1), jieba, gensim, sklearn.
• Data: Download datasets from the repository.
• Running: Utilize provided shell scripts (examples/run_{task}.sh) for training and prediction, or adapt the baseline scripts (baselines/run_classifier.py, etc.).
• Resources: Training requires significant computational resources (GPU recommended) and time, depending on the model and task.
• Documentation: English | 中文说明

Highlighted Details

• Includes 8 diverse biomedical NLU tasks with detailed dataset statistics and evaluation metrics.
• Provides baseline performance results for 11 Chinese pre-trained models across all tasks.
• Offers a format checker to ensure correct submission of prediction results.
• Supports a wide range of Chinese biomedical language models.

Maintenance & Community

The project is associated with the ACL 2022 conference. Further community engagement details (e.g., Discord/Slack) are not explicitly mentioned in the README.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking would require clarification of the license terms.

Limitations & Caveats

The README indicates that state-of-the-art neural models perform significantly worse than the human ceiling on these tasks, suggesting substantial room for improvement. Specific licensing terms for commercial use are not detailed.