Meditron offers open-source medical Large Language Models (LLMs), specifically Meditron-7B and Meditron-70B, adapted from Llama-2. These models are designed for researchers and developers working with medical data, aiming to improve performance on medical reasoning tasks and provide a foundation for specialized healthcare AI applications.

How It Works

Meditron models are created by continuing the pretraining of Llama-2 on a curated medical corpus. This corpus, GAP-Replay, combines clinical guidelines, PubMed abstracts, full-text medical papers, and general domain data. This domain-adaptive pretraining approach enhances the models' understanding and generation capabilities within the medical field, outperforming general LLMs on specific medical reasoning benchmarks.

Quick Start & Requirements

• Install: Load directly from HuggingFace using transformers.

  from transformers import AutoTokenizer, AutoModelForCausalLM
  tokenizer = AutoTokenizer.from_pretrained("epfl-llm/meditron-70b")
  model = AutoModelForCausalLM.from_pretrained("epfl-llm/meditron-70b")
  

• Prerequisites: vllm >= 0.2.1, transformers >= 4.34.0, datasets >= 2.14.6, torch >= 2.0.1.
• Resources: Meditron-70B requires significant computational resources for inference.
• Docs: Inference & Evaluation Instructions, Model Deployment

Highlighted Details

• Meditron-70B outperforms Llama-2-70B, GPT-3.5, and Flan-PaLM on multiple medical reasoning tasks.
• The training corpus, GAP-Replay, includes 46K clinical practice guidelines and 5M PubMed papers.
• Training utilized the Megatron-LLM library with a three-way parallelism scheme (DP, PP, TP).
• Models are available as foundation models for further fine-tuning or instruction-tuning.

Maintenance & Community

• Developed by the EPFL LLM Team.
• Future versions are planned as performance is enhanced.
• GitHub Repository

Licensing & Compatibility

• Model License: LLAMA 2 COMMUNITY LICENSE AGREEMENT.
• Code License: APACHE 2.0 LICENSE.
• The Llama 2 license has specific use restrictions; consult it for commercial or production use.

Limitations & Caveats

The models are not yet adapted for safe or appropriate delivery of medical knowledge in clinical practice. Use in medical applications is strongly discouraged without extensive alignment, testing, and potentially randomized controlled trials. The models are static and knowledge cutoff is August 2023.