GeoGPT provides specialized large language models designed to advance geosciences research. Targeting scientists, researchers, and professionals, these models offer enhanced capabilities in geoscience domains by leveraging state-of-the-art foundation models and tailored post-training processes, promoting open science principles within the global geosciences community.

How It Works

GeoGPT models are built upon foundation models like Llama3.1-70B and Qwen2.5-72B. They undergo a three-stage post-training process: Continual Pre-training (CPT) on diverse geoscience corpora, Supervised Fine-tuning (SFT) using geoscientist-labeled QA pairs and corpus-generated data, and Direct Preference Optimization (DPO) for human preference alignment. Training data includes a geoscience subset of CommonCrawl and open-access publications licensed under CC BY or CC BY-NC, ensuring specialized knowledge integration.

Quick Start & Requirements

Models can be loaded using the transformers library in Python. The provided code snippets demonstrate loading models like Llama3.1-70B-GeoGPT and Qwen2.5-72B-GeoGPT. Key dependencies include transformers and torch. Due to the 70B/72B parameter sizes, significant GPU memory and computational resources are required for inference.

Highlighted Details

• Released comprehensive datasets: GeoGPT-CoT-QA, GeoRAG-QA, and GeoGPT-QA, supporting geoscience NLP tasks.