KoGPT2 is a Korean language model designed to overcome the limitations of existing Korean NLP models, offering improved performance for text generation, classification, and regression tasks. It is targeted at researchers and developers working with Korean text data.

How It Works

KoGPT2 is a decoder-only transformer model, similar to GPT-2, trained on over 40GB of Korean text data. It utilizes a Character BPE tokenizer with a vocabulary size of 51,200, augmented with common emojis and emoticons to enhance recognition. The model also includes unused tokens for custom task-specific definitions.

Quick Start & Requirements

• Install via Hugging Face transformers library: pip install transformers
• Model loading: from transformers import GPT2LMHeadModel, PreTrainedTokenizerFast
• Requires Python 3.6+ and PyTorch.
• Official demo and documentation links are available in the README.

Highlighted Details

• Achieves 89.1% accuracy on NSMC (sentiment analysis) and 77.8 Spearman correlation on KorSTS (semantic textual similarity).
• Trained on diverse Korean datasets including Korean Wikipedia, news articles, and the "Modu-deul-mal-corpus v1.0".
• Tokenizer supports emojis and custom unused tokens.
• Model size is 125M parameters with 12 layers.

Maintenance & Community

• Developed by SKT AI.
• Issues can be reported via the provided link in the README.

Licensing & Compatibility

• Licensed under CC-BY-NC-SA 4.0.
• Non-commercial use and share-alike restrictions apply. Commercial use requires adherence to the full license terms.

Limitations & Caveats

The CC-BY-NC-SA 4.0 license restricts commercial use and requires derivative works to be shared under the same license, which may not be suitable for all commercial applications.