SpanBERT is a research paper implementation for improved pre-training
Top 41.2% on sourcepulse
SpanBERT provides code and pre-trained models for improving BERT's performance by representing and predicting spans, targeting NLP researchers and practitioners. It offers significant performance gains on tasks like question answering and relation extraction compared to standard BERT.
How It Works
SpanBERT introduces novel pre-training objectives: Masked Language Span Prediction (MLSP) and Inter-Span Distance Prediction (IDP). MLSP masks contiguous spans of tokens and trains the model to predict the entire masked span, encouraging better contextual understanding. IDP helps the model learn relationships between spans, improving performance on tasks requiring reasoning about span relationships.
Quick Start & Requirements
pip install apex
(specific commit required: NVIDIA/apex@4a8c4ac)../code/download_finetuned.sh <model_dir> <task>
.Highlighted Details
Maintenance & Community
Licensing & Compatibility
Limitations & Caveats
2 years ago
1 week