This repository provides inference code and configurations for Replit's family of code-focused large language models, ReplitLM. It targets developers and researchers looking to leverage or fine-tune code generation models, offering integration with Hugging Face Transformers and MosaicML's LLM Foundry for advanced training.

How It Works

ReplitLM models are designed for code understanding and generation. The repository facilitates their use via Hugging Face Transformers, allowing direct loading and inference. For fine-tuning and further training, it strongly recommends MosaicML's LLM Foundry and Composer, which provide optimized training pipelines, state-of-the-art techniques, and PyTorch-based components for efficient model adaptation on custom datasets.

Quick Start & Requirements

• Inference: Models are available on Hugging Face (replit/replit-code-v1-3b). Use with Hugging Face Transformers library.
• Training: Requires LLM Foundry and Composer installation. Dataset conversion to Mosaic StreamingDataset format is necessary.
• Prerequisites: Python, PyTorch, Hugging Face libraries. LLM Foundry setup is recommended via Docker. Specific requirements are detailed in requirements.txt.
• Links: Hosted Demo, LLM Foundry, Composer.

Highlighted Details

• replit-code-v1-3b model available, with v1_5 coming soon.
• Models trained on a mixture of 20 languages, with a strong emphasis on programming languages like Python, JavaScript, Java, and Markdown.
• Detailed guides for instruction tuning using Hugging Face Transformers (Alpaca-style) and LLM Foundry.
• Workaround provided for saving checkpoints with tokenizers that include .py files when using Composer.

Maintenance & Community

The project is actively updated by Replit. Further community interaction details (e.g., Discord/Slack) are not explicitly mentioned in the README.

Licensing & Compatibility

• Model Checkpoints & Vocabulary: CC BY-SA 4.0
• Code: Apache 2.0
• The CC BY-SA 4.0 license for models requires attribution and sharing of derivatives under the same license, which may have implications for commercial use or closed-source integration.

Limitations & Caveats

The replit-code-v1_5-3b model is listed as "Coming Soon." A workaround is required for saving checkpoints with certain tokenizers when using LLM Foundry/Composer, indicating potential integration friction. The CC BY-SA 4.0 license for models may restrict commercial applications.