BiLLa is an open-source, bilingual (Chinese/English) LLaMA model designed to enhance reasoning capabilities while preserving English performance. It targets researchers and developers seeking improved Chinese understanding and task-solving logic in LLMs, offering a foundation for more capable AI applications.

How It Works

BiLLa undergoes a three-stage training process. First, it's pre-trained on a mix of Chinese (Wudao), English (PILE), and translation corpora to expand its Chinese vocabulary and understanding. The second stage incorporates task-specific data (math, reading comprehension, code generation, etc.) with ChatGPT-generated explanations, aiming to strengthen the model's grasp of problem-solving logic. The final stage fine-tunes the model using conversational formats of task data and additional instruction datasets (like Alpaca, Dolly 2.0), with full parameter updates throughout.

Quick Start & Requirements

• Model Conversion: Requires owning the original LLaMA model. Use embedding_convert.py to merge BiLLa weights with LLaMA weights.

  python3 embedding_convert.py \
      --model_dir /path_to/BiLLa-7B-SFT \
      --meta_llama_pth_file /path_to/LLaMA/llama-7b/consolidated.00.pth
  

• Inference: Refer to eval_codes/get_model_answer.py for usage examples.
• Dependencies: Python, PyTorch. Specific versions not detailed.
• Resources: Requires LLaMA base model access.

Highlighted Details

• BiLLa-7B-LLM shows significantly better Chinese/English language modeling than Chinese-LLaMA-7B.
• BiLLa-7B-SFT demonstrates superior Chinese reasoning capabilities compared to models like BELLE-LLaMA-Ext-7B.
• GPT-4 evaluations indicate BiLLa-7B-SFT outperforms ChatGLM-6B on English instructions, with comparable Chinese performance but higher scores in problem-solving and coding.
• The project has released training code and data.

Maintenance & Community

• The project is primarily maintained by a single author, with irregular responses to issues.
• A FAQ and "Model Limitations" section are available.
• Links to training code and evaluation scripts are provided.

Licensing & Compatibility

• License: Model weights are released under a non-commercial license due to LLaMA's original license restrictions. The training code and data may have different terms.
• Commercial Use: Explicitly prohibited.

Limitations & Caveats

BiLLa has not undergone RLHF, potentially impacting generalization. Its reasoning focus may have come at the expense of general knowledge, common sense, and real-time information recall. Multi-turn conversation capabilities are noted as potentially weak due to the training data composition. The model may generate harmful content.