How large language models are trained.
- This discussion will be using LLAMA model from meta.
- The training process is a multistage endeavor involving immense computational power and carefully curated datasets.
- Pre-training on a Massive Scale
- Refining the knowledge: Finetuning for specific taks
- Stage 1: The foundational pre-training:
- This is most computationally intensive phase
- Stage 1.1: Data Curation & prepration: For LLAMA this involved a carefully curate mix of publically available data sources
- Common Crawl: A massive data set containing raw webpage data.
- C4: A cleaned version of Common Crawl
- Github
- Books: (Gutenberg & book3)
- ArXiv
- Stack Exchange
- Stage 1.2: Tokenization:
- LLAMA uses Byte-pair Encoding Tokenizer.
- Stage 1.3: Transformer: use decoder only transformers
- Stage 1.4: Training objective: Next token prediction
- Stage 2: Fine tune for usefulness and safety
- Stage 2.1 Supervised Fine Tuning / Instruction Tuning: In this pahase the model is trained on smaller high quality dataset to instruction, input and desired output
- Stage 2.2 RLHF: multi step process:
- Collecting human preference data
- Training a Reward Model
- Fine tuning with Reinforcement learning
Pre-trained models vs Instruct Models
- In the world of LLMs the terms
pre-trained&instructrefer to two distinct and sequential stages of models development - pre-trained model (base model): This is direct result of initial large scale training phase
- Analogy: Think of this as a brilliant but socially awkward scholar who has read every book in a library.
- When to use:
- For research purposes to understand core capabilities
- when we need a text completion engine & can guide it using few-shot prompts
- Instruct models: This model undergoes the crucial second stage of training called as fine-tuning (instruction tuning). This process teaches the model to follow user commands & be a helpful assistant
- Analogy: This is the same scholar from before but have gone through extensive training on how to be an helpful assitant
- When to use instruct models:
- For most practical applcations such as chatbots, virtual assitants, content creation etc
- when we need a model that can reliably follow instructions & engage in conversations
- For task such as summarization, translations, question answering & creative writing
LoRA
-
In LoRA (Low-Rank Adaption), the core idea is fine tuing where instead of updating all weights of LLM you train a small trainable adapter matrices to the frozen weight matrices
-
Rank controls the dimesionality of the low rank update
- Alha is scaling factor of the Lora update
