Principal Scientist and Sr.Manager of Applied Science - Amazon AGI Foundation Models
AI at Meta has released the most powerful open-source model yet as of today: ๐ณ๐ณ๐จ๐ด๐จ3 - 8B and 70B models at 8k context length! They highlight improvements in each of the following aspects as the key differentiator: a) Model Architecture b) Pretraining data c) scaling up pre-training d) Instruction fine-tuning ๐ฅ ๐๐ซ๐๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ & ๐๐๐๐ฅ๐ข๐ง๐ ๐ฎ๐ฉ: โ๏ธ Vocab of 128k tokens & GQA across both 8B and 70B, 8k native context length โ๏ธ Pretrained over 15T tokens, 4 times more code than LLAMA2 โ๏ธ Multi-lingual : over 5% of pretraining data consists of high-quality non English data that covers over 30 languages: don't expect same performance as English โ๏ธ LLAMA2 generates training data (like in Self-Instruct) to generate text classifiers that filter data powering LLAMA3 โ๏ธ Both 8B and 70B improve even after the model is trained on 2 orders of magnitude more data i.e log-linear improvement after 15T tokens training โ๏ธ Training runs on 2 custom-built 24k GPU clusters with an overall training time efficiency of 95% โ๏ธ ๐๐ท๐ฆ๐ณ๐ข๐ญ๐ญ, 3 ๐ต๐ช๐ฎ๐ฆ๐ด ๐ฎ๐ฐ๐ณ๐ฆ ๐ฆ๐ง๐ง๐ช๐ค๐ช๐ฆ๐ฏ๐ต ๐ต๐ฉ๐ข๐ฏ ๐๐๐๐๐2 ๐ฅ ๐ฐ๐๐๐๐๐๐๐๐๐๐ ๐ญ๐๐๐๐๐๐๐๐๐: โ๏ธ Combination of SFT + PPO + DPO + Rejection Sampling โ๏ธ ๐ธ๐๐๐๐๐๐ ๐๐ ๐๐๐๐๐๐๐ ๐๐๐๐ ๐๐ ๐บ๐ญ๐ป, ๐๐๐๐๐๐๐๐๐๐ ๐๐๐๐๐๐๐๐ ๐๐๐๐ ๐๐ ๐ซ๐ท๐ถ, ๐ท๐ท๐ถ ๐๐๐ ๐๐ ๐๐๐๐๐๐๐๐ ๐๐๐๐๐๐ ๐๐ ๐๐๐ ๐๐ ๐๐๐๐๐๐๐๐๐๐๐ โ๏ธ Careful data curation and multiple rounds of QA on annotations from human annotators โ๏ธTraining on preference ranking improved the model performance greatly on coding and reasoning tasks ๐ฅ ๐ณ๐ณ๐จ๐ด๐จ3 ๐ฎ๐๐๐๐ ๐๐๐๐๐: โ๏ธ Updated and new safety tools : LLAMA Gaurd2 & cybersecEval 2 โ๏ธ CodeShield - inference time gaurdrail for filtering insecure code โ๏ธ Instruction Fine-tuned model redteamed for safety : By generating adverserial prompts that try to elicit problematic responses. โ๏ธ LLAMAGaurd - foundational for prompt and response safety, can be easily finetuned to create a new taxonomy ๐ฅ ๐ฐ๐๐๐๐๐๐๐๐ & ๐ญ๐๐๐๐๐ โ๏ธ Despite LLAMA3 having 1B more params than LLAMA2 7B, the improved tokenizer efficiency and GQA contribute to maintaining the inference efficiency on par with Llama 2 7B. โ๏ธ 400B+ parameters, Multilingual, Multimodal, and longer context window LLAMA3 would be available on AWS soon. Try it out! #generativeAI #llm #llama3 #aws #bedrock #sota
Principal Scientist and Sr.Manager of Applied Science - Amazon AGI Foundation Models
3moMedium: https://medium.com/@sravanbabubodapati/llama3-highlights-4725f849e3a4