Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated
•
1 like•1,943 views
The document discusses using multi-armed bandit tests to compare natural language models. It describes training BERT models with TensorFlow and PyTorch, and training a multi-armed bandit model with Vowpal Wabbit for reinforcement learning. It then demonstrates testing the BERT models with the bandit model and scaling multi-armed bandits on AWS.
Report
Share
Smokey and the Multi-Armed Bandit Featuring BERT Reynolds Updated
- 1. Chris Fregly Developer Advocate AI and Machine Learning @AWS Smokey and the Multi-Armed Bandit Featuring BERT Reynolds
- 2. Abstract First, I will train and deploy multiple natural language understanding (NLU) models and compare them in live production using reinforcement learning to dynamically shift traffic to the winning model. Second, I will describe the differences between A/B and multi-armed bandit tests including exploration-exploitation, reward-maximization, and regret-minimization. Third, I will dive deep into the details of building and scaling a multi- armed bandit deployment on AWS using a real-time, stream-based text classifier with TensorFlow, PyTorch, and BERT on 150+ million reviews from the Amazon Customer Reviews Dataset.
- 3. Me Developer Advocate AI and Machine Learning @ AWS (Based in San Francisco) Co-Author of the O'Reilly Book, "Data Science on AWS." Founder of the Advanced Kubeflow Meetup (Global) https://www.datascienceonaws.com data-science-on-aws @cfregly linkedin.com/in/cfregly https://meetup.com/Data-Science-on-AWS
- 4. Data Science on AWS – Book and Workshop Outline https://www.datascienceonaws.com/
- 5. Agenda • Compare A/B Tests vs. Multi-Armed Bandit Tests • Optimize Bandits with Reinforcement Learning • Train 2 BERT Languge Models with TensorFlow • Train a Multi-Armed Bandit Model with Vowpal Wabbit • Test 2 BERT Models with a Bandit • DEMO: Scale Multi-Armed Bandits on AWS
- 6. Traditional A/B Tests • Static • Cannot Add New Models After Test Begins • Static Traffic Split Between Models A and B • May Negatively Impact Business Metrics • Must Run Experiment to Completion • No Concept of Reward for Winning Model
- 7. Multi-Armed Bandit Tests • Add New Models • Dynamically Shift Traffic • Explore-Exploit Strategy • Finish Experiment Early - or Run Longer! • Minimize Regret (Business Impact) • Maximize Reward
- 8. Train 2 BERT Models with TensorFlow (Models A & B) • BERT Mania! • Fine-Tuning BERT
- 9. Train a Bandit Model with Reinforcement Learning (RL) • Popular Reinforcement Learning Strategies • Epsilon Greedy • Thompson’s Sampling • Online Cover • Bagging • Implemented in Vowpal Wabbit (VW)! • Try Our Open Source RL Containers • https://github.com/aws/sagemaker-rl-container
- 10. Test 2 BERT Models with a Multi-Armed Bandit Model
- 11. DEMO: Scale Multi-Armed Bandits on AWS
- 12. DEMO: Scale Multi-Armed Bandits on AWS • BERT Model 1: TensorFlow • BERT Model 2: PyTorch
- 13. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. DEMO!
- 14. More Resources • O’Reilly Book - Data Science on AWS – Early Release Available! • https://datascienceonaws.com • GitHub Repo • https://github.com/data-science-on-aws/workshop • AWS Blog Post on Multi-Armed Bandits • https://aws.amazon.com/blogs/machine-learning/power-contextual-bandits-using-continual-learning- with-amazon-sagemaker-rl/ • Bandit Algorithms • https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Contextual-Bandit-algorithms • Open Source SageMaker Reinforcement Learning Containers • https://github.com/aws/sagemaker-rl-container
- 15. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chris Fregly data-science-on-aws @cfregly linkedin.com/in/cfregly