Training Deep Learning Models on Multiple GPUs in the Cloud by Enrique Otero at Big Data Spain 2017
•
1 like•1,576 views
This document discusses training deep learning models using multiple GPUs in the cloud. It covers the challenges of distributed training including data bottlenecks, communication bottlenecks, and scaling batch sizes and learning rates. It provides benchmarks for frameworks like MXNet and TensorFlow on AWS and discusses the impact of infrastructure like GPU type and interconnect bandwidth on training performance and efficiency. It also analyzes the costs of using different cloud platforms for deep learning training.
Report
Share
Training Deep Learning Models on Multiple GPUs in the Cloud by Enrique Otero at Big Data Spain 2017
- 2. DEEP LEARNING & MULTI GPUs Training Deep Learning Models on Multiple GPUs in the Cloud BEE PART OF THE CHANGE Avenida de Burgos, 16 D, 28036 Madrid hablemos@beeva.com www.beeva.com
- 3. ENRIQUE OTERO enrique.otero@beeva.com @beevalabs_eom Data Scientist in BEEVA hablemos@beeva.com | www.beeva.com The intro: deep learning & GPUs The training: challenges & benchmarks on image classification The lessons: science, engineering, infrastructure & business
- 4. WWW.BEEVA.COM 3 BIG DATA CLOUD COMPUTI NG MACHINE INTELLIGE NCE ● INNOVATION LABS ● INNOVATION SERVICES 100 % +40 %Annual growth rate in last 4 years +65 0Employees in Spain +80 0Employees globally WE MAKE COMPLEX THINGS SIMPLE
- 5. 4 Deep Learning disruption ● Computer Vision ● Speech Recognition ● Machine Translation ● Ranking
- 6. ● more (labeled) data ● more computing power ● some tricks Why now? Source: http://yann.lecun.com/exdb/lenet/
- 8. 7 1º 4º 2º 3º shaders & hacks OpenGL, DirectX GPUs. Nvidia & CUDA
- 9. THANK YOU GAMERS!THANK YOU GAMERS!
- 10. Training times vs. accuracy Accelerate training is essential! Source: https://github.com/sailing-pmls/pmls-caffe/ Source: Canziani et al 2017
- 11. ● Stochastic Gradient Descent (SGD) ● Mini-batch SGD Source: Andrew Ng. Source: http://www.eeng.dcu.ie/~mcguinne/ Error (loss) function Stochastic gradient descent
- 12. Data parallel vs. model parallel ● Faster or larger models? Asynchronous vs. Synchronous ● Fast or precise? Distributed training Source: https://github.com/tensorflow/models/tree/master/research/inception
- 13. (Multi-node) third party benchmarks Seems quite good!! Easy? Source: https://chainer.org
- 14. (Multi-node) third party benchmarks ResNet152 (8 to 256 gpus): 95% to 90% efficiency AlexNet (8 to 256 gpus): 78% to 53% efficiency Source: mxnet on AWS 16 x p2.16x
- 15. (Multi-node) third party benchmarks Small print: ● High speed connections! ● Synthetic data vs. real data ● Bottlenecks in hard disk And more... ● accuracy penalization ● number of parameter serversSource: tensorflow.org Source: https://chainer.org
- 16. (Multi-node) third party benchmarks So let’s start single-node multi-gpu in
- 17. Let’s begin: Tesla K80 K80 GPUs on: ● AWS p2: 1, 8 & 16 ○ ready-to-go AMIs ● Azure NC: 1, 2 & 4 ● Google Cloud Platform: 1 to 8 ○ setup scripts
- 18. Let’s begin: MNIST & cloud
- 19. ● Goal: saturate GPUs! ● Bottlenecks: ○ I/O Reads ○ Image pipelines ■ Decoding ■ Data augmentation ○ Communications: ■ efficient primitives: NCCL ■ Overlap with computation ■ QPI < PCIe < NVLink Data pipeline bottlenecks
- 20. (internal) interconnections matter Azure NC24 (4 K80): Google n1- highmem-32 with 8 K80:
- 21. GPUDirect & PCIe GPUDirect support (internal) Interconnections matter
- 22. GPUDirect & PCIe GPUDirect support (internal) Interconnections matter
- 24. More lessons: CIFAR10 on AWS AWS p2.8x = 8x GPU K80 sync. data-parallel After 8 epochs... mxnet: ● validation accuracy = [0.77, 0.82] tensorflow: ● validation accuracy = [0.47, 0.59]
- 25. Batch sizes matter ● Larger batches reduce communication overhead ○ More throughput ● But degrade convergence. ○ Less accuracy!
- 27. Accuracy vs. throughput Empirical workaround for learning rates: ● warm up: start small ● increase 5 epochs... ● finish on #gpus x lr Scenario: ● NVLink, 50Gbps Ethernet (> 15Gbps) ● Caffe2 + NCCL + gloo ● Synchronous SGD + Momentum Source: Facebook AI Research
- 28. Being practical: fine tuning with MxNet Source: https://mxnet.incubator.apache.org/how_to/finetune.html
- 29. Being practical: fine tuning with MxNet Scenario: ● p2.8x ● ResNet50 ● batch-size: 16 x gpu ● lr = lr_i x gpu ● 1 epoch 94% efficiency :)
- 30. Being practical: fine tuning with MxNet Scenario: ● p2.8x ● ResNet152 ● batch-size: 16,32 x gpu ● lr = lr_i x gpu ● 1 epoch ● val-acc = [0.830, 0.836] 95% efficiency :) < 1% accuracy loss
- 32. Tesla K80 price on premises Source: amazon.com November 2017
- 33. Tesla K80 prices on cloud 1$/h per-second billing only 0.3$/h on AWS spot market Purchase 1 or rent 4000 to 12000 hours! Training ResNet50 Imagenet1K (100 epochs): 180$ to 730$ Fine-tuning (8 epochs): < 2$
- 34. 2014 to 2017: from Kepler... to Volta! Source: aws.amazon.com New! October 2017 And Tesla Pascal P100 beta on Google Cloud Platform New! September 2017. on-demand spot
- 36. Extra: NVIDIA Volta on AWS P3 instances! • Great performance! • Cost-effective (on- demand) • (still) scarce availability
- 37. Summary SCIENCE Batch sizes & learning rates matter! ● high batch sizes degrade convergence ● linear scaling rule & warm-up ENGINEERING Data pipeline matters! ● Data feed ● Overlap computation & communications INFRASTRUCTURE Architecture & Bandwidth matters! ● Volta > Pascal > Kepler ● NVLink > PCle > (25 Gbps) Ethernet BUSINESS Pricing matters! ● Cost effective cloud instances in spot market
- 38. THANKS FOR YOUR TIME hablemos@beeva.com | www.beeva.com And we’re hiring!