Routage à grande échelle des requêtes via RabbitMQ
•
0 likes•128 views
The document describes the distributed queuing system used to process tasks in Instances' control plane. Messages from APIs are routed through RabbitMQ exchanges and bindings to Celery worker queues. This allows scaling components independently and processing tasks asynchronously. Key-value headers and routing rules ensure messages are routed to the correct queues and workers.
Report
Share
Routage à grande échelle des requêtes via RabbitMQ
- 1. 1
- 2. Florent Rivoire DevOps engineer Team « Instances » (previously "Compute") Aime le vélo, les poissons rouges et « Plus Belle La Vie S5E17» (en VOST seulement !) A toujours rêvé de rencontrer M. Propre en vrai ! 2
- 4. 4 Inside Instances’ control plane
- 5. 5 Lots of components communicating
- 6. 6 Including 1000s of hypervisors & VMs HV HV VMs VMs HV VMs …...
- 7. 7 A lot of messages to route HV HV VMs VMs HV VMs …...
- 10. Web console APIs Client-side Instances control plane DBs & auth
- 11. Web console APIs Worker dispatcher Client-side Instances control plane DBs & auth
- 12. Web console APIs Worker dispatcher Client-side Instances control plane DBs & auth Start instance XYZ
- 13. Web console APIs Worker dispatcher Client-side Instances control plane DBs & auth Start instance XYZ
- 14. Queuing 101 14 • Asynchronous processing → « instant » API response • Decoupling of components (smaller) → maintainability, testing • Allow scaling each layer independently → performance, monitoring
- 15. Our queuing stack 15 Celery RabbitMQ Python lib Daemon (erlang) opensource opensource
- 16. RabbitMQ, overview 16 • message broker • accepts, stores and forwards messages • « like a post office »
- 17. RabbitMQ, overview 17 node RabbitMQ cluster Producer ConsumerMessage Messages Messages sorted in « queues »
- 18. Celery 18 • async task queue model • similar to RPC : run a function (task) on a remote worker • based on distributed message passing (like RabbitMQ) • can create complex tasks graph (sequential and/or parallel tasks, error callback, retry, etc.)
- 19. Web console APIs Worker dispatcher Client-side Instances control plane DBs & auth Start instance XYZ
- 20. Worker "dispatcher" 20 • Handling allocation of instances, orchestration of tasks • 1 logical worker • but several "physical" copies (for HA and scaling) • processing one « big » pool of tasks for the whole AZ → simple queuing: 1 queue « dispatcher »
- 21. [import+config of celery] celery.start() @celery.task() def poweron_server(server_id, model): physical_node = allocate(model) prepare(server_id, physical_node) ——————————————————————————————————————————————— celery.send_task("poweron_server", kwargs={'server_id': 12345, 'model': 'GP1-S'} ) 21 Example: simple task Worker side API side
- 22. Web console APIs Worker dispatcher Client-side Instances control plane DBs & auth
- 23. 23 Web console APIs Worker dispatcher Client-side Instances control plane Hypervisor A Hypervisor B DBs & auth
- 24. 24 Web console APIs Worker dispatcher Client-side Instances control plane Hypervisor A VMs Volumes VMs Volumes Hypervisor B DBs & auth
- 25. 25 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth
- 26. 26 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth
- 27. Hypervisor’s workers 27 • Two workers per hypervisor : 1. to manage volumes (local-storage) 2. to manage VMs (start, stop, etc.) • processing only tasks related to this HV precisely → not so common routing pattern
- 28. 28 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth
- 29. 29 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth
- 30. 30 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth download volume 5643 poweron VM 431
- 31. 31 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth
- 32. 32 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth RabbitMQ
- 33. 33 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth RabbitMQ
- 34. 34 Cluster RabbitMQ API & worker dispatcher RabbitMQ routing
- 35. 35 Cluster RabbitMQ API & worker dispatcher RabbitMQ routing Worker dispatcher Worker « vm » HV-A Worker « vm » HV-B Worker « storage » X
- 36. 36 Queue Queue Queue Cluster RabbitMQ API & worker dispatcher RabbitMQ routing Worker dispatcher Worker « vm » HV-A Worker « vm » HV-B Queue Worker « storage » X
- 37. 37 Queue Queue Queue Cluster RabbitMQ API & worker dispatcher RabbitMQ routing to queues Worker dispatcher Worker « vm » HV-A Worker « vm » HV-B ? Queue Worker « storage » X
- 38. 38 Which worker ? Which cluster ? dispatcher storagevm Which cluster ? dispatcher vm 53-11 Which HV ? Which HV ? Which HV ? Which HV ? Which HV ? 53 54 55 53 54 11 12 vm 53-12 Queue Value Criteria vm 54-98 98 99 vm 53-99 …. …. …. Logical routing tree
- 39. Routing challenge 39 • Need routing to proper destination • Simple to use • Reasonable speed (msg/seconds) • But a lot of queues/connections : - 1000s - growing with nb of HVs
- 40. Our solution 40 Use RabbitMQ features at maximum : • « bindings »: routing-rules • « exchange »: router, not storing msg To route Celery messages
- 41. 41 Publish Publish Binding Binding Exchange B Binding Exchange A Binding Binding Exchange C Queue Y Queue Z Queue X Cluster RabbitMQ M M M M M M RabbitMQ, inside the cluster
- 42. 42 Description Example routing-key Scalar value (ASCII) celery-task [not used] headers Key/value pairs - daemon => storage - cluster => 54 - hypervisor => 98 content_type Format of the body application/json body Free format (never read by RabbitMQ) { id: "1326d7f2-fc36-4271-…", task: "download_volume", kwargs: {volume_id: 56789, verify_chksum: True} } RabbitMQ, message structure
- 43. RabbitMQ, binding 43 • 1 src exchange • some condition: - routing-key match - one key/value match • 1 target (exchange / queue) Multiple bindings possible (« many to many »)
- 44. RabbitMQ, binding 44 • 1 src exchange • some condition: - routing-key match - one key/value match • 1 target (exchange / queue) Multiple bindings possible (« many to many ») Example : • msg in exchange « global » • if match: - header « daemon » - is equal to « dispatcher » • route to queue « dispatcher »
- 45. 45 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth RabbitMQ
- 46. Code example 46 Using our "scw-routed-rpc" library • simple wrapper around Celery • simplify configuration of routing : - exchanges - bindings - queues
- 47. [import+config of scw_routed_rpc] scw_routed_rpc.send_task( "download_volume", kwargs={'volume_id': 56789, 'verify_chksum': True}, headers={ 'daemon': 'storage', 'cluster_id': 54, 'hypervisor_id': 98, }, ) 47 Example: send a task
- 48. [import+config of scw_routed_rpc] scw_routed_rpc.send_task( "download_volume", kwargs={'volume_id': 56789, 'verify_chksum': True}, headers={ 'daemon': 'storage', 'cluster_id': 54, 'hypervisor_id': 98, }, ) 48 Example: send a task
- 49. [import+config of scw_routed_rpc] scw_routed_rpc.start({'daemon': 'storage', 'cluster_id': 54, 'hypervisor_id': 98}) @scw_routed_rpc.task() def download_volume(volume_id, verify_chksum): transfer_file(volume_id) finalize_volume(volume_id, verify_chksum) @scw_routed_rpc.task() def upload_volume(volume_id, foo): [...] 49 Example: worker storage
- 50. [import+config of scw_routed_rpc] scw_routed_rpc.start({'daemon': 'storage', 'cluster_id': 54, 'hypervisor_id': 98}) @scw_routed_rpc.task() def download_volume(volume_id, verify_chksum): transfer_file(volume_id) finalize_volume(volume_id, verify_chksum) @scw_routed_rpc.task() def upload_volume(volume_id, foo): [...] 50 Example: worker storage
- 51. 51 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth RabbitMQ
- 52. 52 Queue Queue Queue Cluster RabbitMQ API & worker dispatcher RabbitMQ routing to queues Worker dispatcher Worker « vm » HV-A Worker « vm » HV-B ? Queue Worker « storage » X
- 53. 53 Queue Queue Queue Cluster RabbitMQ API & worker dispatcher RabbitMQ routing to queues Worker dispatcher Worker « vm » HV-A Worker « vm » HV-B Queue Worker « storage » X
- 54. 54 Queue Queue Queue Cluster RabbitMQ API & worker dispatcher RabbitMQ routing to queues Queue
- 55. 55 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues …. Queue ….
- 56. 56 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues …. Queue …. Exch- ange global
- 57. 57 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage"
- 58. 58 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage"
- 59. 59 Example routing-key celery-task [not used] headers - daemon => storage - cluster => 54 - hypervisor => 98 content_type application/json body { id: "1326d7f2-fc36-4271-…", task: "download_volume", kwargs: {volume_id: 56789, verify_chksum: True} } RabbitMQ, message structure
- 60. 60 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage"
- 61. 61 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage storage- 53 storage- 54 vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage" cluster="53" cluster="54"
- 62. 62 Example routing-key celery-task [not used] headers - daemon => storage - cluster => 54 - hypervisor => 98 content_type application/json body { id: "1326d7f2-fc36-4271-…", task: "download_volume", kwargs: {volume_id: 56789, verify_chksum: True} } RabbitMQ, message structure
- 63. 63 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage storage- 53 storage- 54 vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage" cluster="53" cluster="54"
- 64. 64 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage storage- 53 storage- 54 vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage" cluster="53" cluster="54" hv="99" hv="98"
- 65. 65 Example routing-key celery-task [not used] headers - daemon => storage - cluster => 54 - hypervisor => 98 content_type application/json body { id: "1326d7f2-fc36-4271-…", task: "download_volume", kwargs: {volume_id: 56789, verify_chksum: True} } RabbitMQ, message structure
- 66. 66 storage- 54-98 storage- 54-99 dispatcher Cluster RabbitMQ RabbitMQ routing to queues global storage storage- 53 storage- 54 vm …. daemon="dispatcher" daemon="vm" Exch- ange Queue binding …. daemon= "storage" cluster="53" cluster="54" hv="99" hv="98"
- 67. 67 Web console APIs Worker dispatcher Worker(s) Client-side Instances control plane Hypervisor A VMs Volumes Worker(s) VMs Volumes Hypervisor B DBs & auth RabbitMQ
- 68. Experience (1) 68 • Tuning Celery parameters → nb of process, nb of tasks/process, prefetch, etc. • Complex config (lib over-engineered at first) → feature removed: broadcast, namespace, versioning • End2end tracing (1 API-call, N tasks) → generate a « req-id » on API → correlate all tasks with the req-id
- 69. Experience (2) 69 • Too many connections on RabbitMQ → enough RAM + proper limits (erlang process, file- descriptors) • High availability control-plane → clustering RabbitMQ, multiple copies of workers • RabbitMQ management web interface → powerful, use it !!
- 70. Conclusion 70 +
- 71. MERCI Suivez notre actualité, tutoriels inédits et infos cloud sur Twitter et LinkedIn @Scaleway 71 Et retrouvez toutes les présentations du Scaleday sur Slideshare :
- 72. 72