Data Migration at Scale with RabbitMQ and Spring Integration
- 1. Data Migration at Scale with
RabbitMQ and Spring Integration
By Álvaro Videla and Félix López
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 2. Álvaro Videla
• Developer Advocate at Pivotal / RabbitMQ
• Co-Author of RabbitMQ in Action
• Creator of the RabbitMQ Simulator
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/
internals.html
• @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
2
- 3. Álvaro Videla
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
3
Co-authored
!
RabbitMQ in Action
http://bit.ly/rabbitmq
- 4. What is RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
4
- 5. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
5
- 6. RabbitMQ
• Multi Protocol Messaging Server
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
6
- 7. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
7
- 8. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
8
- 9. RabbitMQ
• Multi Protocol Messaging Server
• Open Source (MPL)
• Polyglot
• Written in Erlang/OTP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
9
- 10. Multi Protocol
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
10
http://bit.ly/rmq-protocols
- 11. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
11
- 12. Polyglot
• Java
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
12
- 13. Polyglot
• Java
• node.js
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
13
- 14. Polyglot
• Java
• node.js
• Erlang
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
14
- 15. Polyglot
• Java
• node.js
• Erlang
• PHP
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
15
- 16. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
16
- 17. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
• .Net
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
17
- 18. Polyglot
• Java
• node.js
• Erlang
• PHP
• Ruby
• .Net
• Haskell
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
18
- 19. Polyglot
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
19
Even COBOL!!!11
- 21. Messaging with RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
21
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator
- 22. RabbitMQ Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
22
- 24. RabbitMQ Basic Qos Simulator
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
24
- 25. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
25
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?
- 26. Prevent Unbound Buffers
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
26
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?
Queue length goes to infinity over time.
- 27. Prevent Unbound Buffers
λ = mean arrival time
μ = mean service rate
if λ > μ what happens?
Queue length goes to infinity over time.
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
27
https://www.rabbitmq.com/blog/2014/01/23/preventing-unbounded-buffers-
with-rabbitmq/
- 28. Recommended Reading
Performance Modeling and
Design of Computer Systems:
Queueing Theory in Action
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
28
- 29. What you get out of the box
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
29
- 30. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
30
- 31. Some RabbitMQ Features
• Persistent Messages
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
31
- 32. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
32
- 33. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
33
- 34. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
34
- 35. Some RabbitMQ Features
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
• Message Ordering
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
35
- 36. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
36
- 37. Some RabbitMQ Features
• Dead Letter Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
37
- 38. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
38
- 39. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
39
- 40. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
40
- 41. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
41
- 42. Some RabbitMQ Features
• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
• Shovel
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
42
- 43. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
43
- 44. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
44
• SSL Connections
- 45. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
45
• SSL Connections
• User Authentication
- 46. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
46
• SSL Connections
• User Authentication
• Per resource authorisation
- 47. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
47
• SSL Connections
• User Authentication
• Per resource authorisation
• Different kinds of users
- 48. Some RabbitMQ Features
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
48
• SSL Connections
• User Authentication
• Per resource authorisation
• Different kinds of users
• Various authentication mechanisms via plugins (LDAP)
- 49. Management Interface / REST API
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
49
- 50. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
50
- 51. RabbitMQ
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
51
Flexibility
- 52. Questions?
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
52
- 53. Learn More. Stay Connected
@old_sound @flopezluis
@springcentral | spring.io/video
Unless otherwise indicated, these slides are © 2013-2014 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 53