Event Driven Automation Meetup May 14/2015
- 2. About myself
• Past:
– Opalis Software (now aka M$ SC Orchestrator)
– VMware
• Present:
– StackStorm CTO & co-founder
– Mistral core team member
– I don’t ops (but most Stormers do)
- 18. Solution
• Event Driven Automation – with modern twist
– FBAR (saving 1532 hours/day)
– Salt Conf - Event Driven Infrastructure
– Microsoft – new Azure Automation (RunBooks)
- 22. Zoom to Workflow, and Get Practical
• From now on I focus on workflow
• Reminder: EDA != Workflow, but Workflow is a
big part of it.
- 23. Patterns vs Practice
• ~100 patterns
http://www.workflowpatterns.com/
• Practice – IMAO: only few sufficient
• Workflow do two things well:
– Keeps state
– Carry data across systems
- 25. Basic: Data Passing
t1.code=0
msg=“Some string..”
t1 t2
examples.data_pass:
input:
- host
tasks:
t1_diagnose:
action: diag.run_mysql_diag
input:
host: <% $.host %>
publish:
- msg: <% t1_diagnose.stdout.summary %>
on-complete: t2_cleanup_logs
t2_post_to_chat:
action: chatops.say
input:
header: Returned <% $.t1_diagnose.code %>
details: <% $.msg %>
- 27. Basic: Conditions on Data
t1
t3
t2
t1_diagnose:
action: ops.run_mysql_diag
publish:
- code: <% t1_diagnose.return_code %>
on-complete:
- t2_post_to_chat: <% $.code == 0 %>
- t3_page_mysql_admin: <% $.code > 0 %>
t2_post_to_chat:
action: chatops.say
input:
header: "mysql checked, OK"
t3_page_mysql_admin:
action: pagerduty.launch_incident
input:
details: Have to wake up dude...
details: <% $.t1_diagnose.stdout %>
t1.code==0
t1.code >0
- 29. More: Parallel Execution
t1
t4
t2
...
t1_do_build:
action: cicd.do_build_and_packages
on-success:
- t2_test_ubuntu14
- t3_test_fedora20
- t3_test_rhel6
t2_test_ubuntu14:
action: cicd.deploy_and_test distro="UBUNTU14"
t3_test_fedora20:
action: cicd.deploy_and_test distro="F20"
t4_test_rhel6:
action: cicd.deploy_and_test distro="RHEL6"
t3
- 32. More: Join – Simple Merge
t5
t4
t2
...
t2_test_ubuntu14:
action: cicd.deploy_and_test distro="UBUNTU14”
on-success: t5_post_status
t3_test_fedora20:
action: cicd.deploy_and_test distro="F20"
on-success: t5_post_status
t4_test_rhel6:
action: cicd.deploy_and_test distro="RHEL6"
on-success: t5_post_status
t5_post_status:
action: chatops.say
input:
header: Test completed!
t3
http://www.workflowpatterns.com/patterns/control/basic/wcp5.php
Simple Merge
t5
t5
- 33. More: Join – AND Join
t5
t4
t2
...
t2_test_ubuntu14:
action: cicd.deploy_and_test distro="UBUNTU14”
on-success: t5_post_status
t3_test_fedora20:
action: cicd.deploy_and_test distro="F20"
on-success: t5_post_status
t4_test_rhel6:
action: cicd.deploy_and_test distro="RHEL6"
on-success: t5_post_status
t5_tag_release:
join: all
action: cicd.tag_release
t3
http://www.workflowpatterns.com/patterns/control/new/wcp33.php
Full AND Join
- 34. More: Join - Discriminator
t5
t4
t2
...
t2_test_ubuntu14:
action: cicd.deploy_and_test distro="UBUNTU14”
on-failure: t5_report_and_fail
t3_test_fedora20:
action: cicd.deploy_and_test distro="F20"
on-failure: t5_report_and_fail
t4_test_rhel6:
action: cicd.deploy_and_test distro="RHEL6"
on-failure: t5_report_and_fail
t5_report_and_fail:
join: one
action: chatops.say header=“FAILURE!”
on-complete: fail
t3
http://www.workflowpatterns.com/patterns/control/advanced_branching/wcp9.php
Discriminator
- 35. More: Multiple Data
t1 t2
ip_list=[...]
...
t1_get_ip_list:
action: myaws.allocate_floating_ips num=4
publish:
- ip_list: <% $.t1_get_ip_list.ips %>
on-complete: t2_create_vms
t2_create_vms:
with-items: ip in <% $. ip_list %>
action: myaws.create_vms ip=<% $.ip %>
- 36. And More Details…
• Nesting
– Nothing to say except
– Input and output
– Nested workflow is an action, not a task
• Retries, Waits, Pause/Resume
• Default task policies
- 38. What else
• Other than pattern support:
• Reliability
• Manageability – API, CLI, DSL, infra as code…
• Good to have: good GUI
- 39. Summary
• Event Driven Automation is coming back
– with a new twist
• EDA > Workflow,
but Workflow is a key component
• Shameless plug
StackStorm is covering it all
- 40. • OpenSource Event Automation Platform
• Github: github.com/stackstorm/st2
• Twitter: Stack_Storm
• IRC: #stackstorm on FreeNode
• www.stackstorm.com
Editor's Notes
- Why listen to me…
Created one of the legacy RunBook automation products
Currently, I am set to fix my past mistakes core member of Mistral team
- All started with Business Process Automation
- Applied software to business
BPM come to life
Body of Comp Sci research on Workflow dated late 90s. Petri-net, math, workflow nomenclature, definitions, pattersn – all started there.
-
Tibco – who was - apply to IT systems? Enterprise message bus… IT automation
- Others picked up the idea,
Run Book Automation
- Servers took days to deploy (and tickets were the say to go)
Docker deploys at split seconds
Speed is addictive – we now hate JIRA and love Slack and Chatops
- Tools – ways more
- Tools – ways more
- Close the loop:
O.O.D.A
- Why workflows are better than scripts –leave the proof to the reader as an exercise, actually Brian covered it
- Walk you through these pattersns, show Mistral as Example
- Pre-conditions, post conditions
- Pre-conditions, post conditions
For simple case both work, for advanced patterns – more/less friendly.
- Example: run full deployment and e2e tests on 3 platforms
You can do it sequentually but it takes forever.
- How many times t5 is gonna run?
- How many times t5 is gonna run?
- How many times chatops_say is gonna run?
- How many times t5 is gonna run now? Once!
- How many times t5 is gonna run now? Once!
-
Cool: Watch, ma, the multi-data are running in parallel! And the final data
Check concurrency
- There are few more nuances within these patternns
Which in the interest of time, I just mention in passing:
- This is the minimal set that gives enough power but keeps it simple to create, track, and reason.