SlideShare a Scribd company logo
Stefan Vallin, Ph. D.
Principal Engineer
29 June 2016
Use Things You Can Program,
and Program the Things You Use
DevOps for
Networking
Walls of Confusion
New and Changed
Requirements
Users
Product Mgr
New/Changed App Code
Configuration Change
Application
Developers
Network
Engineers
Operations
What is DevOps?
http://dev.mobify.com/blog/devops-101-best-practices/
“The cool-kids way of
stringing stuff together with
shell scripts.
It exists because we can
now wrap things in shell
scripts that we used to only
dream of; the world is now
programmable at a much
larger scale, and we have
many new tools and
techniques for taking
advantage of it.”
Takes a new or
enhanced feature all
the way to production—
everyone is adding
value to the customer
A method that stresses
communication, collaboration,
integration, automation,
and cooperation across
organizational units to
deliver features quickly
and efficiently.
Based on lean and
agile principles
DevOps is the marrying
of process, infrastructure,
and product
New and Changed
Requirements
Customers
Users
New/ChangedApp Code
Configuration Change
Application
Developers
Network
Engineers
Operations
Why Should You Care?
Better Customer Experience
Faster Better
Increased capacity to innovate
Let a small team multiply
their effectiveness
Faster time to value
Fewer failures
Recover faster from failures
More predictable behavior
Why Should You Care?
We want
more change!
Dev
“
”
We want
less change!
Ops
“
”
You can have both!
Network Automation
Historically DevOps
Tools: GIT, Python, NETCONF
• An IT/OSS project
• Waterfall
• Few network domain-experts
• Driven from the network
• Agile
• Designed and implemented by network
engineers with a DevOps mentality
DevOps
Underpinning: Automation
People, Process
Culture
Practices Tools
• Sharing
• Collaboration
• Learning
• Understanding other roles
• Understanding business
• No blame-culture
People, Process, Culture
Source: IBM: “DevOps for Dummies”
People, Process
Culture
Across Roles
Product
Owners
DevOps
Team
AT&T Domain 2.0:Product Owners
Business
People
People, Process
Culture
Everyone need to change-
-not just network engineers
- Meet in same practices
- Meet in same tools
- Learn and educate
- More hands-on
• Hypothesis-driven
• Slices that can be implemented in a
week
• Teams have a good understanding
of the flow of work from the
business all the way through to
customers
• Visibility
• Regularly seek customer feedback,
and incorporate this feedback into
the design of their products.
Lean Product Management
Source: Puppe Labs: “2016 State of DevOps Report”
DevOps Networking Skills
Culture
Networking
Software
Engineering
Education ?
“ There remains much to do before this
vision can be implemented, including
pivots from networking craft to software
engineering, and from carrier operations
models to cloud “DevOps” models. ”
AT&T Domain 2.0:
People, Process
Culture
• An increased knowledge of other IT disciplines
• More knowledge of the business
• More understanding of applications
• More emphasis on programming
Changes in Network
Professionals’ Skills
Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud
People, Process
Culture
DevOps
People, Process
Culture
Practices Tools
• Infrastructure as code
• Automation
• Deploy with repeatable, reliable processes
• Self-service environments
• Automated environment for de-provisioning
• Automated recovery roll-back and roll-forward
• Hypothesis-driven development
• Dev and ops team shall use same tools
• Develop and test against production-like systems
• Amplify feedback loops
• Continuous integration
• Automated testing
• Continuous deployment
• Availability monitoring
• Monitor and validate operational quality
• Release management
• App performance monitoring
• Load-testing and auto-scale
Practices
Before After
Before and After
Tools: Ticketing, Excel, Device CLI Tools: GIT, Python, NETCONF
• Operator role picks up ticket
in ITSM tool
• Implements the configuration
change manually or pseudo-
automatically
Cut and paste snippets
• Closes ticket
• Users provision their network
services directly
• DevOps role implements automation
• What do I do manually?
• How can I automate?
• How can I validate it?
• How do I handle failures?
So DevOps is just scripting?
More than a shell script in:
~/my-cool-automation-scripts
Distribution, Execution, Monitoring, Error-Recovery, Logging,
Validation, Versioning, Feature Management, No-Hands
Open Source, Open APIs, OpenFlow,
Open vSwitch, OpenDaylight, OpenConfig…
• Things can be said about all of the above,
but clear values coming out of it
• Standard APIs like NETCONF
• Scripting on the devices
• Linux-based devices, the shell prompt
• Networking devices are now open for
programming
Enabler: The Open Networking Trend
~ Use Things You
Can Program,
and Program the
Things You Use ~
Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud
Deep-Dive: DevOps is
about Managing Change
—
What is Changing?
Separation of Concerns, Change life-cycles
Run-time fine-grained configuration
changes for individual services
• Modify QoS policy
• Add e-mail user
• Modify firewall rule
• Add backend to load-balancer
Network
Services
VNFs/VMs
Applications
Platforms
Backbone /
DC Network
Short
Lifecycle
Long
Lifecycle
Do Not
Blur/Mix
Make resources available to
provide services
• Install
• Start
• Stop
• Upgrade
• Provision L2 network in data center
connecting VNFs
Nature of Configuration Changes
• Static “Golden Configs”
• Configuration
templates with variable
substitution
• Configuration
templates with loops,
if-statements
• Scripted configuration
based on stateServices
VNFs/VMs
Applications
Platforms
Backbone /
DC Network
Short
Lifecycle
Long
Lifecycle
Do Not
Blur/Mix
Different requirements for tools
Comparison: Changes in Different Domains
Examples Changes Change Rate
Applications Latest code-base for web-shop In-house developed code.
Software changes
Daily
IT-Platforms New patch on WordPress Server
New Database Server
External Software releases and
patches.
Weekly
mysql config file Config file contents
Network Functions New VPN
Add leg to VPN
Add new VAS to VPN
Change QoS params for VPN
Configuration changes Thousands
per day
New device OS version New and upgraded devices Quarterly
DevOps
People, Process
Culture
Practices Tools
Typical Tools
Services
Applications
VNFs
VMs
Platforms
Backbone /
DC Network
Short
Lifecycle
Long
Lifecycle
• Service Orchestrators
• Scripts
• Vendor tools and UIs
• And the below
• Deployment:
• Chef, Puppet
• Juju charms, Heat
Templates, TOSCA
• Code and static configs
• GIT
• Build & Test
• Jenkins
Application Centric Stack
Applications
Stitched
VMs
• IT Tools like Chef, Puppet,
TOSCA
• IT Tools like Chef, Puppet,
TOSCA
• Fine-grained configuration
tools for networking might be
needed
Network Centric Stack
Network
Services
Virtual
Network
VNFs
• Tools like Chef, Puppet,
TOSCA not optimal
• Service-Orchestrators
• IT Tools like Chef, Puppet, TOSCA
• BUT, exposed networking
• AND Fine-tuned for packet-forwarding
DevOps
People, Process
Culture
Practices Tools
• Winner in Service Agility:
• Business Innovation and Product Owners in close co-op with DevOps team
• Data-Models and Running Code
• Sign of lagging behind
• Organization controlled by
“Architects in PowerPoint”
• OSS-projects
• Internal “purchase-orders”
Summary
Start creating your DevOps culture now
Material and Reading
• John Willis, ”The Convergence of DevOps”
• http://itrevolution.com/the-convergence-of-devops/
• Kyle Young, Mobify: “DevOps 101: Best Practices for Optimizing and Automating
Your Infrastructure"
• http://dev.mobify.com/blog/devops-101-best-practices/
• David Tesar, “DevOps Practices”
• http://www.itproguy.com/devops-practices/
• Adrian Cockcroft: “Ops, DevOps and PaaS (NoOps) at Netflix”
• http://perfcap.blogspot.com/2012/03/ops-devops-and-noops-at-netflix.html
• Ashton, Metzler & Associates: “The Changing Role of the IT & Network Professional”
• http://www.ashtonmetzler.com/Quali%20Fifth%20Paper%20V2.0.pdf
• Lori MacVittie:”Network Engineers: Don’t Fear the Code”
• http://www.networkcomputing.com/careers/network-engineers-dont-fear-code/1347793692
Material and Reading
• IBM: “DevOps for Dummies”
• https://www.ibm.com/marketing/iwm/iwm/web/signup.do?source=swg-rtl-sd-wp&S_PKG=ov18162
• Barry O’Reilly: “How to Implement Hypothesis-Driven Development”
• https://www.thoughtworks.com/insights/blog/how-implement-hypothesis-driven-development
• Julien Stroheker: “DevOps: Where do I start ? Cheat Sheet”
• https://blogs.technet.microsoft.com/juliens/2016/02/14/devops-where-do-i-start-cheat-sheet/
• Stefan Vallin and Caroline Chappel, Light Reading Upskill: “Virtualization, Automation”
• http://www.lightreading.com/virtualization-automation/l/d-id/722320
• Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud
• Puppet, “State of DevOps Report”
• https://puppet.com/resources/white-paper/2016-state-of-devops-report
DevOps for Network Engineers

More Related Content

DevOps for Network Engineers

  • 1. Stefan Vallin, Ph. D. Principal Engineer 29 June 2016 Use Things You Can Program, and Program the Things You Use DevOps for Networking
  • 2. Walls of Confusion New and Changed Requirements Users Product Mgr New/Changed App Code Configuration Change Application Developers Network Engineers Operations
  • 3. What is DevOps? http://dev.mobify.com/blog/devops-101-best-practices/ “The cool-kids way of stringing stuff together with shell scripts. It exists because we can now wrap things in shell scripts that we used to only dream of; the world is now programmable at a much larger scale, and we have many new tools and techniques for taking advantage of it.” Takes a new or enhanced feature all the way to production— everyone is adding value to the customer A method that stresses communication, collaboration, integration, automation, and cooperation across organizational units to deliver features quickly and efficiently. Based on lean and agile principles DevOps is the marrying of process, infrastructure, and product New and Changed Requirements Customers Users New/ChangedApp Code Configuration Change Application Developers Network Engineers Operations
  • 4. Why Should You Care? Better Customer Experience Faster Better Increased capacity to innovate Let a small team multiply their effectiveness Faster time to value Fewer failures Recover faster from failures More predictable behavior
  • 5. Why Should You Care? We want more change! Dev “ ” We want less change! Ops “ ” You can have both!
  • 6. Network Automation Historically DevOps Tools: GIT, Python, NETCONF • An IT/OSS project • Waterfall • Few network domain-experts • Driven from the network • Agile • Designed and implemented by network engineers with a DevOps mentality
  • 8. • Sharing • Collaboration • Learning • Understanding other roles • Understanding business • No blame-culture People, Process, Culture Source: IBM: “DevOps for Dummies” People, Process Culture
  • 9. Across Roles Product Owners DevOps Team AT&T Domain 2.0:Product Owners Business People People, Process Culture Everyone need to change- -not just network engineers - Meet in same practices - Meet in same tools - Learn and educate - More hands-on
  • 10. • Hypothesis-driven • Slices that can be implemented in a week • Teams have a good understanding of the flow of work from the business all the way through to customers • Visibility • Regularly seek customer feedback, and incorporate this feedback into the design of their products. Lean Product Management Source: Puppe Labs: “2016 State of DevOps Report”
  • 11. DevOps Networking Skills Culture Networking Software Engineering Education ? “ There remains much to do before this vision can be implemented, including pivots from networking craft to software engineering, and from carrier operations models to cloud “DevOps” models. ” AT&T Domain 2.0: People, Process Culture
  • 12. • An increased knowledge of other IT disciplines • More knowledge of the business • More understanding of applications • More emphasis on programming Changes in Network Professionals’ Skills Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud People, Process Culture
  • 14. • Infrastructure as code • Automation • Deploy with repeatable, reliable processes • Self-service environments • Automated environment for de-provisioning • Automated recovery roll-back and roll-forward • Hypothesis-driven development • Dev and ops team shall use same tools • Develop and test against production-like systems • Amplify feedback loops • Continuous integration • Automated testing • Continuous deployment • Availability monitoring • Monitor and validate operational quality • Release management • App performance monitoring • Load-testing and auto-scale Practices
  • 15. Before After Before and After Tools: Ticketing, Excel, Device CLI Tools: GIT, Python, NETCONF • Operator role picks up ticket in ITSM tool • Implements the configuration change manually or pseudo- automatically Cut and paste snippets • Closes ticket • Users provision their network services directly • DevOps role implements automation • What do I do manually? • How can I automate? • How can I validate it? • How do I handle failures?
  • 16. So DevOps is just scripting? More than a shell script in: ~/my-cool-automation-scripts Distribution, Execution, Monitoring, Error-Recovery, Logging, Validation, Versioning, Feature Management, No-Hands
  • 17. Open Source, Open APIs, OpenFlow, Open vSwitch, OpenDaylight, OpenConfig… • Things can be said about all of the above, but clear values coming out of it • Standard APIs like NETCONF • Scripting on the devices • Linux-based devices, the shell prompt • Networking devices are now open for programming Enabler: The Open Networking Trend ~ Use Things You Can Program, and Program the Things You Use ~ Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud
  • 18. Deep-Dive: DevOps is about Managing Change — What is Changing?
  • 19. Separation of Concerns, Change life-cycles Run-time fine-grained configuration changes for individual services • Modify QoS policy • Add e-mail user • Modify firewall rule • Add backend to load-balancer Network Services VNFs/VMs Applications Platforms Backbone / DC Network Short Lifecycle Long Lifecycle Do Not Blur/Mix Make resources available to provide services • Install • Start • Stop • Upgrade • Provision L2 network in data center connecting VNFs
  • 20. Nature of Configuration Changes • Static “Golden Configs” • Configuration templates with variable substitution • Configuration templates with loops, if-statements • Scripted configuration based on stateServices VNFs/VMs Applications Platforms Backbone / DC Network Short Lifecycle Long Lifecycle Do Not Blur/Mix Different requirements for tools
  • 21. Comparison: Changes in Different Domains Examples Changes Change Rate Applications Latest code-base for web-shop In-house developed code. Software changes Daily IT-Platforms New patch on WordPress Server New Database Server External Software releases and patches. Weekly mysql config file Config file contents Network Functions New VPN Add leg to VPN Add new VAS to VPN Change QoS params for VPN Configuration changes Thousands per day New device OS version New and upgraded devices Quarterly
  • 23. Typical Tools Services Applications VNFs VMs Platforms Backbone / DC Network Short Lifecycle Long Lifecycle • Service Orchestrators • Scripts • Vendor tools and UIs • And the below • Deployment: • Chef, Puppet • Juju charms, Heat Templates, TOSCA • Code and static configs • GIT • Build & Test • Jenkins
  • 24. Application Centric Stack Applications Stitched VMs • IT Tools like Chef, Puppet, TOSCA • IT Tools like Chef, Puppet, TOSCA • Fine-grained configuration tools for networking might be needed
  • 25. Network Centric Stack Network Services Virtual Network VNFs • Tools like Chef, Puppet, TOSCA not optimal • Service-Orchestrators • IT Tools like Chef, Puppet, TOSCA • BUT, exposed networking • AND Fine-tuned for packet-forwarding
  • 27. • Winner in Service Agility: • Business Innovation and Product Owners in close co-op with DevOps team • Data-Models and Running Code • Sign of lagging behind • Organization controlled by “Architects in PowerPoint” • OSS-projects • Internal “purchase-orders” Summary Start creating your DevOps culture now
  • 28. Material and Reading • John Willis, ”The Convergence of DevOps” • http://itrevolution.com/the-convergence-of-devops/ • Kyle Young, Mobify: “DevOps 101: Best Practices for Optimizing and Automating Your Infrastructure" • http://dev.mobify.com/blog/devops-101-best-practices/ • David Tesar, “DevOps Practices” • http://www.itproguy.com/devops-practices/ • Adrian Cockcroft: “Ops, DevOps and PaaS (NoOps) at Netflix” • http://perfcap.blogspot.com/2012/03/ops-devops-and-noops-at-netflix.html • Ashton, Metzler & Associates: “The Changing Role of the IT & Network Professional” • http://www.ashtonmetzler.com/Quali%20Fifth%20Paper%20V2.0.pdf • Lori MacVittie:”Network Engineers: Don’t Fear the Code” • http://www.networkcomputing.com/careers/network-engineers-dont-fear-code/1347793692
  • 29. Material and Reading • IBM: “DevOps for Dummies” • https://www.ibm.com/marketing/iwm/iwm/web/signup.do?source=swg-rtl-sd-wp&S_PKG=ov18162 • Barry O’Reilly: “How to Implement Hypothesis-Driven Development” • https://www.thoughtworks.com/insights/blog/how-implement-hypothesis-driven-development • Julien Stroheker: “DevOps: Where do I start ? Cheat Sheet” • https://blogs.technet.microsoft.com/juliens/2016/02/14/devops-where-do-i-start-cheat-sheet/ • Stefan Vallin and Caroline Chappel, Light Reading Upskill: “Virtualization, Automation” • http://www.lightreading.com/virtualization-automation/l/d-id/722320 • Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud • Puppet, “State of DevOps Report” • https://puppet.com/resources/white-paper/2016-state-of-devops-report

Editor's Notes

  1. There is a constant feed of new and changed requirements from customers and users. Normally the developers and network engineers are not in close contact with the end-users. In contrast they get new requirements thrown over the wall without interaction.   In turn: - the developers develop new apps to fulfill the requirements and throw over the wall to operations team. - network engineers manually reconfigures the network and throw that over the wall to the operations team.   These two walls create: - confusion: from requirements to implementation, from reconfigured network to operational issues - time lag: the hand-over process is time-consuming
  2. It is very important to distinguish different lifecycles. Two important layers are illustrated in the figure. BOTTOM: The long-lived lifecycle with running VMs/VNFs and TOP: the short-lived providing end-user services: telco:services, IT:applications Walk through the example bullets. Talk bottom-up Design separately, maybe different tools (more on this soon). BUT, they act together in a client-server relationship. Needs clear API linkage between the layers. The top-layer requests resources from the bottom layer. The top-layer does run-time config of the bottom layer. The bottom layer notifies the top layer on changes. The top layer need to adopt.
  3. It is very important to distinguish different lifecycles. Two important layers are illustrated in the figure. BOTTOM: The long-lived lifecycle with running VMs/VNFs and TOP: the short-lived providing end-user services: telco:services, IT:applications Walk through the example bullets. Talk bottom-up Design separately, maybe different tools (more on this soon). BUT, they act together in a client-server relationship. Needs clear API linkage between the layers. The top-layer requests resources from the bottom layer. The top-layer does run-time config of the bottom layer. The bottom layer notifies the top layer on changes. The top layer need to adopt.
  4. Look at the tools to the right. There are more well-known tools for the bottom layer. This also means that there is a risk that these tools are applied in the top-layer since they are known. Top-layer: APIs and fine-grained operations Bottom-layer: more templating Also, the “services” to use an over-loaded word in the top-layer varies much more than in the bottom-layer. A business VPN is very different between service providers, whereas starting a MYSQL server is more main-stream. Tools in the top-layer also depends heavily on if you are an IT Application-centric shop or a network service provider. For the first category the tools in the bottom layer might be ok.
  5. If you are selling applications, the tools for long-lived life-cycles are appropriate also in the top-layer. The networking can be abstracted in the bottom layer. VIMs like OpenStack will hide/hard-code networking. Same with the tuning for the VMs, less effort in tuning the VIM for different VMs. I am not saying that IT is easy. Main challenge: Deployment of new application code several times per day. Needs extreme control on DevOps process for changing the application code itself. If you have internal control this is of course easier. External dev team, puts extreme focus on regression tests and deployment process.
  6. For a network centric offering: Only the underlay network is performed in the bottom layer The top layer does all overlay networking The network layer can not be abstracted by for example OpenStack and VMWare or in the automation tools like Juju, HEAT,… More effort in tuning the virtual infrastructure VIMs for packet processing applications In the previous slide we talked about application centric offerings. There we said that a major challenge is that the application code might change several times per day. For a network centric offering it is the configuration that changes extremely frequently, thousands per day. the application code is changed with very low frequency. Since the top-layer is about high-frequency fine-grained configuration changes for varying offerings we need: Explicit data-models for the offerings APIs that support fine-grained operations
  7. Process: Learning Sharing Collaboration Understanding business, other roles Practices: Automation Lean and agile Infrastructure as code Self-Service Automated rollback and recovery Tools: Share tools across organisation Configuration changes of different nature, different tools Pick-up tools from distributed programming, tools act upon text, not vendor blobs