SlideShare a Scribd company logo

Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS for Magento Development / Magento Master

•Download as PPTX, PDF•
•
Stacey Whitney
Stacey Whitney

Learn how to make development and operations work together while using Magento. This session will explore Magento dev-ops community resources, leading practices, common tools, and software. A quick look at server choices under today’s cloud environments will illustrate how to simplify the whole process while developing or deploying with Magento 1 or Magento 2.

Related slideshows

FUG Agile software engineering practices
FUG Agile software engineering practicesFUG Agile software engineering practices
FUG Agile software engineering practices
 
Infrastructure as code with test approach
Infrastructure as code with test approachInfrastructure as code with test approach
Infrastructure as code with test approach
 
Packaged vs. Custom Application Testing
Packaged vs. Custom Application TestingPackaged vs. Custom Application Testing
Packaged vs. Custom Application Testing
 
1 of 37
Fully Stacked: Less Oops, More Ops for Magento Development Magento Master Mover & Magento Certified Developer
Director of Systems Engineering @ Robofirm Mathew Beane
Todays Plan • Foundations: An overview of Magento Development and Operations. • Team: Discussing the culture of processes, technology and people that are found throughout the Devops Lifecycle. • Building Blocks: Examining the moving parts that make up Devops. • Creating a Blueprint: A brief look into the setting up Devops.
Magento Development & Operations Foundations, an overview of Devops.
What is Devops? “Devops is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” From: http://bit.ly/1SVOYGU
Devops Goals • Release on demand • Prepare technology and processes • Infrastructure as code • Eliminate technical debt • Fail without fear • Measure and document everything • Provide a knowledge and cultural feedback loop
Devops Foundation: CAMS Cycle Term coined by John Willis in 2010 • http://bit.ly/20cIRRq • This is a feedback loop which can be built around simple building blocks • Produces incremental growth of the Devops culture, which creates more automation, monitoring and sharing
CAMS: Culture “People and process first. If you don’t have culture, all automation attempts will be fruitless.”
CAMS: Automation “Start to stitch together an automation fabric for Devops. Tools for release management, provisioning, configuration management, systems integration, monitoring and control, and orchestration become important pieces in building a Devops fabric.”
CAMS: Measurement “If you can’t measure, you can’t improve. A successful Devops implementation will measure everything it can as often as it can… performance metrics, process metrics, and even people metrics.”
CAMS: Sharing “Sharing is the loopback in the CAMS cycle. Creating a culture where people share ideas and problems is critical.”
Team Devops Discussing the processes, technology and people that are found throughout the Devops Lifecycle.
Team Devops Each Devops role has people, processes and technologies that are used to deliver the end product.
Devops Individual Roles Roles may be more individualized for some of your “Star Players” • Automation Architect • Release Manager • Security Engineer • Experience/Quality Assurance • Disruptive Technology Research Technician
Business / Operations Business • Sales • Support • Administration • Business Operations • Business Snowflake Logic Operations • “The Store” and any hardware or software that is providing it • Support for customers, administrators and other users • Secondary and tertiary applications that support “The Store”
Development / Engineering • People who write and test code • Tools that are used for Development • Methods and Practices that are encouraged • The development and testing infrastructure
Testing / Monitoring • A wide variety of technologies and tools exist for testing • Monitoring can turn up things tests missed • Responding to monitoring is a must for it to be effective • Testing and Monitoring should be throughout the Devops lifecycle
Customers / End Users • Users of the store/website including administrators and customers • Any service that consumes data from the website • Processes that are related to “User Experience” • Feedback from this group is critical to operational success
Building Blocks Examining some of the moving parts that make up Devops.
Devops Infrastructure Overview
Example Setup of Development Environment
Example Setup of Testing Environment
Infrastructure and Configuration as Code Automated builds using Vagrant and Terraform are a great way to treat infrastructure as code. Configuration Management Tools: • Ansible • Chef • Puppet • Salt Stack • And many others…
AWS
AWS Continuous Full Coverage http://fbrnc.net/blog/2016/03/continuous-load-testing
Continuous Integration and Delivery Tools Automated build, deploy, test and release. CI/CD Tools that work well with Magento • Amazon CodePipeline • Bamboo • Jenkins • TravisCI • And many others…
Continuous Testing * Testing is a broad subject, deserving of more attention. Continuous Integration tools like bamboo provides hooks to allow you to test using several different methods. Magento 2: Testing is built in and easy to tie into your CI/CD Magento 1: More difficult, it is possible with some effort
Continuous Monitoring Nagios:
Continuous Stats See metrics from all of your apps, tools & services in one place using Datadog. This can be tied to a variety of inputs and outputs: • Newrelic • AWS Cloudwatch • Slack • VictorOps / Pagerduty • And Many More
Metrics
Logs With the constant fragmentation of the infrastructure you end up with some pretty incredible logging requirements. Data sources of interest: • Infrastructure metrics (CPU, Network IO) • Business metrics (Sign ups, logins, placed orders) • Events (deployments, auto-scaling activity) • Test metrics (response times, error rates)
Creating a Devops Blueprint A brief look into setting up Devops.
“The Devops Plan” Define, Document & Empower Devops Culture 1. Identify and Document Processes 2. Find people within the Roles 3. Focus on building the foundation for feedback loop Start Automating 1. Application build process 2. Development environment builds 3. UAT environments 4. Production environments Build in monitoring, metrics and create feedback loop.
Small Simple Building Blocks • Start with the git workflow • Application build process • Testing and peer review • Release build process • Automate production deployments • Use operations and monitoring to plan the next release
Keep it simple. With so many choices for all of the technology, its important to focus on small solutions that fit within your existing infrastructure that are easy to implement and maintain.
Questions & Answers
Thanks! • My Family • Robofirm • Mage Titans • The Magento Community • Fabrizio Branca The people who have contributed to the meta-culture that inspired this talk. Follow me on twitter: @aepod See my blog at http://aepod.com/ for a more detailed list of sources and inspiration for this talk.

More Related Content

Mage Titans USA 2016 - Mathew Beane - Edit Fully Stacked: Less OOPS, More OPS for Magento Development / Magento Master

  • 1. Fully Stacked: Less Oops, More Ops for Magento Development Magento Master Mover & Magento Certified Developer
  • 2. Director of Systems Engineering @ Robofirm Mathew Beane
  • 3. Todays Plan • Foundations: An overview of Magento Development and Operations. • Team: Discussing the culture of processes, technology and people that are found throughout the Devops Lifecycle. • Building Blocks: Examining the moving parts that make up Devops. • Creating a Blueprint: A brief look into the setting up Devops.
  • 5. What is Devops? “Devops is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.” From: http://bit.ly/1SVOYGU
  • 6. Devops Goals • Release on demand • Prepare technology and processes • Infrastructure as code • Eliminate technical debt • Fail without fear • Measure and document everything • Provide a knowledge and cultural feedback loop
  • 7. Devops Foundation: CAMS Cycle Term coined by John Willis in 2010 • http://bit.ly/20cIRRq • This is a feedback loop which can be built around simple building blocks • Produces incremental growth of the Devops culture, which creates more automation, monitoring and sharing
  • 8. CAMS: Culture “People and process first. If you don’t have culture, all automation attempts will be fruitless.”
  • 9. CAMS: Automation “Start to stitch together an automation fabric for Devops. Tools for release management, provisioning, configuration management, systems integration, monitoring and control, and orchestration become important pieces in building a Devops fabric.”
  • 10. CAMS: Measurement “If you can’t measure, you can’t improve. A successful Devops implementation will measure everything it can as often as it can… performance metrics, process metrics, and even people metrics.”
  • 11. CAMS: Sharing “Sharing is the loopback in the CAMS cycle. Creating a culture where people share ideas and problems is critical.”
  • 12. Team Devops Discussing the processes, technology and people that are found throughout the Devops Lifecycle.
  • 13. Team Devops Each Devops role has people, processes and technologies that are used to deliver the end product.
  • 14. Devops Individual Roles Roles may be more individualized for some of your “Star Players” • Automation Architect • Release Manager • Security Engineer • Experience/Quality Assurance • Disruptive Technology Research Technician
  • 15. Business / Operations Business • Sales • Support • Administration • Business Operations • Business Snowflake Logic Operations • “The Store” and any hardware or software that is providing it • Support for customers, administrators and other users • Secondary and tertiary applications that support “The Store”
  • 16. Development / Engineering • People who write and test code • Tools that are used for Development • Methods and Practices that are encouraged • The development and testing infrastructure
  • 17. Testing / Monitoring • A wide variety of technologies and tools exist for testing • Monitoring can turn up things tests missed • Responding to monitoring is a must for it to be effective • Testing and Monitoring should be throughout the Devops lifecycle
  • 18. Customers / End Users • Users of the store/website including administrators and customers • Any service that consumes data from the website • Processes that are related to “User Experience” • Feedback from this group is critical to operational success
  • 19. Building Blocks Examining some of the moving parts that make up Devops.
  • 21. Example Setup of Development Environment
  • 22. Example Setup of Testing Environment
  • 23. Infrastructure and Configuration as Code Automated builds using Vagrant and Terraform are a great way to treat infrastructure as code. Configuration Management Tools: • Ansible • Chef • Puppet • Salt Stack • And many others…
  • 24. AWS
  • 25. AWS Continuous Full Coverage http://fbrnc.net/blog/2016/03/continuous-load-testing
  • 26. Continuous Integration and Delivery Tools Automated build, deploy, test and release. CI/CD Tools that work well with Magento • Amazon CodePipeline • Bamboo • Jenkins • TravisCI • And many others…
  • 27. Continuous Testing * Testing is a broad subject, deserving of more attention. Continuous Integration tools like bamboo provides hooks to allow you to test using several different methods. Magento 2: Testing is built in and easy to tie into your CI/CD Magento 1: More difficult, it is possible with some effort
  • 29. Continuous Stats See metrics from all of your apps, tools & services in one place using Datadog. This can be tied to a variety of inputs and outputs: • Newrelic • AWS Cloudwatch • Slack • VictorOps / Pagerduty • And Many More
  • 31. Logs With the constant fragmentation of the infrastructure you end up with some pretty incredible logging requirements. Data sources of interest: • Infrastructure metrics (CPU, Network IO) • Business metrics (Sign ups, logins, placed orders) • Events (deployments, auto-scaling activity) • Test metrics (response times, error rates)
  • 32. Creating a Devops Blueprint A brief look into setting up Devops.
  • 33. “The Devops Plan” Define, Document & Empower Devops Culture 1. Identify and Document Processes 2. Find people within the Roles 3. Focus on building the foundation for feedback loop Start Automating 1. Application build process 2. Development environment builds 3. UAT environments 4. Production environments Build in monitoring, metrics and create feedback loop.
  • 34. Small Simple Building Blocks • Start with the git workflow • Application build process • Testing and peer review • Release build process • Automate production deployments • Use operations and monitoring to plan the next release
  • 35. Keep it simple. With so many choices for all of the technology, its important to focus on small solutions that fit within your existing infrastructure that are easy to implement and maintain.
  • 37. Thanks! • My Family • Robofirm • Mage Titans • The Magento Community • Fabrizio Branca The people who have contributed to the meta-culture that inspired this talk. Follow me on twitter: @aepod See my blog at http://aepod.com/ for a more detailed list of sources and inspiration for this talk.

Editor's Notes

  1. Devops is a goal centric practice based on culture. It’s a practice that builds upon itself over time continuous improvement based on real measurements A planned response, a reaction not just a plan
  2. Devops is goal based on incremental improvements through a few key goals Lack of Devops Problems Maintaining unique snowflakes can be very difficult. Low confidence in build and deploy Failure is frustrating and fruitless Infrastructure and Operations are Out of Scope Configurations tend to drift over time Lack of responsibility for critical parts of infrastructure Training and support is difficult because processes are unique to the people working on the tasks
  3. “There is a increase in feedback loops between business, all parts of the delivery process and operations… thanks to this feedback loops we increase the quality and speed up the flow” - Patrick Debois (December 2009)
  4. Without a strongly defined culture it’s near impossible to automate or measure or even share the results. Magento has a strong Devops meta-culture, with many people participating in the discussion. People, processes and technology Work environment is the stage Inclusive across the entire organization or product Develop cultural growth: Disrupt roles by pairing across disciplines Document processes, actually document everything “Build it and they will come” – You get the best people
  5. Automation requires a Devops culture Use simple tools, woven into an automation fabric Automate one small piece at a time Reducing human effort through automation will mean: an increase in frequency and reliability of releases simplified integrations for monitoring less truck factor in your business
  6. Define metrics and standards Measure everything/anything as you can Utilize tools to store, aggregate and visualize the data Measurement and Testing is critical to Devops growth It’s the foundation for sharing the information in the CAMS cycle
  7. Sharing provides a direct feedback loop for Devops Culture Share success and failure stories with others Contribute to Open Source tools and solutions Get involved in the community Build a training program for clients and employees
  8. Business / Owners Development / Systems Engineering Testing / Monitoring Operations / “The Store” Customers / End users
  9. Devops will affect change throughout the entire operation, and the team roles help sort out how, who and what will be affected. Devops functions best when roles are transparent The Role groups are just examples of the potential boundaries Integrating Devops into a existing process can be disruptive Key players will have a more insight into how things work together
  10. Business Includes people and processes from multiple companies Agencies Hosting Store Owners Licensed Software Support (Magento, Redhat, etc) ----- Devops operations teams should focus on Planned and Emergency Responses Timely resolution of production issues Horizontal visibility of production issues
  11. Includes Systems Engineers and others such as Senior Developers System Architects Support at hosting companies, agencies and other places that get involved in creating the website
  12. A wide variety of technologies and tools exist for testing Frontend: jMeter, gatling, Selenium and others Code: Functional and Regression testing Monitoring can turn up things tests missed syslogd, snmp, newrelic and a ton of other tools exist to help this Responding to monitoring is a must for it to be effective Testing and Monitoring should be found throughout the Devops lifecycle
  13. Devops brings all the separated groups together, acting like a hub and filling the gaps
  14. Environments Development: Individualized local boxes for developers. These can be Virtual Machines, containers, cloud boxes or even just a local OS. These are the only boxes where people “edit code”. Testing: Build tests, user acceptance tests, and integration tests for production. Leading practice has this match your production and emulate any integrations the production site depends on. It is extremely beneficial to have branching setup so releases can be tested concurrently. Production: Blue/Green deployed production site. Immutable infrastructure and code, and best if fully automated. Devops Lifecycle Using AWS as your Infrastructure Continuous Toolsets Continuous Toolsets: Integration and Deployments Testing Monitoring Metrics Logs
  15. Vagrant for local virtual machines Terraform manages cloud resources: Physical Machines Virtual Machines (Cloud) Network Switches and Firewalls Containers Plan/Test before you execute Simple shell script and configuration, easy to use.
  16. Amazon Web Services Fast, reliable, scalable and amazing Typical Services used: EC2 Containers: These are your “boxes” RDS: Relational Database service ElastiCache: Redis & Memcache service S3 Bucket: Scalable storage in the cloud Route53: Rock solid DNS Additional Services: Elastic Load Balancing & Auto Scaling Lambda: Noops response based computing CloudWatch & CloudTrail: Monitoring and log collectioN
  17. As Fabrizio says: Just stick Continuous in front of anything and it sounds cool.
  18. “Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.” - Martin Fowler, Chief Scientist, ThoughtWorks
  19. M2: Performance, Selenium & Unit Tests all provided by the Magento Testing Framework (MTF) M1: Performance Tests: Magento Performance Toolkit Selenium Tests: Magiumlib can be utilized Unit Tests: Typically using the EcomDev_PHPUnit tests.
  20. SNMP: Simple Network Monitoring Protocol Pagerduty: Endpoint for Alerts Ties back to Jira automatically creating tickets on issues
  21. Newrelic: Real-time insight into performance across application and infrastructure Allows for incredible horizontal visibility into metrics Can be configured to collect a lot of application specific data Has a robust API that can tie to other systems
  22. Using ELK can really help sort this out, SaaS provider logz.io is a great place to start. Continuous Load Testing and Monitoring: http://fbrnc.net/blog/2016/03/continuous-load-testing
  23. Dev infrastructure builds first (simple and flat one per box) UAT and staging environments (add branching and other challenges) Production Environments (Autoscaling and other concerns are primary)
  24. Rock Solid Magento Development – By Fabrizio Branca is a great way to get started with your workflow
  25. The biggest downside of the Devops toolbox is that it’s has to be a bit of a Pandora's box, we are responsible for bringing in new tools to almost every aspect of the projects we work on. Start small, work through the pieces one at a time and weaving each process into the devops fabric.