Performance Monitoring for Docker Environments - Docker Amsterdam June Meetup

Spenser Reinhardt's presentation on Detecting Security Breaches With Docker, Honeypots, & Nagios. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference

Nirmata is an enterprise platform that uses Netflix OSS like Eureka, Zuul, Ribbon, and Archaius to build cloud applications. It allows users to model their business services and deploy them along with infrastructure services to sandbox environments using Docker containers. Services are deployed by launching Docker containers from a central repository in a specific order to ensure dependencies are met. Nirmata addresses challenges in running Docker containers together by configuring ports, service communication, and dependency injection.

Docker Indy Meetup - Monitoring 30-Aug-2016 - Monitoring platforms - Built in Healthchecks - Sysdig demo - Swarm mode demo + healthchecks Link to demo repository: https://github.com/mbentley/docker-healthcheck-demos

Customers from over all over the world streamed Forty Two Billion hours of Netflix content last year. The Netflix streaming service had been powered by the Amazon cloud with virtual machines for over five years, blazing a trail for similar architectures. In the last year, it invested in containers for batch-style jobs and service-style applications. Andrew Spyker will explain the potential containers have to help Netflix create a more productive development experience while simultaneously deepening its control over resource management. Join Andrew to see why Netflix is moving forward with containers, how it can leverage its existing operational machinery, and how it’s running containers with a similar guarantee of high availability as current Netflix infrastructure provides.

This document discusses how Netflix OSS and Spring Cloud can be used together to implement a microservices architecture. It describes various libraries like Eureka for service discovery, Hystrix for circuit breaking, Ribbon for load balancing, and Zuul for API gateway functionality. It also explains how Spring Boot simplifies application development and Spring Cloud integrates these Netflix libraries and provides additional features like a configuration server. Overall, Netflix OSS and Spring Cloud provide the tools needed to build resilient microservices with service discovery, load balancing, and other capabilities in a transparent way for developers.

Andrew Spyker presented on the Netflix Cloud Platform and ZeroToDocker project. The following key points were discussed: - ZeroToDocker provides Docker images of Netflix OSS projects like Eureka, Zuul and Asgard to more easily evaluate the technologies. However, the images are not intended for direct production use. - A demo showed running a microservices application and supporting Netflix OSS services like Eureka and Zuul using Docker containers on a single machine. - While Docker aids development and evaluation, additional tooling is needed to operationalize containers at production scale across multiple hosts for tasks like networking, security, logging and scheduling. Competing ecosystems are emerging to address these needs.

Spring Cloud/Netflix OSS way of building microservices on Kubernetes -- preso from Spring One Platform 2016

The document discusses microservices and APIs. It covers how microservices optimize for speed by shedding dependencies and having dependencies on demand through services and APIs. It discusses consumer contracts for APIs and service versioning. It also discusses using an API gateway pattern for scalability, security, monitoring and more. It promotes API management for benefits like access control, analytics, and monetization of microservices.

What are, or aren't, microservices? There's a lot of hype and buzz, but microservices emerged organically vs how some of the other distributed architectural styles were "handed down to us", so I believe there's some good things once you cut through the hype. In this talk I discussed what are and are NOT microservices, introduced some concepts, and discussed some concrete open-source libraries and frameworks that can help you develop and manage microservice style deployments.

Fugro Chance Inc. oversees ship surveys globally using IoT and Docker. They developed a solution using AWS, Docker, and microservices to support a real-time web application for ship tracking. Key challenges included supporting services that need to run together and efficiently deploying new versions. They addressed this using SupervisorD to run multiple services in a single Docker container. This allows flexible development and deployment of future microservices.

What is the right balance between moving fast, innovating, experimenting with new technology, and protecting the personal data of our customers and interests of our stakeholders? How can we safely try new ideas in production without risking costly downtime? Does the utopia where developers are free from lock-in and operators enjoy the calm of a steadily running system exist in the real world? Is it possible to have open platforms with better security? At Kroger Digital we are still working through these questions every day but are redesigning our systems with the goals of true operational maturity and security. Discover how we are building capabilities for monitoring, A/B testing, and continuous delivery with Docker Datacenter, plugins, and open source building blocks such as NGiNX, ElasticSearch, and more.

Customer trust and security is paramount for Salesforce. While containerization is great for DevOps due to flexibility, speed, isolation, transient existence, ease of management and patching, it becomes a challenging environment when the sensitivity level of the data traversing the environment increases. Monitoring systems, applications and network; performing disk, memory and network forensics in case of an incident; and vulnerability detection can easily become daunting tasks in such a volatile environment. In this presentation we would like to discuss the infrastructure we have built to address these issues and to secure our Docker container platform while we rapidly containerize Salesforce. Our solutions focus on securing the container pipeline, building security into the architecture, monitoring, Docker forensics (disk, memory, network), and automation. We also would like to demonstrate some of our live memory analysis capabilities we leverage to assure container and application integrity during execution.

In this session we will talk about HealthDirect’s journey with Docker. We will follow the life cycle of a container through our CD process to its home in our swarm cluster with just a git commit thanks to configuration management. We will cover the CD process for Docker, Docker swarm, Docker networking and service discovery. The audience will leave with a solid foundation of how to build a production ready swarm cluster (A github repo with code will be given). They will also have the knowledge of how to implement a CD framework using Docker.

Our motto "Imagination at work" is the belief in driving innovation that builds, powers, moves and cures the world. At GE, we have 9,000+ legacy apps powering 9 business units across every major industry from oil and gas, healthcare to household appliances generating over $148B in revenue. With legacy apps and infrastructure, our app teams were facing issues with long development cycles, deploying apps and scaling features and services. How do you migrate legacy data center built apps to a new microservices and hybrid cloud architecture at this organizational scale and business diversity? In this talk, the GE Digital team will share their journey to a modern microservices platform built with Docker Datacenter, Rails, Chef, Sensu, Gems, AWS, Azure and Rackspace on-prem to modernize these apps and automate processes to enable agile development and rapid deployment. This session will cover both the technical and organizational sides of the project to take legacy apps and infrastructure at GE to multi cloud microservices.

1. The document describes a Docker implementation of NetflixOSS microservices on IBM SoftLayer. 2. Key aspects discussed include networking Docker containers across multiple SoftLayer datacenters, managing the Docker API across multiple hosts, and integrating Docker images with SoftLayer image management. 3. Lessons learned include the need for a proxy for the Docker remote API across multiple hosts, and approaches for keeping Docker advantages like image portability when integrating with an IaaS platform.

The SDACK architecture stands for Spark, Docker, Akka, Cassandra, and Kafka. At TrendMicro, we adopted the SDACK architecture to implement a security event inspection platform for APT attack analysis. In this talk, we will introduce SDACK stack with Spark lambda architecture, Akka and Kafka for streaming data pipeline, Cassandra for time series data, and Docker for microservices. Specifically, we will show you how we Dockerize each SDACK component to facilitate the RD team of algorithms development, help the QA team test the product easily, and use the Docker as a Service strategy to ship our products to customers. Next, we will show you how we monitor each Docker container and adjust the resource usage based on monitoring metrics. And then, we will share our Docker security policy which ensures our products are safety before shipping to customers. After that, we'll show you how we develop an all-in-one Docker based data product and scale it out to multi-host Docker cluster to solve the big data problem. Finally, we will share some challenges we faced during the product development and some lesson learned.

Riot builds a lot of software. At the start of 2015 we were looking at 3000 build jobs over a hundred different applications and dozens of teams. We were handling nearly 750 jobs per hour and our build infrastructure needed to grow rapidly to meet demand. We needed to give teams total control of the “stack” used to build their applications and we needed a solution that enabled agile delivery to our players. On top of that, we needed a scalable system that would allow a team of four engineers to support over 250. After as few explorations, we built an integrated Docker solution using Jenkins that accepts docker images submitted as build environments by engineers around the company . Our “containerized” farm now creates over 10,000 containers a week and handles nearly 1000 jobs at a rate of about 100 jobs an hour. In this occasionally technical talk, we’ll explore the decisions that led Riot to consider Docker, the evolutionary stages of our build infrastructure, and how the open source and in-house software we combined to achieve our goals at scale. You’ll come away with some best practices, plenty of lessons learned, and insight into some of the more unique aspects of our system (like automated testing of submitted build environments, or testing node.js apps in containers with Chromium and xvfb).

Performance monitoring for Docker Challenges - Anomaly detection - CoScale demo For more info about how to use CoScale Docker monitoring, some reading material here: http://www.coscale.com/blog/how-to-monitor-docker-containers-with-coscale and http://www.coscale.com/blog/how-to-monitor-your-kubernetes-cluster A summary of CoScale Docker performance monitoring can be found here: http://www.coscale.com/docker-monitoring

On Friday 5 June 2015 I gave a talk called Cluster Management with Kubernetes to a general audience at the University of Edinburgh. The talk includes an example of a music store system with a Kibana front end UI and an Elasticsearch based back end which helps to make concrete concepts like pods, replication controllers and services.

Be a better developer with Docker: tricks of the trade (revision 3) The talk will teach developers how to approach their development environment setups using Docker, covering awesome tricks to make the experience smooth, fast, powerful and repeatable. The talk is logically divided in five parts: - What is Docker - Why Docker makes developers happier - Workflows and techniques - Tips and tricks - Future developments

Docker can be used as an everyday development tool. It allows building, shipping and running applications securely by using containers. Containers allow encapsulating applications from the host machine and provide resource isolation using features like cgroups and namespaces. The key Docker concepts include images, containers, volumes, and the Docker engine. Docker Compose can be used to define and run multi-container Docker applications using a YAML file.

Docker can be used as an everyday work tool for developers and system administrators. It provides tools to work with containers, which enable operating-system-level virtualization. Docker images contain executable packages that include code, runtimes, and configuration files to run software. Containers run as isolated processes on the host machine, using resources from the host operating system. Common Docker commands include docker run to launch containers, docker build to build images, and docker ps to view running containers. Docker Compose allows defining and running multi-container applications using a YAML configuration file.

Containers and other forms of dynamic infrastructure can prove challenging to monitor. How do you define normal, when your infrastructure is intentionally in motion and change from minute to minute? Join us as we discuss proven strategies for monitoring your containerized infrastructure on AWS and ECS.

In this session we'll explore measuring VM performance and evaluating changes to settings or infrastructure which can affect performance positively. We'll also share the best current practice for architecture for high performance clouds from our experience.

This presentation gives an overview of the steps in the workshop labs for Oracle Management Cloud APM and Log Analytics. The labs themselves and all sources are found at GitHub: https://github.com/lucasjellema/APM-Demo-App-WorldView .

This document discusses building smarter applications that incorporate machine learning models. It provides an overview of combining predictive models with applications, deploying models in production, and a concrete use case of a consumer loan application. The use case involves building two predictive models using H2O - one for predicting if a loan will be bad, and one for predicting the interest rate. The document outlines the steps to build such a smarter application and integrate predictive models via a REST API. It also describes the data, models, and software tools used in the example application code provided.

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

This document discusses cloud computing and DevOps. It provides background on the speaker and explains how Morningstar adapted to use cloud computing to scale their infrastructure. Cloud computing allows for much higher server-to-engineer ratios and elastic scaling. DevOps aims to break down silos between development and operations through automation, measurement, and culture change. It emphasizes infrastructure as code and continuous delivery to improve business agility.

This session brings to your attention how several millions of dollars are wasted and what you can do to save money. Optimizing garbage collection performance not only saves money, but also improves the overall customer experience as well.

This document discusses challenges with running containers at scale and how artificial intelligence for IT operations (AIOps) can help address those challenges. It defines AIOps and outlines how it utilizes techniques like machine learning and analytics to provide proactive, personalized insights for infrastructure and application monitoring. Specific challenges covered include reactive monitoring of dynamic container environments, metrics explosions, and performing proactive tasks like capacity planning, cluster scheduling, and dynamic configuration optimization. The document provides examples of how AIOps has helped companies optimize infrastructure usage through techniques like exhaustive testing of hardware/software combinations, live traffic load testing, bottleneck identification, batch scheduling, and controlled resource oversubscription while maintaining service level objectives.

The talk covers the following topics: 1. Introduction to event sourcing. 2. How event sourcing and Redux are similar. 3. How to implement offline mode for React Native application. 4. How everything from above was run in a production.

Rx is a generic abstraction of computation expressed through Observable<Element> interface, which lets you broadcast and subscribe to values and other events from an Observable stream.

Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.