Go Observability (in practice)

This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana

In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.

This document discusses using GitOps and the Weaveworks Terraform Controller to manage AWS Lambda functions on Kubernetes. Key points include: - Flux is used to bootstrap the Terraform Controller on Kubernetes which then reconciles any changes to the Terraform manifest stored in Git. - The Terraform manifest defines an AWS Lambda resource and references the Git repo, AWS credentials secret, and outputs secret. - AWS access keys are stored as a Kubernetes secret referenced by the Terraform configuration to provision the Lambda function.

Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time-series data, and allows users to query and visualize the data. It works by scraping metrics over HTTP from applications and servers, storing the data in its time-series database, and providing a UI and query language to analyze the data. Prometheus is useful for monitoring system metrics like CPU usage and memory as well as application metrics like HTTP requests and errors.

This document provides an overview of OpenTelemetry, including: - OpenTelemetry is an observability framework that assists in generating and capturing telemetry data from cloud-native software across traces, metrics, and logs. - It includes vendor-agnostic APIs, SDKs, and tools for generating, collecting, and exporting telemetry data to analysis tools. - OpenTelemetry has reached general availability for tracing and is in release candidate for metrics, with client libraries available for many popular programming languages.

Presentation given at Cloud Native Copenhagen, Cloud Native Aalborg, and Cloud Native Aarhus in December 2020

Monitoring containerised apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this talk, Brice Fernandes from Weaveworks will introduce and demo the open source Prometheus monitoring toolkit and its integration with Kubernetes. After this talk, you'll be able to use Prometheus to monitor your microservices on a Kubernetes cluster. We'll cover: - An introduction to Kubernetes to manage containers; - The monitoring maturity model; - An overview of whitebox and blackbox monitoring; - Monitoring with Prometheus; - Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system

In this training webinar, Samantha Wang will walk you through the basics of Telegraf. Telegraf is the open source server agent which is used to collect metrics from your stacks, sensors and systems. It is InfluxDB’s native data collector that supports nearly 300 inputs and outputs. Learn how to send data from a variety of systems, apps, databases and services in the appropriate format to InfluxDB. Discover tips and tricks on how to write your own plugins. The know-how learned here can be applied to a multitude of use cases and sectors. This one-hour session will include the training and time for live Q&A. Join this training as Samantha Wang dives into: Types of Telegraf plugins (i.e. input, output, aggregator and processor) Specific plugins including Execd input plugins and the Starlark processor plugin How to install and start using Telegraf

Log Management Log Monitoring Log Analysis Need for Log Analysis Problem with Log Analysis Some of Log Management Tool What is ELK Stack ELK Stack Working Beats Different Types of Server Logs Example of Winlog beat, Packetbeat, Apache2 and Nginx Server log analysis Mimikatz Malicious File Detection using ELK Practical Setup Conclusion

Presented at GDG Devfest Ukraine 2018. Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.

Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time series data, and allows users to query and alert on that data. It is designed for dynamic cloud environments and has built-in service discovery integration. Core features include simplicity, efficiency, a dimensional data model, the PromQL query language, and service discovery.

Loki is an open source logging aggregation system that indexes the metadata of logs rather than the full contents. It consists of several microservices including the distributor, ingester, query frontend, and querier. The distributor routes logs to the ingesters which store the data in chunks in object storage. The querier handles log queries. Promtail is an agent that can be deployed to scrape logs from files and systemd on servers and ship them to Loki with labels for indexing. Compared to other logging solutions, Loki stores data more cost efficiently and is optimized for scaling.

This document provides an overview of Grafana, an open source metrics dashboard and graph editor for Graphite, InfluxDB and OpenTSDB. It discusses Grafana's features such as rich graphing, time series querying, templated queries, annotations, dashboard search and export/import. The document also covers Grafana's history and alternatives. It positions Grafana as providing richer features than Graphite Web and highlights features like multiple y-axes, unit formats, mixing graph types, thresholds and tooltips.

The monolith to cloud-native, microservices evolution has driven a shift from monitoring to observability. OpenTelemetry, a merger of the OpenTracing and OpenCensus projects, is enabling Observability 2.0. This talk gives an overview of the OpenTelemetry project and then outlines some production-proven architectures for improving the observability of your applications and systems.

NDC18에서 발표하였습니다. 현재 보고 계신 슬라이드는 1부 입니다.(총 2부) - 1부 링크: https://goo.gl/3v4DAa - 2부 링크: https://goo.gl/wpoZpY (SlideShare에 슬라이드 300장 제한으로 2부로 나누어 올렸습니다. 불편하시더라도 양해 부탁드립니다.)

A general introduction and demo of the prometheus monitoring solution and ecosystem with a live demo, given at FLOSSUK 2018.

This document summarizes a presentation about logs and metrics gathering with the OpenShift EFK stack. It introduces the OpenShift logging team and their objectives of collecting distributed logs in a common data model with security and scalability. It describes the main components of Fluendt for collection and normalization and Elasticsearch for storage. It provides examples of using the logging stack with OpenShift, OpenStack, and oVirt and advice for custom application logging.

Back in the days, you had a single machine and you could scroll down the single log file to figure out what is going on. In this Big Data world you need to combine a lot of logs together to figure out what is going on. Data is coming in huge volumes, with high speed so choosing important information and getting rid of noise becomes real challenge. There is a need for a centralized monitoring platform which will aid the engineers operating the systems, and serve the right information at the right time. This talk will try to help you understand all the challenges and you will get an idea which tools and technology stacks are good fit to successfully monitor Big Data systems. The focus will be on open source and free solutions. The problem can be separated in two domains which both are the subject of this talk: metrics stack to gather simple metrics on central place and log stack to aggregate logs from different machines to central place. We will finish up with a combined stack and ideas how it can be improved even further with alerting and automated failover scenarios.

Prometheus is a next-generation monitoring system with a time series database at it's core. Once you have a time series database, what do you do with it though? This talk will look at getting data in, and more importantly how to use the data you collect productively. Contact us at prometheus@robustperception.io

Brian Brazil is an engineer passionate about reliable systems. He has experience at Google SRE and Boxever. He is the founder of Robust Perception and a contributor to open source projects including Prometheus. Prometheus is a monitoring system designed for microservices that allows inclusive, scalable monitoring across languages and services. It uses labels, queries, and federation to provide powerful yet manageable monitoring of dynamic environments.

This document discusses observability for modern applications. It begins by defining observability as the ability to observe what is happening inside a system. Observability helps measure key performance indicators and allows teams to react faster to issues. In cloud native environments, observability fits by instrumenting applications to capture logs, traces, metrics and health data which are then transmitted to analytics tools. The document outlines the different pillars of application instrumentation - logs to see what happened, traces to see how it happened, metrics to see how much happened, and health checks to see system status. It discusses OpenTelemetry as an open source observability framework to address prior vendor lock-in issues and competing standards.