SlideShare a Scribd company logo
DEEP DIVE INTO ELASTICSEARCH
Establish A Powerful Log Analysis System
With Elastic Stack.
On Premises vs SaaS Elastic Stack
Comparisons.
Tyler
DevOps Engineer
NFQ Asia Company
Agenda
• Intro.
• Overview: Elastic Stack.
• Establish a powerful log analysis system with Elastic Stack.
• Elastic stack options from cloud providers.
• Which one would be fit for us?
• Cost Reflections.
• In conclusion.
About Me
• In tech for 7+ years.
• Technical Project Coordinator @ AVASO Technology Solutions.
• Infrastructure Technical Lead @ Betfair Group PLC.
• DevOps Engineer @ NFQ Asia.
• Member of Vietnam Elasticsearch Community.
• Bash/PowerShell languages.
• A dog parent :D
About NFQ Asia
• Member of NFQ Company.
• 15+ years’ experience in e-business
strategy and software development
• 300+ professionals.
• 4 countries: offices in Lithuania,
Germany, Vietnam, Singapore.
• Founded in Vietnam since 2015.
• Having organized 5 community
events/hackathons in Vietnam.

Recommended for you

[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7

Monitor availability and performance of applications hosted in the Amazon cloud. Monitor your Amazon EC2 and RDS instances and gain insight into the performance of your cloud computing environment, troubleshoot and resolve problems before end users are affected. Forums: https://forums.site24x7.com/ Facebook: http://www.facebook.com/Site24x7 Twitter: http://twitter.com/site24x7 Google+: https://plus.google.com/+Site24x7 LinkedIn: https://www.linkedin.com/company/site24x7 View Blogs: http://blogs.site24x7.com/

amazon s3amazon rds instance monitoringamazon web services monitoring
Azure Application insights - An Introduction
Azure Application insights - An IntroductionAzure Application insights - An Introduction
Azure Application insights - An Introduction

These slides give an introduction to the features App Insights provide. Also, it will present some tips and tricks you might find useful.

azureazure cloudsoftware development
SharePoint best practices
SharePoint best practicesSharePoint best practices
SharePoint best practices

The document outlines a capacity management model for SharePoint implementations and customizations including planning, building, operating, and maintaining a SharePoint farm. The plan phase includes capacity planning, analyzing impacts, and governance. The build phase covers implementing a proof of concept, using adequate hardware, and proper farm installation. The operate phase focuses on user management, regular monitoring, maintenance, and patch management. Customization is also addressed including development processes, solutions planning, and best practices.

DATA
Cost
Operations
Features
Platforms
Plugins
Capability
Mapping
Processors
Aggregations
APIs
Monitoring
Security
Encryption
Supports
Backup
Database
Searching
Analytics
ComplexityArchitecture
APIsFlexibility
Availability
Compatibility
Centralization
Elasticsearch is everywhere
What is Elastic Stack?
• Formerly known as ELK Stack.
• ELK - The acronym for three open source
projects: elasticsearch, logstash, and kibana.
• Distributed, scalable, and highly available
(both on premises or SaaS).
• The Elastic Stack is the next evolution of ELK.
• Supports the lightweight Beats data shippers
from ES v2.1.1.
Elasticsearch
• “You know, for Search”
• Free, Open Source.
• Search engine based on Lucene.
• Near real-time searching, analytics and
visualization capabilities.
• Sophisticated Restful API.

Recommended for you

Hands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech TalksHands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech Talks

Learning Objectives: - Securing network access to Amazon EC2 Instances with Security Groups - Launch and configure a Windows virtual machine - Bootstrapping using Powershell - Creating Key Pairs for authentication

awswebinarcloud computing
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn

Good monitoring can be the difference between a great night's sleep or hearing your phone go off at 2:37 a.m. because of a production outage. Couchbase Server provides a large number of metrics which can be overwhelming if you do not know the critical things to focus on or how to expose that information to your monitoring system. In this talk we will look at example production incidents, going in depth around specific things to monitor, and how this information can be used to find issues, work out root cause, and discover trends.

couchbasealertingmonitoring
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...

LinkedIn serves traffic for its 467 million members from four data centers and multiple PoPs spread geographically around the world. Serving live traffic from from many places at the same time has taken us from a disaster recovery model to a disaster avoidance model where we can take an unhealthy data center or PoP out of rotation and redistribute its traffic to a healthy one within minutes, with virtually no visible impact to users. The geographical distribution of our infrastructure also allows us to optimize the end-user's experience by geo routing users to the best possible PoP and datacenter. This talk provide details on how LinkedIn shifts traffic between its PoPs and data centers to provide the best possible performance and availability for its members. We will also touch on the complexities of performance in APAC, how IPv6 is helping our members and how LinkedIn stress tests data centers verify its disaster recovery capabilities.

linkedinengineeringdns
Logstash
• Open source data collection engine that unifies
data from disparate sources, normalizes it and
distributes it.
• The ingestion workhorse for elasticsearch and
more.
• Real-time capabilities and pluggable pipeline
architecture.
• Community-extensible and developer-friendly
plugin ecosystem.
Kibana
• Open source analytics and visualization
platform designed to work with elasticsearch.
• Specialized for large volumes of streaming and
real-time data.
• No code, no additional infrastructure required.
• Easily and quickly understandable through
graphic representation.
Beats Platform
• “Data shippers” that are installed on servers
as agents.
• Either elasticsearch directly or through
logstash.
• Library written based on Golang.
• Supports create your own beat for specific
use cases.
ESTABLISH A POWERFUL LOG ANALYSIS
SYSTEM WITH ELASTIC STACK

Recommended for you

RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...

In this webinar you’ll learn how to enable your organization with fast, self-service access to your VMware vSphere resources as well as public clouds such as AWS, Google and Azure. We’ll cover the following topics: 1. Create a Self-Service Portal for vSphere and Public Clouds Are virtualization and cloud the same thing? How does vSphere fit with cloud self-service? 2. Create and Manage Portable Workloads Can I move from vSphere to AWS? and back? Choosing between in-place management, migration and portability. 3. Provide Visibility and Control for Self-Service Ensure workloads meet corporate standards. Manage capacity and costs.

cloud computingcloud managementrightscale webinars
SignalR 101
SignalR 101SignalR 101
SignalR 101

This document discusses different approaches for building real-time web functionality and introduces SignalR as a solution. It describes how earlier approaches like periodic pooling and long polling have disadvantages. Web sockets are ideal but have limitations in support across browsers, proxies, and web servers. SignalR abstracts these transports to provide a simple and robust library for adding real-time functionality to ASP.NET applications without these constraints. The document demonstrates SignalR and provides resources to learn more.

microsoft developers - philippinesmicrosoftsignalr
GAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure FunctionsGAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure Functions

GAB 2017 presentation for Logic Apps and Azure Function, discussing the basic concepts of each technology and presenting a hands on scenario.

azure logicapps azurefunctions integration
Rationale
• What is log?
• How do we solve the production issue as usual?
• How much time do you spend investigating the
production issue?
• Where are the archived log?
• Visualization and dashboards?
The Challenge
How do you satisfy the search needs of the application system’s over 2,000 docs
per second while simultaneously providing tactical operational insights that help
both Development Team and Operation Team iteratively improve the customer
experience?
The Simple Log Analysis Diagram
Demonstration

Recommended for you

Monitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba CloudMonitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba Cloud

In this presentation we are going to learn following Alibaba Cloud Services/tools used to monitoring the Containerized application. -CAdvisor -Heapster/InflexDB -Grafana -Prometheus

alibaba cloudgrafanamonitoring container
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016

This document discusses LinkedIn's use of Couchbase as an in-memory data store. It describes how LinkedIn has grown to rely heavily on Couchbase, now running it in production, staging, and corporate environments. It also outlines some of the key use cases Couchbase supports at LinkedIn, such as serving as a read-through cache, storing counters, and acting as a source of truth datastore for some internal tools. Finally, it discusses the operational tooling and processes LinkedIn has developed to support Couchbase at scale.

srecachingcouchbase
Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service

Organizations are seeking to drive agility by offering developers a self-service portal to access cloud resources. In order to provide push-button access to the cloud, IT DevOps teams need to orchestrate the deployment, configuration and integration of entire technology stacks or applications.

rightscalecloudworkloads
Scalability Rationale
• High availability.
• Petabyte-scale data is written and/or read frequently.
• High scalability.
• Sufficient data allocation.
• Costs.
The Elasticsearch Hot-Warm Architecture
The Elasticsearch Hot-Warm Architecture (cont.)
ON PREMISES VS SaaS ELASTIC STACK
COMPARISONS

Recommended for you

Mastering Azure Monitor
Mastering Azure MonitorMastering Azure Monitor
Mastering Azure Monitor

This document summarizes a presentation about mastering Azure Monitor. It introduces Azure Monitor and its components, including metrics, logs, dashboards, alerts, and workbooks. It provides a brief history of how Azure Monitor was developed. It also explains the different data sources that can be monitored like the Azure platform, Application Insights, and Log Analytics. The presentation encourages attendees to navigate the "maze" of Azure Monitor and provides resources to help learn more, including an upcoming virtual event and blog post series on monitoring.

azureazure monitorlog analytics
David Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to EspressoDavid Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to Espresso

This talk is about our experience at LinkedIn migrating our content ingestion system from using Oracle to using our internal database system Espresso. I explain some of the reasons for doing the migration as well as how we met the challenges of swapping database technologies with no down time and in a way that was transparent to our clients. This talk was delivered at the SATURN 2018 conference in Plano, TX on May 9, 2018.

saturn18software developmentarchitecture
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoStateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho

This presentation discusses how to bring stateful behaviors to serverless architecture using Redis. It introduces the problem of enabling statefulness in serverless applications and proposes using Redis as a solution. Key considerations for the Redis-based architectural approach are discussed, including topology, high availability and scalability, and Redis configuration tuning. A demo is then presented to illustrate "Redis in Serverless" in action.

WHAT IS AWS ELASTICSEARCH
SERVICE?
• Managed service in AWS Cloud.
• Introduced in Oct 2015.
• Fully managed; Zero admin.
• Highly available and reliable.
• Built-in Kibana support.
• Integrated with other services in AWS ecosystem.
The AWS Integration
What is Elastic Cloud?
• Launched in Oct 2015.
• Provided by Elastic.
• High provisioning and scaling.
• Hosted in the Cloud Providers.
• Service-oriented architecture.
• Containerization using Docker.
• Fully supports custom plugins and API.
Elastic Cloud Architecture

Recommended for you

Master thesis
Master thesisMaster thesis
Master thesis

My master thesis presentation: "Avoiding CRUD operations lock-in in NoSQL databases: extension of the CPIM library"

nosqldatabasecrud
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production Systems

LinkedIn created an auto-remediation system named Nurse which leverages SaltStack and the CherryPy API to auto-triage and remediate issues with production systems. See how LinkedIn uses SaltStack with Nurse in its production environment and learn how to architect your own auto-triage and remediation system.

infrastructurelinkedinit
Nuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 HighlightsNuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 Highlights

Get a glimpse of the main features supported in Nuxeo Platform LTS 2015. With this LTS version of the Nuxeo Platform, we’re changing how we assign product version names and numbers. The name for each LTS version is now based on the release year. Nuxeo Platform LTS 2015 is the result of the four Fast Track releases throughout the past year. Highlights of Nuxeo Platform LTS 2015 include: - Nuxeo Live Connect: Native Integration with Google Drive & Dropbox - Content Analytics & Data Visualisation - Elasticsearch: API Passthrough, Hints for NXQL, Security - Massive Scalability with MongoDB Integration - New Document Viewer - Automation Scripting - Nuxeo Drive 2 - Automated Media Conversions

nuxeomongodbecm
HOW DO I KNOW WHICH ONE IS FIT
WITH ME?
Specifications comparison sheet
Self-managed Elastic Stack AWS Elasticsearch Service Elastic Cloud Enterprise
Pros
More options and features.
Complete control settings and
capacity.
Access to other APIs
Comprehensive ES monitoring
solutions.
Lowest costs.
SaaS.
Simplify the operations via APIs.
Security by IAM.
Automated snapshots*.
Encryption at rest.
Monitoring included*.
Technical supported.
SaaS.
Fully control through APIs.
Technical Supported.
Uptime SLA.
Feature-rich and complete
monitoring product.
Available on Marketplace.
Cons
Self maintenance.
Infrastructure matters.
No technical supported.
X-Pack limit features.
Limited control.
Less capacity and scalability.
Backup once time per day.
No plugins, no logs.
Medium expensive.
Only support I2 series EC2
instances.
Most expensive.
Imperfect for AWS-hosted
solutions.
Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons.
Costs Comparison Chart
8,400.38 10,678.56 11,512.51
75,303.17
81,375.17
11,316.98 14,500.26
25,201.1525,201.15
32,035.68 34,537.54
203,318.55
219,712.95
28,319.95
38,295.63
50,402.30
0
50,000
100,000
150,000
200,000
250,000
Elastic Stack (AWS) Elastic Stack (GCP) AWS Elasticsearch Services Elastic Cloud (GCP) Elastic Cloud (AWS)
Cost($)
Service Models
One Year One Year (All Upfront) Three Years Three Years (All Upfront)
*Costs calculated based on 3TB-data cluster in multi-AZ in
Frankfurt region

Recommended for you

Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsLessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatterns

The document discusses lessons learned from embedding Cassandra in the xPatterns big data analytics platform. It provides an agenda that includes discussing Cassandra usage in xPatterns, the necessary developments like data modeling optimizations, robust REST APIs, geo-replication, and a demo of exporting to NoSQL APIs. Key lessons learned since Cassandra versions 0.6 to 2.0.6 are also summarized, such as the need for consistent clocks, reducing column families, and monitoring.

cassandrasparkmesos
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.

architecturecore kafkaanalytics
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape

A talk about various Open Source tools for availability and functional monitoring, time series and metrics as well log and event management.

graylog2elasticfluentd
In Conclusion
• Elasticsearch leverage the power of analysis ability for both Dev/Ops teams.
• Easily operate/maintain the huge cluster of servers and microservices.
• Choose the proper architecture depend on application/system.
• Estimate the budget to meet the requirements.
• Optimize the aggregation to adopt the resources.
• High availability oriented system.
We are hiring…
• Java Senior/Lead Developer
• PHP Senior Developer
• PHP Technical Lead
• Front-end Senior Developer
• Front-end Technical Lead
• Technical Project Manager
Simply send us an email with your enclosed
updated CV to: career@nfq.asia
Contact Me
LinkedIn: linkedin.com/in/tylernguyen91
Email: tai.nguyen@nfq.asia
Telegram: @tylern91

More Related Content

What's hot

Project Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on DockerProject Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on Docker
RightScale
 
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedInReducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
Michael Kehoe
 
Opening the Outage Door: Integrating OMS into CIS
Opening the Outage Door: Integrating OMS into CISOpening the Outage Door: Integrating OMS into CIS
Opening the Outage Door: Integrating OMS into CIS
SSP Innovations
 
[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7
Site24x7
 
Azure Application insights - An Introduction
Azure Application insights - An IntroductionAzure Application insights - An Introduction
Azure Application insights - An Introduction
Matthias Güntert
 
SharePoint best practices
SharePoint best practicesSharePoint best practices
SharePoint best practices
Dinusha Kumarasiri
 
Hands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech TalksHands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech Talks
Amazon Web Services
 
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Michael Kehoe
 
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
Michael Kehoe
 
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale
 
SignalR 101
SignalR 101SignalR 101
GAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure FunctionsGAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure Functions
Wagner Silveira
 
Monitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba CloudMonitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba Cloud
gavaskar s
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016
Michael Kehoe
 
Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service
RightScale
 
Mastering Azure Monitor
Mastering Azure MonitorMastering Azure Monitor
Mastering Azure Monitor
Richard Conway
 
David Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to EspressoDavid Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to Espresso
David Max
 
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoStateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Redis Labs
 
Master thesis
Master thesisMaster thesis
Master thesis
Fabio Arcidiacono
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production Systems
Michael Kehoe
 

What's hot (20)

Project Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on DockerProject Sherpa: How RightScale Went All in on Docker
Project Sherpa: How RightScale Went All in on Docker
 
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedInReducing MTTR and False Escalations: Event Correlation at LinkedIn
Reducing MTTR and False Escalations: Event Correlation at LinkedIn
 
Opening the Outage Door: Integrating OMS into CIS
Opening the Outage Door: Integrating OMS into CISOpening the Outage Door: Integrating OMS into CIS
Opening the Outage Door: Integrating OMS into CIS
 
[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7[Webinar] AWS Monitoring with Site24x7
[Webinar] AWS Monitoring with Site24x7
 
Azure Application insights - An Introduction
Azure Application insights - An IntroductionAzure Application insights - An Introduction
Azure Application insights - An Introduction
 
SharePoint best practices
SharePoint best practicesSharePoint best practices
SharePoint best practices
 
Hands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech TalksHands on Lab: Windows Workloads - AWS Online Tech Talks
Hands on Lab: Windows Workloads - AWS Online Tech Talks
 
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
 
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
APRICOT 2017: Trafficshifting: Avoiding Disasters & Improving Performance at ...
 
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
RightScale Webinar: Provide a Self-Service Portal for vSphere, AWS and Other ...
 
SignalR 101
SignalR 101SignalR 101
SignalR 101
 
GAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure FunctionsGAB 2017 - Logic Apps and Azure Functions
GAB 2017 - Logic Apps and Azure Functions
 
Monitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba CloudMonitoring Containerized Application in Alibaba Cloud
Monitoring Containerized Application in Alibaba Cloud
 
Couchbase Connect 2016
Couchbase Connect 2016Couchbase Connect 2016
Couchbase Connect 2016
 
Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service Orchestrating Cloud Workloads with RightScale Self-Service
Orchestrating Cloud Workloads with RightScale Self-Service
 
Mastering Azure Monitor
Mastering Azure MonitorMastering Azure Monitor
Mastering Azure Monitor
 
David Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to EspressoDavid Max SATURN 2018 - Migrating from Oracle to Espresso
David Max SATURN 2018 - Migrating from Oracle to Espresso
 
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoStateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
 
Master thesis
Master thesisMaster thesis
Master thesis
 
Using SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production SystemsUsing SaltStack to Auto Triage and Remediate Production Systems
Using SaltStack to Auto Triage and Remediate Production Systems
 

Similar to Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons.

Nuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 HighlightsNuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 Highlights
Nuxeo
 
Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsLessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatterns
Claudiu Barbura
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
confluent
 
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape
NETWAYS
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
Mathew Beane
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
Amazon Web Services
 
What's New in IBM Streams V4.1
What's New in IBM Streams V4.1What's New in IBM Streams V4.1
What's New in IBM Streams V4.1
lisanl
 
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
Amazon Web Services
 
First Look at Azure Logic Apps (BAUG)
First Look at Azure Logic Apps (BAUG)First Look at Azure Logic Apps (BAUG)
First Look at Azure Logic Apps (BAUG)
Daniel Toomey
 
Service quality monitoring system architecture
Service quality monitoring system architectureService quality monitoring system architecture
Service quality monitoring system architecture
Matsuo Sawahashi
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
aspyker
 
The Best of re:invent 2016
The Best of re:invent 2016The Best of re:invent 2016
The Best of re:invent 2016
Amazon Web Services
 
Kubernetes Infra 2.0
Kubernetes Infra 2.0Kubernetes Infra 2.0
Kubernetes Infra 2.0
Deepak Sood
 
Net Devops Overview
Net Devops OverviewNet Devops Overview
Net Devops Overview
Joel W. King
 
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
Amazon Web Services
 
Geek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure EnvironmentsGeek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure Environments
IDERA Software
 
IncQuery Server for Teamwork Cloud - Talk at IW2019
IncQuery Server for Teamwork Cloud - Talk at IW2019IncQuery Server for Teamwork Cloud - Talk at IW2019
IncQuery Server for Teamwork Cloud - Talk at IW2019
Istvan Rath
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)
Mathew Beane
 
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Lucas Jellema
 
Why real integration developers ride Camels
Why real integration developers ride CamelsWhy real integration developers ride Camels
Why real integration developers ride Camels
Christian Posta
 

Similar to Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons. (20)

Nuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 HighlightsNuxeo Platform LTS 2015 Highlights
Nuxeo Platform LTS 2015 Highlights
 
Lessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatternsLessons learned from embedding Cassandra in xPatterns
Lessons learned from embedding Cassandra in xPatterns
 
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...
 
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape
 
Elk ruminating on logs
Elk ruminating on logsElk ruminating on logs
Elk ruminating on logs
 
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
AWS re:Invent 2016: Accenture Cloud Platform Serverless Journey (ARC202)
 
What's New in IBM Streams V4.1
What's New in IBM Streams V4.1What's New in IBM Streams V4.1
What's New in IBM Streams V4.1
 
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
 
First Look at Azure Logic Apps (BAUG)
First Look at Azure Logic Apps (BAUG)First Look at Azure Logic Apps (BAUG)
First Look at Azure Logic Apps (BAUG)
 
Service quality monitoring system architecture
Service quality monitoring system architectureService quality monitoring system architecture
Service quality monitoring system architecture
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
 
The Best of re:invent 2016
The Best of re:invent 2016The Best of re:invent 2016
The Best of re:invent 2016
 
Kubernetes Infra 2.0
Kubernetes Infra 2.0Kubernetes Infra 2.0
Kubernetes Infra 2.0
 
Net Devops Overview
Net Devops OverviewNet Devops Overview
Net Devops Overview
 
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
(ENT211) Migrating the US Government to the Cloud | AWS re:Invent 2014
 
Geek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure EnvironmentsGeek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure Environments
 
IncQuery Server for Teamwork Cloud - Talk at IW2019
IncQuery Server for Teamwork Cloud - Talk at IW2019IncQuery Server for Teamwork Cloud - Talk at IW2019
IncQuery Server for Teamwork Cloud - Talk at IW2019
 
ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)ELK Ruminating on Logs (Zendcon 2016)
ELK Ruminating on Logs (Zendcon 2016)
 
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
 
Why real integration developers ride Camels
Why real integration developers ride CamelsWhy real integration developers ride Camels
Why real integration developers ride Camels
 

Recently uploaded

The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
Larry Smarr
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
Matthew Sinclair
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
UiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs ConferenceUiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs Conference
UiPathCommunity
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
ScyllaDB
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
Stephanie Beckett
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
Larry Smarr
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
huseindihon
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
Enterprise Wired
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf
Sally Laouacheria
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
SynapseIndia
 
Measuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at TwitterMeasuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at Twitter
ScyllaDB
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Bert Blevins
 

Recently uploaded (20)

The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
UiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs ConferenceUiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs Conference
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
 
find out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challengesfind out more about the role of autonomous vehicles in facing global challenges
find out more about the role of autonomous vehicles in facing global challenges
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf20240702 Présentation Plateforme GenAI.pdf
20240702 Présentation Plateforme GenAI.pdf
 
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
 
Measuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at TwitterMeasuring the Impact of Network Latency at Twitter
Measuring the Impact of Network Latency at Twitter
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
 

Deep Dive Into Elasticsearch: Establish A Powerful Log Analysis System With Elastic Stack | On Premises vs SaaS Elastic Stack Comparisons.

  • 1. DEEP DIVE INTO ELASTICSEARCH Establish A Powerful Log Analysis System With Elastic Stack. On Premises vs SaaS Elastic Stack Comparisons. Tyler DevOps Engineer NFQ Asia Company
  • 2. Agenda • Intro. • Overview: Elastic Stack. • Establish a powerful log analysis system with Elastic Stack. • Elastic stack options from cloud providers. • Which one would be fit for us? • Cost Reflections. • In conclusion.
  • 3. About Me • In tech for 7+ years. • Technical Project Coordinator @ AVASO Technology Solutions. • Infrastructure Technical Lead @ Betfair Group PLC. • DevOps Engineer @ NFQ Asia. • Member of Vietnam Elasticsearch Community. • Bash/PowerShell languages. • A dog parent :D
  • 4. About NFQ Asia • Member of NFQ Company. • 15+ years’ experience in e-business strategy and software development • 300+ professionals. • 4 countries: offices in Lithuania, Germany, Vietnam, Singapore. • Founded in Vietnam since 2015. • Having organized 5 community events/hackathons in Vietnam.
  • 7. What is Elastic Stack? • Formerly known as ELK Stack. • ELK - The acronym for three open source projects: elasticsearch, logstash, and kibana. • Distributed, scalable, and highly available (both on premises or SaaS). • The Elastic Stack is the next evolution of ELK. • Supports the lightweight Beats data shippers from ES v2.1.1.
  • 8. Elasticsearch • “You know, for Search” • Free, Open Source. • Search engine based on Lucene. • Near real-time searching, analytics and visualization capabilities. • Sophisticated Restful API.
  • 9. Logstash • Open source data collection engine that unifies data from disparate sources, normalizes it and distributes it. • The ingestion workhorse for elasticsearch and more. • Real-time capabilities and pluggable pipeline architecture. • Community-extensible and developer-friendly plugin ecosystem.
  • 10. Kibana • Open source analytics and visualization platform designed to work with elasticsearch. • Specialized for large volumes of streaming and real-time data. • No code, no additional infrastructure required. • Easily and quickly understandable through graphic representation.
  • 11. Beats Platform • “Data shippers” that are installed on servers as agents. • Either elasticsearch directly or through logstash. • Library written based on Golang. • Supports create your own beat for specific use cases.
  • 12. ESTABLISH A POWERFUL LOG ANALYSIS SYSTEM WITH ELASTIC STACK
  • 13. Rationale • What is log? • How do we solve the production issue as usual? • How much time do you spend investigating the production issue? • Where are the archived log? • Visualization and dashboards?
  • 14. The Challenge How do you satisfy the search needs of the application system’s over 2,000 docs per second while simultaneously providing tactical operational insights that help both Development Team and Operation Team iteratively improve the customer experience?
  • 15. The Simple Log Analysis Diagram
  • 17. Scalability Rationale • High availability. • Petabyte-scale data is written and/or read frequently. • High scalability. • Sufficient data allocation. • Costs.
  • 19. The Elasticsearch Hot-Warm Architecture (cont.)
  • 20. ON PREMISES VS SaaS ELASTIC STACK COMPARISONS
  • 21. WHAT IS AWS ELASTICSEARCH SERVICE? • Managed service in AWS Cloud. • Introduced in Oct 2015. • Fully managed; Zero admin. • Highly available and reliable. • Built-in Kibana support. • Integrated with other services in AWS ecosystem.
  • 23. What is Elastic Cloud? • Launched in Oct 2015. • Provided by Elastic. • High provisioning and scaling. • Hosted in the Cloud Providers. • Service-oriented architecture. • Containerization using Docker. • Fully supports custom plugins and API.
  • 25. HOW DO I KNOW WHICH ONE IS FIT WITH ME?
  • 26. Specifications comparison sheet Self-managed Elastic Stack AWS Elasticsearch Service Elastic Cloud Enterprise Pros More options and features. Complete control settings and capacity. Access to other APIs Comprehensive ES monitoring solutions. Lowest costs. SaaS. Simplify the operations via APIs. Security by IAM. Automated snapshots*. Encryption at rest. Monitoring included*. Technical supported. SaaS. Fully control through APIs. Technical Supported. Uptime SLA. Feature-rich and complete monitoring product. Available on Marketplace. Cons Self maintenance. Infrastructure matters. No technical supported. X-Pack limit features. Limited control. Less capacity and scalability. Backup once time per day. No plugins, no logs. Medium expensive. Only support I2 series EC2 instances. Most expensive. Imperfect for AWS-hosted solutions.
  • 28. Costs Comparison Chart 8,400.38 10,678.56 11,512.51 75,303.17 81,375.17 11,316.98 14,500.26 25,201.1525,201.15 32,035.68 34,537.54 203,318.55 219,712.95 28,319.95 38,295.63 50,402.30 0 50,000 100,000 150,000 200,000 250,000 Elastic Stack (AWS) Elastic Stack (GCP) AWS Elasticsearch Services Elastic Cloud (GCP) Elastic Cloud (AWS) Cost($) Service Models One Year One Year (All Upfront) Three Years Three Years (All Upfront) *Costs calculated based on 3TB-data cluster in multi-AZ in Frankfurt region
  • 29. In Conclusion • Elasticsearch leverage the power of analysis ability for both Dev/Ops teams. • Easily operate/maintain the huge cluster of servers and microservices. • Choose the proper architecture depend on application/system. • Estimate the budget to meet the requirements. • Optimize the aggregation to adopt the resources. • High availability oriented system.
  • 30. We are hiring… • Java Senior/Lead Developer • PHP Senior Developer • PHP Technical Lead • Front-end Senior Developer • Front-end Technical Lead • Technical Project Manager Simply send us an email with your enclosed updated CV to: career@nfq.asia
  • 31. Contact Me LinkedIn: linkedin.com/in/tylernguyen91 Email: tai.nguyen@nfq.asia Telegram: @tylern91