The MySQL world is full of tradeoffs and choosing a High Availability (HA) solution is no exception. This session aims to look at all of the alternatives in an unbiased nature. While the landscape will be covered, including but not limited to MySQL replication, MHA, DRBD, Galera Cluster, etc. the focus of the talk will be what is recommended for today, and what to look out for. Thus, this will include extensive deep-dive coverage of ProxySQL, semi-sync replication, Orchestrator, MySQL Router, and Galera Cluster variants like Percona XtraDB Cluster and MariaDB Galera Cluster. I will also touch on group replication.
Learn how we do this for our nearly 4000+ customers!
OSDC 2018 | Spicing up VMWare with Ansible and InSpec by Martin Schurz and S...
VMWare is a common hypervisor choice in large organizations, and it comes with a zoo of additional tools, options and licenses. But once you add a little bit of OpenSource to the mix, things start to get interesting. There are open APIs which integrate well with Tools like Ansible and InSpec. So it is easy to write your own scripts to verify a configuration or harden a ESX host. We will show you, how we automated our deployment, what problems we encountered and how we added some nice features.
Building stateful applications on Kubernetes with Rook
Deploying stateful applications such a Wordpress and Jenkins on top of Kubernetes or any other container orchestrator can be a challenging task. In this context, Rook will be used to showcase how to automatically manage the volume's lifecycle through the its Kubernetes operators (operator pattern approach) by leveraging the recently added CSI GA support.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.
Opcito Technologies is a proud partner with Kubernetes, an open-source system for container orchestration.
We will be talking about:
• Features of Kubernetes 1.6
• RBAC Configurations
• RBAC Use Cases
• Running Prometheus in Kubernetes
• Prometheus Operator - Deployment, Cluster & Service Monitoring
The document provides information and tips for preparing for the CKA (Certified Kubernetes Administrator) exam. It outlines what to know prior to taking the exam such as it being offered by CNCF, the exam format, curriculum covered, and required equipment. It recommends practicing Linux CLI, kubectl commands, getting familiar with Kubernetes documentation, examples and APIs. Suggested study methods include using Minikube, Kube-Ansible, online courses, and YouTube videos. Registering for the exam and free learning materials are also mentioned.
Fabric8 is a tool that aims to reduce the gap between development and operations by allowing developers to deploy and manage applications from development through production. It uses profiles and containers to deploy applications in a unified way across environments. Fabric8 supports various containers like Tomcat, Docker, and OpenShift. It includes integrated monitoring via Hawt.io and a shell to help with scripting and diagnostics. The goal is to make developers more self-sufficient by enabling them to take on more operations tasks in their own sandboxes.
Kubernetes currently has two load balancing mode: userspace and IPTables. They both have limitation on scalability and performance. We introduced IPVS as third kube-proxy mode which scales kubernetes load balancer to support 50,000 services. Beyond that, control plane needs to be optimized in order to deploy 50,000 services. We will introduce alternative solutions and our prototypes with detailed performance data.
Kubernetes Introduction & Whats new in Kubernetes 1.6
- Introduction to Kubernetes features
- A look at Kubernetes Networking and Service Discovery
- New features in Kubernetes 1.6
- Kubernetes Installation options
To know more about our Kubernetes expertise, visit our center of excellence at: http://www.opcito.com/kubernetes/
This document discusses container image management in enterprises and introduces Project Harbor, an open source container registry. It covers key topics like container image basics, Project Harbor features, maintaining consistency of images between environments, security and access control of images, image distribution strategies, and high availability of container registries. Project Harbor provides an enterprise-grade private registry with features for user management, image replication, security, and integration with systems like LDAP. It aims to help organizations securely manage the distribution and storage of container images.
The document provides best practices for deploying SUSE CaaS Platform. It discusses requirements like hardware needs, software subscriptions required, and support options. It covers planning and sizing considerations like cluster topology and disk space needs. Deployment best practices include steps like preparing the infrastructure, installing base software, verifying the infrastructure, installing CaaS Platform, and deploying Kubernetes addons. Testing and operations topics like monitoring, logging, backups are also covered.
The document provides an overview of Red Hat OpenShift Enterprise, which is Red Hat's container application platform. It discusses key OpenShift concepts like Docker containers, Kubernetes cluster management, projects for sandboxing environments, image builds, deployments, services and routes. It also summarizes added value of OpenShift over plain Docker and Kubernetes through features like templates, web console, multi-language support, automated builds and deployments, and enterprise capabilities.
Service Discovery in kubernetes is all about how services of kubernetes get discovered internally and externally. How does a single POD communicate to another POD the within the cluster and how does a user request reach to a specific POD in the cluster? These are some questions that are answered by this TOPIC.
In this video from the Docker Workshop at ISC 2015, Christian Kniep from QNIB Solutions shows how he uses Docker in his efforts to provide a HPC software stack in a box, encapsulating each layer in the HPC stack within a Linux Container.
Watch the video presentation: http://wp.me/p3RLHQ-eos
Learn more: http://qnib.org/about/
Presentation delivered at LinuxCon China 2017. Rethinking the Operating System.
A new wave of Operating Systems optimized for containers appeared on the horizon making us excited and puzzeled at the same time.
"Why do we need anything different for containers when traditional OSs served us well in the last 25+ years?" "Isn't Kubernetes just another package to install on top of my favorite distro?"" Will this obsolete my whole infrastructure?" are some of the questions this talk will shed some light on.
Explore the journey SUSE made in rethinking the OS: From a conservative linux distribution to a platform that goes hand in hand with the needs of Microservices.
You will get an insight at what lessons were learned during the intense development effort that lead to SUSE Containers as a Service Platform, how the obstacles along the way were lifted and why "Upstream first" is - and should always be - the rule.
In this deck from the Docker Workshop at ISC 2015, Andreas Schmidt from Cassini Consulting describes Docker in a Nutshell
"As the newest flavor of Linux Containers, Docker gained a lot of momentum in the last 12 months. With a very convenient and open API-driven architecture Docker is able to help decrease the complexity of operations and increase the productivity of computation. During the last two years Andreas, Christian, and Wolfgang gained a lot of experience with Docker and were thrilled by its possible impact early on. Andreas started working with Docker in mid-2013 and is interested in developing tools for solving Enterprise IT requirements on networking and security. In 2014 he held talks and workshops about these topics. Christian started using Docker in 2013 to virtualize a complete HPC cluster stack and since then held multiple talks about how Docker might impact HPC. Wolfgang and his partner Burak Yenier introduced Docker as a corner-stone of the UberCloud Marketplace to drastically improve and simplify access to HPC cloud resources. UberCloud just announced their new containers for computational fluid dynamics software like Fluent, STAR-CCM+ and OpenFOAM."
Watch the video presentation: http://wp.me/p3RLHQ-enP
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Presentation delivered at LinuxCon China 2017.
Operating systems need to move faster without sacrificing stability. New hardware, new software features, and bugfixes are making it into distribution components every day. To maintain stability, packagers and distribution developers are looking toward lessons learned in the DevOps movement to implement Continuous Integration/Continuous Delivery (CI/CD) workflows that provide quicker test feedback to developers.
This talk highlights some of the coming trends in Fedora such as: streamlined base package sets, userspace applications delivered as containers, continuous validation of individual distro components and the distro as a whole, and collaboration with the CentOS Project.
Have you ever tried Java on AWS Lambda but found that the cold-start latency and memory usage were far too high? In this session, we will show how we optimized Java for serverless applications by leveraging GraalVM with Quarkus to provide both supersonic startup speed and a subatomic memory footprint.
SUSE CaaSP: deploy OpenFaaS and Ethereum Blockchain on Kubernetes
This document discusses potential use cases for Kubernetes and provides examples of deploying serverless/Function as a Service (FaaS) workloads and blockchain databases on Kubernetes. It introduces OpenFaaS as an easy way to deploy FaaS on Kubernetes and deploy a demo Ethereum blockchain on Kubernetes to illustrate how blockchain concepts map to Kubernetes components. The document encourages finding a new use case in your organization to start using Kubernetes and provides resources for learning more about deploying Kubernetes.
Best practices for MySQL/MariaDB Server/Percona Server High Availability
Best practices for MySQL/MariaDB Server/Percona Server High Availability - presented at Percona Live Amsterdam 2016. The focus is on picking the right High Availability solution, discussing replication, handling failure (yes, you can achieve a quick automatic failover), proxies (there are plenty), HA in the cloud/geographical redundancy, sharding solutions, how newer versions of MySQL help you, and what to watch for next.
The MariaDB/MySQL world is full of tradeoffs, and choosing a high availability (HA) solution is no exception. This session aims to look at all the alternatives in an unbiased way. Preference is of course only given to open source solutions.
How do you choose between: asynchronous/semi-synchronous/synchronous replication, MHA (MySQL high availability tools), DRBD, Tungsten Replicator, or Galera Cluster? Do you integrate Pacemaker and Heartbeat like Percona Replication Manager? The cloud brings even more fun, especially if you are dealing with a hybrid cloud and must think about geographical redundancy.
What about newer solutions like using Consul for MySQL HA?
When you’ve decided on your solution, how do you provision and monitor these solutions?
This and more will be covered in a walkthrough of MySQL HA options and when to apply them.
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
MariaDB is like the "new" MySQL, and its available everywhere. This talk was given at LinuxCon Europe in Dublin in October 2015. Learn about all the new features, considering the release was just around the corner. Changes in replication are also very interesting
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
Presented at the Tokyo MariaDB Server meetup in July 2016, this is an overview of what you can see and use in MariaDB Server 10.1, but more importantly what is planned to arrive in 10.2
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...
Scalability with MariaDB and MaxScale talks about MariaDB 10, and MaxScale, a pluggable router for your queries. These are technologies developed at MariaDB Corporation, made opensource, and will help scale your MariaDB and MySQL workloads
Meet MariaDB 10.1 at the Bulgaria Web Summit, held in Sofia in February 2016. Learn all about MariaDB Server, and the new features like encryption, audit plugins, and more.
MariaDB 10: A MySQL Replacement. Current up to 10.0.9, right before the 10.0.10 GA release presented the weekend before the release in Hong Kong, at the Hong Kong Open Source Conference.
This document provides an overview of MariaDB 10 and the MariaDB Foundation. It discusses the history and development of MariaDB, including key features added in versions 5.1 through 10.0 such as new storage engines, performance improvements, and features backported from MySQL. It outlines the goals of MariaDB to be compatible with MySQL while adding new features, and describes the community-led development model. The roadmap aims to have MariaDB be a drop-in replacement for MySQL 5.6 by releasing version 10.1.
Choosing between Codership's MySQL Galera, MariaDB Galera Cluster and Percona...
There are many Galera Cluster distributions and sometimes differences are well worth noting. We get a lot of queries about which Galera Cluster to use, or why one should use one distribution over the other.
Learn about Galera Cluster with MySQL 5.7 from Codership, and we’ll compare it with Galera Cluster 4 with MariaDB 10.4, and Percona XtraDB Cluster 5.7 with Galera 3. This is also the webinar where we preview Galera Cluster 4 with MySQL 8.0 as well as compare it with the preview release of Percona XtraDB Cluster 8.0.
Overall, learn why distributions exists, and how you can get the most out of your Galera Cluster experience.
This is my third iteration of the talk presented in Tokyo, Japan - first was at a keynote at rootconf.in in April 2016, then at the MySQL meetup in New York, and now for dbtechshowcase. The focus is on database failures of the past, and how modern MySQL / MariaDB Server technologies could have helped them avoid such failure. The focus is on backups and verification, replication and failover, and security and encryption.
The document summarizes the history and current state of the MySQL database server ecosystem. It discusses the origins and development of MySQL, MariaDB, Percona Server, and other related projects. It also describes some of the key features and innovations in recent versions of these database servers. The ecosystem is very active with contributions from many organizations and the future remains promising with ongoing work.
MySQL is a unique adult (now 21 years old) in many ways. It supports plugins. It supports storage engines. It is also owned by Oracle, thus birthing two branches of the popular opensource database: Percona Server and MariaDB Server. It also once spawned a fork: Drizzle. Lately a consortium of web scale users (think a chunk of the top 10 sites out there) have spawned WebScaleSQL.
You're a busy DBA having to maintain a mix of this. Or you're a CIO planning to choose one branch. How do you go about picking? Supporting multiple databases? Find out more in this talk. Also covered is a deep-dive into what feature differences exist between MySQL/Percona Server/MariaDB/WebScaleSQL, how distributions package the various databases differently. Within the hour, you'll be informed about the past, the present, and hopefully be knowledgeable enough to know what to pick in the future.
Note, there will also be coverage of the various trees around WebScaleSQL, like the Facebook tree, the Alibaba tree as well as the Twitter tree.
OSDC 2017 | Lessons from database failures by Colin Charles
Lets learn from MySQL failures at scale, because we tie in the topic of High Availability, in where people are thinking about geographical redundancy, and even things like automatic failover. In the talk there will be case study material, e.g. where automatic failure caused a site to go offline, where a social network started of with not using fully automated failovers but evolved, etc. How is the MySQL world making things better, for example by allowing you to use semi-synchronous replication to run fully scalable services. The talk starts off with an even almost stupid example of how a business died due to incorrect MySQL backup procedures. It will go on to talk about security and encryption at rest as well. So a mix of problems from the field, big “fail whales”, and how you should avoid them by properly architecting solutions.
MariaDB started life as a database to host the Maria storage engine in 2009. Not long after its inception, the MySQL community went through yet another change in ownership, and it was deemed that MariaDB will be a complete database branch developed to extend MySQL, but with constant merging of upstream changes.
The goal of the MariaDB project is to ensure that everyone is part of the community, including employees of the major steering companies. MariaDB also features enhanced features, some of which are common with the Percona Performance Server. Most importantly, MariaDB is a drop-in replacement and is completely backward compatible with MySQL. In 2010, MariaDB released 5.1 in February, and 5.2 in November – two major releases in a span of one calendar year is a feat that was achieved!
DBAs and developers alike will gain an introduction to MariaDB, what is different with MySQL, how to make use of the feature enhancements, and more.
The MySQL ecosystem - understanding it, not running away from it!
You're a busy DBA thinking about having to maintain a mix of this. Or you're a CIO planning to choose one branch over another. How do you go about picking? Supporting multiple databases? Find out more in this talk. Also covered is a deep-dive into what feature differences exist between MySQL/Percona Server/MariaDB Server. Within 20 minutes, you'll leave informed and knowledgable on what to pick.
A base blog post to get started: https://www.percona.com/blog/2017/11/02/mysql-vs-mariadb-reality-check/
This was a short 25 minute talk, but we go into a bit of a history of MySQL, how the branches and forks appeared, what's sticking around today (branch? Percona Server. Fork? MariaDB Server). What should you use? Think about what you need today and what the roadmap holds.
MariaDB is a community developed branch of MySQL that is feature enhanced and backward compatible. It aims to be a 100% drop-in replacement for MySQL that is stable, bug-free, and released under the GPLv2 license. Major releases of MariaDB include new storage engines like XtraDB and Aria, as well as new features for performance, scalability, and compatibility. MariaDB is developed as an open source project and supported by Monty Program and other community contributors and service providers.
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdf
MySQL is still hot, with Percona XtraDB Cluster (PXC) and MariaDB Server. Welcome back post-pandemic to see what is on offer in the current ecosystem.
Did you know that Amazon RDS now uses semi-sync replication rather than DRBD for multi-AZ deployments? Did you know that Galera Cluster for MySQL 8 is much more efficient with CLONE SST rather than using the xtrabackup method for SST? Did you know that Percona Server continues to extend MyRocks? Did you know that MariaDB Server has more Oracle syntax compatibility? This and more will be covered in the session, while short and quick, should leave you wandering to discover new features for production.
OSDC 2018 | Highly Available Cloud Foundry on Kubernetes by Cornelius SchumacherNETWAYS
This document discusses running Cloud Foundry on Kubernetes to provide highly available cloud platforms. It begins with an overview of cloud computing models and introduces Cloud Foundry. It then discusses deploying Cloud Foundry using Kubernetes primitives like pods, services, and stateful sets for high availability. The document demonstrates how to install Cloud Foundry on Kubernetes using Helm charts and configure for high availability. It shows the components have been made highly available to prevent downtime during failures or upgrades. Finally, it provides a demo of deploying a sample application on Cloud Foundry on Kubernetes under chaotic conditions to showcase the high availability.
OSDC 2018 | Spicing up VMWare with Ansible and InSpec by Martin Schurz and S...NETWAYS
VMWare is a common hypervisor choice in large organizations, and it comes with a zoo of additional tools, options and licenses. But once you add a little bit of OpenSource to the mix, things start to get interesting. There are open APIs which integrate well with Tools like Ansible and InSpec. So it is easy to write your own scripts to verify a configuration or harden a ESX host. We will show you, how we automated our deployment, what problems we encountered and how we added some nice features.
Building stateful applications on Kubernetes with RookRoberto Hashioka
Deploying stateful applications such a Wordpress and Jenkins on top of Kubernetes or any other container orchestrator can be a challenging task. In this context, Rook will be used to showcase how to automatically manage the volume's lifecycle through the its Kubernetes operators (operator pattern approach) by leveraging the recently added CSI GA support.
Securing & Monitoring Your K8s Cluster with RBAC and Prometheus”.Opcito Technologies
Opcito Technologies is a proud partner with Kubernetes, an open-source system for container orchestration.
We will be talking about:
• Features of Kubernetes 1.6
• RBAC Configurations
• RBAC Use Cases
• Running Prometheus in Kubernetes
• Prometheus Operator - Deployment, Cluster & Service Monitoring
The document provides information and tips for preparing for the CKA (Certified Kubernetes Administrator) exam. It outlines what to know prior to taking the exam such as it being offered by CNCF, the exam format, curriculum covered, and required equipment. It recommends practicing Linux CLI, kubectl commands, getting familiar with Kubernetes documentation, examples and APIs. Suggested study methods include using Minikube, Kube-Ansible, online courses, and YouTube videos. Registering for the exam and free learning materials are also mentioned.
Fabric8 - Being devOps doesn't suck anymoreHenryk Konsek
Fabric8 is a tool that aims to reduce the gap between development and operations by allowing developers to deploy and manage applications from development through production. It uses profiles and containers to deploy applications in a unified way across environments. Fabric8 supports various containers like Tomcat, Docker, and OpenShift. It includes integrated monitoring via Hawt.io and a shell to help with scripting and diagnostics. The goal is to make developers more self-sufficient by enabling them to take on more operations tasks in their own sandboxes.
Kubernetes currently has two load balancing mode: userspace and IPTables. They both have limitation on scalability and performance. We introduced IPVS as third kube-proxy mode which scales kubernetes load balancer to support 50,000 services. Beyond that, control plane needs to be optimized in order to deploy 50,000 services. We will introduce alternative solutions and our prototypes with detailed performance data.
- Introduction to Kubernetes features
- A look at Kubernetes Networking and Service Discovery
- New features in Kubernetes 1.6
- Kubernetes Installation options
To know more about our Kubernetes expertise, visit our center of excellence at: http://www.opcito.com/kubernetes/
This document discusses container image management in enterprises and introduces Project Harbor, an open source container registry. It covers key topics like container image basics, Project Harbor features, maintaining consistency of images between environments, security and access control of images, image distribution strategies, and high availability of container registries. Project Harbor provides an enterprise-grade private registry with features for user management, image replication, security, and integration with systems like LDAP. It aims to help organizations securely manage the distribution and storage of container images.
The document provides best practices for deploying SUSE CaaS Platform. It discusses requirements like hardware needs, software subscriptions required, and support options. It covers planning and sizing considerations like cluster topology and disk space needs. Deployment best practices include steps like preparing the infrastructure, installing base software, verifying the infrastructure, installing CaaS Platform, and deploying Kubernetes addons. Testing and operations topics like monitoring, logging, backups are also covered.
Open shift enterprise 3.1 paas on kubernetesSamuel Terburg
The document provides an overview of Red Hat OpenShift Enterprise, which is Red Hat's container application platform. It discusses key OpenShift concepts like Docker containers, Kubernetes cluster management, projects for sandboxing environments, image builds, deployments, services and routes. It also summarizes added value of OpenShift over plain Docker and Kubernetes through features like templates, web console, multi-language support, automated builds and deployments, and enterprise capabilities.
Service Discovery in kubernetes is all about how services of kubernetes get discovered internally and externally. How does a single POD communicate to another POD the within the cluster and how does a user request reach to a specific POD in the cluster? These are some questions that are answered by this TOPIC.
In this video from the Docker Workshop at ISC 2015, Christian Kniep from QNIB Solutions shows how he uses Docker in his efforts to provide a HPC software stack in a box, encapsulating each layer in the HPC stack within a Linux Container.
Watch the video presentation: http://wp.me/p3RLHQ-eos
Learn more: http://qnib.org/about/
Presentation delivered at LinuxCon China 2017. Rethinking the Operating System.
A new wave of Operating Systems optimized for containers appeared on the horizon making us excited and puzzeled at the same time.
"Why do we need anything different for containers when traditional OSs served us well in the last 25+ years?" "Isn't Kubernetes just another package to install on top of my favorite distro?"" Will this obsolete my whole infrastructure?" are some of the questions this talk will shed some light on.
Explore the journey SUSE made in rethinking the OS: From a conservative linux distribution to a platform that goes hand in hand with the needs of Microservices.
You will get an insight at what lessons were learned during the intense development effort that lead to SUSE Containers as a Service Platform, how the obstacles along the way were lifted and why "Upstream first" is - and should always be - the rule.
In this deck from the Docker Workshop at ISC 2015, Andreas Schmidt from Cassini Consulting describes Docker in a Nutshell
"As the newest flavor of Linux Containers, Docker gained a lot of momentum in the last 12 months. With a very convenient and open API-driven architecture Docker is able to help decrease the complexity of operations and increase the productivity of computation. During the last two years Andreas, Christian, and Wolfgang gained a lot of experience with Docker and were thrilled by its possible impact early on. Andreas started working with Docker in mid-2013 and is interested in developing tools for solving Enterprise IT requirements on networking and security. In 2014 he held talks and workshops about these topics. Christian started using Docker in 2013 to virtualize a complete HPC cluster stack and since then held multiple talks about how Docker might impact HPC. Wolfgang and his partner Burak Yenier introduced Docker as a corner-stone of the UberCloud Marketplace to drastically improve and simplify access to HPC cloud resources. UberCloud just announced their new containers for computational fluid dynamics software like Fluent, STAR-CCM+ and OpenFOAM."
Watch the video presentation: http://wp.me/p3RLHQ-enP
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Presentation delivered at LinuxCon China 2017.
Operating systems need to move faster without sacrificing stability. New hardware, new software features, and bugfixes are making it into distribution components every day. To maintain stability, packagers and distribution developers are looking toward lessons learned in the DevOps movement to implement Continuous Integration/Continuous Delivery (CI/CD) workflows that provide quicker test feedback to developers.
This talk highlights some of the coming trends in Fedora such as: streamlined base package sets, userspace applications delivered as containers, continuous validation of individual distro components and the distro as a whole, and collaboration with the CentOS Project.
Have you ever tried Java on AWS Lambda but found that the cold-start latency and memory usage were far too high? In this session, we will show how we optimized Java for serverless applications by leveraging GraalVM with Quarkus to provide both supersonic startup speed and a subatomic memory footprint.
SUSE CaaSP: deploy OpenFaaS and Ethereum Blockchain on KubernetesJuan Herrera Utande
This document discusses potential use cases for Kubernetes and provides examples of deploying serverless/Function as a Service (FaaS) workloads and blockchain databases on Kubernetes. It introduces OpenFaaS as an easy way to deploy FaaS on Kubernetes and deploy a demo Ethereum blockchain on Kubernetes to illustrate how blockchain concepts map to Kubernetes components. The document encourages finding a new use case in your organization to start using Kubernetes and provides resources for learning more about deploying Kubernetes.
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityColin Charles
Best practices for MySQL/MariaDB Server/Percona Server High Availability - presented at Percona Live Amsterdam 2016. The focus is on picking the right High Availability solution, discussing replication, handling failure (yes, you can achieve a quick automatic failover), proxies (there are plenty), HA in the cloud/geographical redundancy, sharding solutions, how newer versions of MySQL help you, and what to watch for next.
Best practices for MySQL High AvailabilityColin Charles
The MariaDB/MySQL world is full of tradeoffs, and choosing a high availability (HA) solution is no exception. This session aims to look at all the alternatives in an unbiased way. Preference is of course only given to open source solutions.
How do you choose between: asynchronous/semi-synchronous/synchronous replication, MHA (MySQL high availability tools), DRBD, Tungsten Replicator, or Galera Cluster? Do you integrate Pacemaker and Heartbeat like Percona Replication Manager? The cloud brings even more fun, especially if you are dealing with a hybrid cloud and must think about geographical redundancy.
What about newer solutions like using Consul for MySQL HA?
When you’ve decided on your solution, how do you provision and monitor these solutions?
This and more will be covered in a walkthrough of MySQL HA options and when to apply them.
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)Colin Charles
MariaDB is like the "new" MySQL, and its available everywhere. This talk was given at LinuxCon Europe in Dublin in October 2015. Learn about all the new features, considering the release was just around the corner. Changes in replication are also very interesting
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB MeetupColin Charles
Presented at the Tokyo MariaDB Server meetup in July 2016, this is an overview of what you can see and use in MariaDB Server 10.1, but more importantly what is planned to arrive in 10.2
[db tech showcase Tokyo 2014] B15: Scalability with MariaDB and MaxScale by ...Insight Technology, Inc.
Scalability with MariaDB and MaxScale talks about MariaDB 10, and MaxScale, a pluggable router for your queries. These are technologies developed at MariaDB Corporation, made opensource, and will help scale your MariaDB and MySQL workloads
Meet MariaDB 10.1 at the Bulgaria Web SummitColin Charles
Meet MariaDB 10.1 at the Bulgaria Web Summit, held in Sofia in February 2016. Learn all about MariaDB Server, and the new features like encryption, audit plugins, and more.
MariaDB 10: A MySQL Replacement - HKOSC Colin Charles
MariaDB 10: A MySQL Replacement. Current up to 10.0.9, right before the 10.0.10 GA release presented the weekend before the release in Hong Kong, at the Hong Kong Open Source Conference.
Maria db 10 and the mariadb foundation(colin)kayokogoto
This document provides an overview of MariaDB 10 and the MariaDB Foundation. It discusses the history and development of MariaDB, including key features added in versions 5.1 through 10.0 such as new storage engines, performance improvements, and features backported from MySQL. It outlines the goals of MariaDB to be compatible with MySQL while adding new features, and describes the community-led development model. The roadmap aims to have MariaDB be a drop-in replacement for MySQL 5.6 by releasing version 10.1.
There are many Galera Cluster distributions and sometimes differences are well worth noting. We get a lot of queries about which Galera Cluster to use, or why one should use one distribution over the other.
Learn about Galera Cluster with MySQL 5.7 from Codership, and we’ll compare it with Galera Cluster 4 with MariaDB 10.4, and Percona XtraDB Cluster 5.7 with Galera 3. This is also the webinar where we preview Galera Cluster 4 with MySQL 8.0 as well as compare it with the preview release of Percona XtraDB Cluster 8.0.
Overall, learn why distributions exists, and how you can get the most out of your Galera Cluster experience.
This is my third iteration of the talk presented in Tokyo, Japan - first was at a keynote at rootconf.in in April 2016, then at the MySQL meetup in New York, and now for dbtechshowcase. The focus is on database failures of the past, and how modern MySQL / MariaDB Server technologies could have helped them avoid such failure. The focus is on backups and verification, replication and failover, and security and encryption.
The document summarizes the history and current state of the MySQL database server ecosystem. It discusses the origins and development of MySQL, MariaDB, Percona Server, and other related projects. It also describes some of the key features and innovations in recent versions of these database servers. The ecosystem is very active with contributions from many organizations and the future remains promising with ongoing work.
MySQL is a unique adult (now 21 years old) in many ways. It supports plugins. It supports storage engines. It is also owned by Oracle, thus birthing two branches of the popular opensource database: Percona Server and MariaDB Server. It also once spawned a fork: Drizzle. Lately a consortium of web scale users (think a chunk of the top 10 sites out there) have spawned WebScaleSQL.
You're a busy DBA having to maintain a mix of this. Or you're a CIO planning to choose one branch. How do you go about picking? Supporting multiple databases? Find out more in this talk. Also covered is a deep-dive into what feature differences exist between MySQL/Percona Server/MariaDB/WebScaleSQL, how distributions package the various databases differently. Within the hour, you'll be informed about the past, the present, and hopefully be knowledgeable enough to know what to pick in the future.
Note, there will also be coverage of the various trees around WebScaleSQL, like the Facebook tree, the Alibaba tree as well as the Twitter tree.
OSDC 2017 | Lessons from database failures by Colin CharlesNETWAYS
Lets learn from MySQL failures at scale, because we tie in the topic of High Availability, in where people are thinking about geographical redundancy, and even things like automatic failover. In the talk there will be case study material, e.g. where automatic failure caused a site to go offline, where a social network started of with not using fully automated failovers but evolved, etc. How is the MySQL world making things better, for example by allowing you to use semi-synchronous replication to run fully scalable services. The talk starts off with an even almost stupid example of how a business died due to incorrect MySQL backup procedures. It will go on to talk about security and encryption at rest as well. So a mix of problems from the field, big “fail whales”, and how you should avoid them by properly architecting solutions.
MariaDB started life as a database to host the Maria storage engine in 2009. Not long after its inception, the MySQL community went through yet another change in ownership, and it was deemed that MariaDB will be a complete database branch developed to extend MySQL, but with constant merging of upstream changes.
The goal of the MariaDB project is to ensure that everyone is part of the community, including employees of the major steering companies. MariaDB also features enhanced features, some of which are common with the Percona Performance Server. Most importantly, MariaDB is a drop-in replacement and is completely backward compatible with MySQL. In 2010, MariaDB released 5.1 in February, and 5.2 in November – two major releases in a span of one calendar year is a feat that was achieved!
DBAs and developers alike will gain an introduction to MariaDB, what is different with MySQL, how to make use of the feature enhancements, and more.
The MySQL ecosystem - understanding it, not running away from it! Colin Charles
You're a busy DBA thinking about having to maintain a mix of this. Or you're a CIO planning to choose one branch over another. How do you go about picking? Supporting multiple databases? Find out more in this talk. Also covered is a deep-dive into what feature differences exist between MySQL/Percona Server/MariaDB Server. Within 20 minutes, you'll leave informed and knowledgable on what to pick.
A base blog post to get started: https://www.percona.com/blog/2017/11/02/mysql-vs-mariadb-reality-check/
This was a short 25 minute talk, but we go into a bit of a history of MySQL, how the branches and forks appeared, what's sticking around today (branch? Percona Server. Fork? MariaDB Server). What should you use? Think about what you need today and what the roadmap holds.
MariaDB is a community developed branch of MySQL that is feature enhanced and backward compatible. It aims to be a 100% drop-in replacement for MySQL that is stable, bug-free, and released under the GPLv2 license. Major releases of MariaDB include new storage engines like XtraDB and Aria, as well as new features for performance, scalability, and compatibility. MariaDB is developed as an open source project and supported by Monty Program and other community contributors and service providers.
MySQL Ecosystem in 2023 - FOSSASIA'23 - Alkin.pptx.pdfAlkin Tezuysal
MySQL is still hot, with Percona XtraDB Cluster (PXC) and MariaDB Server. Welcome back post-pandemic to see what is on offer in the current ecosystem.
Did you know that Amazon RDS now uses semi-sync replication rather than DRBD for multi-AZ deployments? Did you know that Galera Cluster for MySQL 8 is much more efficient with CLONE SST rather than using the xtrabackup method for SST? Did you know that Percona Server continues to extend MyRocks? Did you know that MariaDB Server has more Oracle syntax compatibility? This and more will be covered in the session, while short and quick, should leave you wandering to discover new features for production.
Similar to OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+ by Colin Charles (20)
Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent
Connectors integrate Apache Kafka® with external data systems, enabling you to move away from a brittle spaghetti architecture to one that is more streamlined, secure, and future-proof. However, if your team still spends multiple dev cycles building and managing connectors using just open source Kafka Connect, it’s time to consider a faster and cost-effective alternative.
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...Hironori Washizaki
Hironori Washizaki, "Charting a Course for Equity: Strategies for Overcoming Challenges and Promoting Inclusion in the Metaverse", IEEE COMPSAC 2024 D&I Panel, 2024.
Efficient hot work permit software for safe, streamlined work permit management and compliance. Enhance safety today. Contact us on +353 214536034.
https://sheqnetwork.com/work-permit/
introduction of Ansys software and basic and advance knowledge of modelling s...sachin chaurasia
Ansys Mechanical enables you to solve complex structural engineering problems and make better, faster design decisions. With the finite element analysis (FEA) solvers available in the suite, you can customize and automate solutions for your structural mechanics problems and parameterize them to analyze multiple design scenarios. Ansys Mechanical is a dynamic tool that has a complete range of analysis tools.
Discover the Power of ONEMONITAR: The Ultimate Mobile Spy App for Android Dev...onemonitarsoftware
Unlock the full potential of mobile monitoring with ONEMONITAR. Our advanced and discreet app offers a comprehensive suite of features, including hidden call recording, real-time GPS tracking, message monitoring, and much more.
Perfect for parents, employers, and anyone needing a reliable solution, ONEMONITAR ensures you stay informed and in control. Explore the key features of ONEMONITAR and see why it’s the trusted choice for Android device monitoring.
Share this infographic to spread the word about the ultimate mobile spy app!
Ansys Mechanical enables you to solve complex structural engineering problems and make better, faster design decisions. With the finite element analysis (FEA) solvers available in the suite, you can customize and automate solutions for your structural mechanics problems and parameterize them to analyze multiple design scenarios. Ansys Mechanical is a dynamic tool that has a complete range of analysis tools.
Explore the rapid development journey of TryBoxLang, completed in just 48 hours. This session delves into the innovative process behind creating TryBoxLang, a platform designed to showcase the capabilities of BoxLang by Ortus Solutions. Discover the challenges, strategies, and outcomes of this accelerated development effort, highlighting how TryBoxLang provides a practical introduction to BoxLang's features and benefits.
WhatsApp Tracker - Tracking WhatsApp to Boost Online Safety.pdfonemonitarsoftware
WhatsApp Tracker Software is an effective tool for remotely tracking the target’s WhatsApp activities. It allows users to monitor their loved one’s online behavior to ensure appropriate interactions for responsive device use.
Download this PPTX file and share this information to others.
OSDC 2018 | Scaling & High Availability MySQL learnings from the past decade+ by Colin Charles
1. Scaling & High Availability MySQL
learnings from the past decade+
Colin Charles, Chief Evangelist, Percona Inc.
colin.charles@percona.com / byte@bytebot.net
http://www.bytebot.net/blog/ | @bytebot on Twitter
OSDC.de, Berlin, Germany
12 June 2018
2. whoami
• Chief Evangelist, Percona Inc
• Focusing on the MySQL ecosystem (MySQL, Percona Server, MariaDB
Server), as well as the MongoDB ecosystem (Percona Server for
MongoDB) + 100% open source tools from Percona like Percona
Monitoring & Management, Percona xtrabackup, Percona Toolkit, etc.
• Founding team of MariaDB Server (2009-2016), previously at Monty
Program Ab, merged with SkySQL Ab, now MariaDB Corporation
• Formerly MySQL AB (exit: Sun Microsystems)
• Past lives include The Fedora Project (FESCO), OpenOffice.org
• MySQL Community Contributor of the Year Award winner 2014
7. Uptime
Percentile target Max downtime per year
90% 36 days
99% 3.65 days
99.5% 1.83 days
99.9% 8.76 hours
99.99% 52.56 minutes
99.999% 5.25 minutes
99.9999% 31.5 seconds
8. Estimates of levels of availability
Method
Level of
Availability
Simple replication 98-99.9%
Master-Master/MMM 99%
SAN 99.5-99.9%
DRBD, MHA 99.9%
NDBCluster, Galera Cluster,
InnoDB Cluster
99.999%
9. HA is Redundancy
• RAID: disk crashes? Another works
• Clustering: server crashes? Another works
• Power: fuse blows? Redundant power supplies
• Network: Switch/NIC crashes? 2nd network route
• Geographical: Datacenter offline/destroyed? Computation to another
DC
10. Durability
• Data stored on disks
• Is it really written to the disk?
▪ being durable means calling fsync() on each commit
• Is it written in a transactional way to guarantee atomicity, crash
safety, integrity?
11. High Availability for databases
• HA is harder for databases
• Hardware resources and data need to be redundant
• Remember, this isn’t just data - constantly changing data
• HA means the operation can continue uninterrupted, not by
restoring a new/backup server
• uninterrupted: measured in percentiles
12. Redundancy through client-side XA
transactions
• Client writes to 2 independent but identical databases
• HA-JDBC (http://ha-jdbc.github.io/)
• No replication anywhere
13. InnoDB “recovery” time
• innodb_log_file_size
• larger = longer recovery times
• In 2007, Wikipedia reported 40 minutes with 256MB logs;
sometimes it takes 5-10 minutes, sometimes hours
• Percona Server 5.5 (XtraDB) - innodb_recovery_stats
• But today, there has been paradigm change…
14. Redundancy through shared storage
• Requires specialist hardware, like a SAN
• Complex to operate
• One set of data is your single point of failure
• Cold standby
• failover 1-30 minutes
• this isn’t scale-out
• Active/Active solutions: Oracle RAC
15. Redundancy through disk replication
• DRBD
• Linux administration vs. DBA skills
• Synchronous
• Second set of data inaccessible for use
• Passive server acting as hot standby
• Failover: 1-30 minutes
• Performance hit compared to single node performance, with higher
average latencies
16. Redundancy through MySQL replication
• MySQL replication
• Tungsten Replicator
• Galera Cluster
• MySQL InnoDB Cluster
• MySQL Cluster (NDBCLUSTER)
• Storage requirements are multiplied
• Huge potential for scaling out
17. MySQL Replication
• Statement based generally
• Row based became available in 5.1, and the default in 5.7
• mixed-mode, resulting in STATEMENT except if calling
▪ UUID function, UDF, CURRENT_USER/USER function, LOAD_FILE
function
▪ 2 or more AUTO_INCREMENT columns updated with same statement
▪ server variable used in statement
▪ storage engine doesn’t allow statement based replication, like
NDBCLUSTER
▪ default in MariaDB Server 10.2
18. MySQL Replication II
• Asynchronous by default
• Semi-synchronous plugin in 5.5+
• However the holy grail of fully synchronous replication is not part of
standard MySQL replication (yet?)
• MariaDB Galera Cluster is built-in to MariaDB Server 10.1
• MySQL InnoDB Cluster is available in MySQL 8.0, combining the
likes of group replication (the Galera Cluster equivalent), InnoDB,
and mysqlsh for maintaining it; it also uses MySQL Router for load
balancing purposes
19. Semi-synchronous replication
• semi-sync capable slave acknowledges transaction event only after
written to relay log & flushed to disk
• timeout occurs? master reverts to async replication; resumes when
slaves catch up
• at scale, Facebook runs semi-sync: http://
yoshinorimatsunobu.blogspot.com/2014/04/semi-synchronous-
replication-at-facebook.html
20. MySQL Replication in 5.6
• Global Transaction ID (GTID)
• Server UUID
• Ignore (master) server IDs (filtering)
• Per-schema multi-threaded slave
• Group commit in the binary log
• Binary log (binlog) checksums
• Crash safe binlog and relay logs
• Time delayed replication
• Parallel replication (per database)
21. Replication: START TRANSACTION WITH
CONSISTENT SNAPSHOT
• Works with the binlog, possible to obtain the binlog position corresponding to
a transactional snapshot of the database without blocking any other queries.
• by-product of group commit in the binlog to view commit ordering
• Used by the command mysqldump--single-transaction --master-
data to do a fully non-blocking backup
• Works consistently between transactions involving more than one storage
engine
• https://kb.askmonty.org/en/enhancements-for-start-transaction-with-
consistent/
• Percona Server improved it, by session ID, and also introducing backup locks
22. Multi-source replication
• Multi-source replication - (real-time) analytics, shard provisioning,
backups, etc.
• @@default_master_connection contains current connection name
(used if connection name is not given)
• All master/slave commands take a connection name now (like CHANGE
MASTER “connection_name”, SHOW SLAVE “connection_name”
STATUS, etc.)
23. Global Transaction ID (GTID)
• Supports multi-source replication
• GTID can be enabled or disabled independently and online for masters or
slaves
• Slaves using GTID do not have to have binary logging enabled.
• (MariaDB) Supports multiple replication domains (independent binlog streams)
• Queries in different domains can be run in parallel on the slave.
24. Why MariaDB GTID is different compared to
5.6?
• MySQL 5.6 GTID does not support multi-source replication (only 5.7
supports this)
• Supports —log-slave-updates=0 for efficiency (like 5.7)
• Enabled by default
• Turn it on without having to restart the topology (just like 5.7)
25. Parallel replication
• Multi-source replication from different masters executed in parallel
• Queries from different domains are executed in parallel
• Queries that are run in parallel on the master are run in parallel on the
slave (based on group commit).
• Transactions modifying the same table can be updated in parallel on
the slave!
• Supports both statement based and row based replication.
26. All in… sometimes it can get out of sync
• Changed information on slave directly
• Statement based replication
• non-deterministic SQL (UPDATE/DELETE with LIMIT and without ORDER BY)
• triggers & stored procedures
• Master in MyISAM, slave in InnoDB (deadlocks)
• --replication-ignore-db with fully qualified queries
• Binlog corruption on master
• PURGE BINARY LOGS issued and not enough files to update slave
• read_buffer_size larger than max_allowed_packet
• Bugs?
27. Replication Monitoring
• Percona Toolkit is important
• pt-slave-find: find slave information from master
• pt-table-checksum: online replication consistency check
• executes checksum queries on master
• pt-table-sync: synchronise table data efficiently
• changes data, so backups important
29. mysqlbinlog versions
• ERROR: Error in Log_event::read_log_event(): 'Found invalid event in
binary log', data_len: 56, event_type: 30
• 5.6 ships with a “streaming binlog backup server” - v.3.4; MariaDB
10 doesn’t - v.3.3 (fixed in 10.2 - MDEV-8713)
• GTID variances!
• Beware mysql-client from your Linux distribution
30. Slave prefetching
• Replication Booster
• https://github.com/yoshinorim/replication-booster-for-mysql
• Prefetch MySQL relay logs to make the SQL thread faster
• Tungsten has slave prefetch
• Percona Server till 5.6 + MariaDB till 10.1 have InnoDB fake changes
31. Changing paradigm: What replaces slave
prefetching?
• In Percona Server 5.7, slave prefetching has been replaced by doing
intra-schema parallel replication
• Feature removed from XtraDB
• MariaDB Server 10.2 also has this feature removed, as they switched
to InnoDB!
32. Galera Cluster
• Inside MySQL, a replication plugin (wsrep)
• Replaces MySQL replication (but can work alongside it too)
• True multi-master, active-active solution
• Virtually Synchronous
• WAN performance: 100-300ms/commit, works in parallel
• No slave lag or integrity issues
• Automatic node provisioning
34. Percona XtraDB Cluster 5.7
• Engineering within Percona
• Load balancing with ProxySQL (bundled)
• PMM integration
• Benefits of all the MySQL 5.7 feature-set
• ProxySQL admin tool
• Safety features enabled (e.g. no accidentally using MyISAM, etc.)
35. Group replication
• Fully synchronous replication (update everywhere), self-healing, with
elasticity, redundancy
• Single primary mode supported
• MySQL InnoDB Cluster - a combination of group replication, Router, to
make magic!
• Recent blogs:
• https://www.percona.com/blog/2017/02/24/battle-for-synchronous-
replication-in-mysql-galera-vs-group-replication/
• https://www.percona.com/blog/2017/02/15/group-replication-shipped-
early/
36. Summary of Replication Performance
• SAN has "some" latency overhead compared to local disk. Can be
great for throughput.
• DRBD has a performance penalty
• Replication, when implemented correctly, has no performance penalty
• But MySQL replication with disk bound data set has single-threaded
issues!
• Semi-sync is poorer on WAN compared to async
• Galera & InnoDB Cluster provide read/write scale-out, thus more
performance
37. Handling failure
• How do we find out about failure?
• Polling, monitoring, alerts...
• Error returned to and handled in client side
• What should we do about it?
• Direct requests to the spare nodes (or DCs)
• How to protect data integrity?
• Master-slave is unidirectional: Must ensure there is only one master at all times.
• DRBD and SAN have cold-standby: Must mount disks and start mysqld.
• In all cases must ensure that 2 disconnected replicas cannot both commit
independently. (split brain)
40. MySQL MHA
• Like MMM, specialized solution for MySQL replication
• Developed by Yoshinori Matsunobu at DeNA
• Automated and manual failover options
• Topology: 1 master, many slaves
• Choose new master by comparing slave binlog positions
• Can be used in conjunction with other solutions
• http://code.google.com/p/mysql-master-ha/
43. MySQL Router
• Routing between applications and any backend MySQL servers
• Failover
• Load Balancing
• Pluggable architecture (connection routing)
44. MariaDB MaxScale
• “Pluggable router” that offers
connection & statement based
load balancing
• Possibilities are endless - use it
for logging, writing to other
databases (besides MySQL),
preventing SQL injections via
regex filtering, route via hints,
query rewriting, have a binlog
relay, etc.
45. ProxySQL
• High Performance MySQL proxy with a GPL license
• Performance is a priority - the numbers prove it
• Can query rewrite
• Sharding by host/schema or both, with rule engine + modification to
SQL + application logic
46. JDBC/PHP drivers
• JDBC - multi-host failover feature (just specify master/slave hosts in
the properties)
• true for MariaDB Java Connector too
• PHP handles this too - mysqlnd_ms
• Can handle read-write splitting, round robin or random host
selection, and more
47. Clustering: solution or part of problem?
• "Causes of Downtime in Production MySQL Servers" whitepaper,
Baron Schwartz VividCortex
• Human error
• SAN
• Clustering framework + SAN = more problems
• Galera/group replication is replication based, has no false positives
as there’s no “failover” moment, you don’t need a clustering
framework (JDBC or PHP can load balance), and is relatively elegant
overall
48. Replication type
• Competence choices
• Replication: MySQL DBA manages
• DRBD: Linux admin manages
• SAN: requires domain controller
• Operations
• DRBD (disk level) = cold standby =
longer failover
• Replication = hot standby =
shorter failover
• GTID helps tremendously
• Performance
• SAN has higher latency than local
disk
• DRBD has higher latency than
local disk
• Replication has little overhead
• Redundancy
• Shared disk = SPoF
• Shared nothing = redundant
49. SBR vs RBR? Async vs sync?
• row based: deterministic
• statement based: dangerous
• GTID: easier setup & failover of complex topologies
• async: data loss in failover
• sync: best
• multi-threaded slaves: scalability (hello 5.6+, Tungsten)
50. What about the cloud?
• Usually scalability & high availability is more or less “built-in”
• e.g. RDS has multi-AZ (synchronous data replication), but doesn’t
give you a read replica; Cloud SQL uses semi-sync replication
• Watch out for the SLAs (and automatic upgrades)
• Monitoring via PERFORMANCE_SCHEMA
• “Bad” nodes do exist; do not assume node provisioning is quick
51. Conclusion
• MySQL replication is amazing if you know it (and monitor it) well
enough
• Large sites run just fine with semi-sync + tooling for automated
failover
• Galera Cluster/MySQL InnoDB Cluster is great for virtually fully
synchronous replication
• Don’t forget the need for a load balancer: ProxySQL is nifty
• When thinking scaling, think scale out (it is more efficient, and fits
modern cloud mantras too!)
52. At Percona, we care about your High
Availability
• Percona XtraDB Cluster 5.7 with support for ProxySQL and Percona
Monitoring & Management (PMM)
• Percona Monitoring & Management (PMM) with Orchestrator
• Percona Toolkit
• Percona Server for MySQL 5.7
• Percona XtraBackup