SlideShare a Scribd company logo
MySQL HA
with PaceMaker
   Kris Buytaert
Kris Buytaert
●   CTO and Open Source Consultant @inuits.eu
●   „Infrastructure Architect“
●   I don't remember when I started using MySQL
●   Specializing in Automated , Large Scale
    Deployments , Highly Available infrastructures,
    since 2008 also known as “the Cloud”
                    th
●   Surviving the 10 floor test
●   Cofounded devopsdays.org
In this presentation
●   High Availability ?
●   MySQL HA Solutions
●   MySQL Replication
●   Linux HA / Pacemaker
What is HA Clustering ?

●   One service goes down
=> others take over its work
●   IP address takeover, service takeover,
●   Not designed for high-performance
●   Not designed for high troughput (load
    balancing)

Recommended for you

Introducing Galera 3.0
Introducing Galera 3.0Introducing Galera 3.0
Introducing Galera 3.0

From these slides you'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks. ABOUT GALERA CLUSTER Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release: Galera Cluster features and benefits Support for MySQL 5.6 Integration with MySQL Global Transaction Identifiers Mixing Galera synchronous replication and asynchronous MySQL replication Deploying in WAN and Cloud environments Handling high-latency networks Management of Galera

mysqlhigh availabilitygalera
Maria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High AvailabilityMaria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High Availability

Want to understand how to set high availability solutions for MySQL using MariaDB Galera Cluster? Join this webinar, and learn from experts. During this webinar, you will also get guidance on how to implement MariaDB Galera Cluster.

mariadbdatabasecluster
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6

You'll learn how Galera integrates with MySQL 5.6 and Global Transaction IDs to enable cross-datacenter and cloud replication over high latency networks. The benefits are clear; a globally distributed MySQL setup across regions to deliver Severalnines availability and real-time responsiveness. Galera Cluster for MySQL is a true multi-master MySQL replication plugin, and has been proven in mission-critical infrastructures of companies like Ping Identity, AVG Technologies, KPN and HP Cloud DNS. In this webcast you¹ll learn about the following Galera Cluster capabilities, including the latest innovations in the new 3.0 release: Galera Cluster features and benefits Support for MySQL 5.6 Integration with MySQL Global Transaction Identifiers Mixing Galera synchronous replication and asynchronous MySQL replication Deploying in WAN and Cloud environments Handling high-latency networks Management of Galera

database administratormysqldatabases
Does it Matter ?

●   Downtime is expensive
●   You mis out on $$$
●   Your boss complains
●   New users don't return
Lies, Damn Lies, and
Statistics
         Counting nines
            (slide by Alan R)




 99.9999%                        30 sec
 99.999%                          5 min
 99.99%                          52 min
 99.9%                            9 hr
 99%                            3.5 day
The Rules of HA

●   Keep it Simple
●   Keep it Simple
●   Prepare for Failure
●   Complexity is the enemy of reliability
●   Test your HA setup
You care about ?
●   Your data ?
•Consistent
•Realitime
•Eventual Consistent
●   Your Connection
•Always
•Most of the time

Recommended for you

DIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterDIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL Cluster

Live from the International PHP Conference 2013: MySQL Cluster is a distributed, auto-sharding database offering 99,999% high availability. It runs on Rasperry PI as well as on a cluster of multi-core machines. A 30 node cluster was able to deliver 4.3 billion (not million) read transactions per second in 2012. Take a deeper look into the theory behind all the MySQL replication/clustering solutions (including 3rd party) and learn how they differ.

virtual synchronymysqlgalera
Galera explained 3
Galera explained 3Galera explained 3
Galera explained 3

This presentation is aim to give an initial understanding of how MySQL/Galera works, and some advice.

galeramysqlcluster
Highly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlndHighly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlnd

This document discusses how to achieve high availability in PHP/MySQL applications using the mysqlnd driver. It describes different MySQL high availability configurations including master-slave replication, multi-master replication using Galera or NDB Cluster, and how mysqlnd's mysqlnd_ms plugin allows applications to connect to these clustered MySQL instances in a highly available manner by handling failover between nodes. The document provides examples of mysqlnd_ms connection configuration for both master-slave and multi-master setups.

mysqlnd_mshigh availabilitymysqlnd
Eliminating the SPOF
●   Find out what Will Fail
•Disks
•Fans
•Power (Supplies)
●   Find out what Can Fail
•Network
•Going Out Of Memory
Split Brain
●   Communications failures can lead to separated
    partitions of the cluster
●   If those partitions each try and take control of
    the cluster, then it's called a split-brain
    condition
●   If this happens, then bad things will happen
•http://linux-ha.org/BadThingsWillHappen
Historical MySQL HA
●   Replication
•1 read write node
•Multiple read only nodes
•Application needed to be modified
Solutions Today
●   BYO
●   DRBD
●   MySQL Cluster NDBD
●   Multi Master Replication
●   MySQL Proxy
●   MMM / Flipper
●   Galera
●   Percona XtraDB Cluster

Recommended for you

Taking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master ClusterTaking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master Cluster

we will discuss important topics related to multi-master setups: * Practical considerations when using Galera in a multi-master setup * Evaluating the characteristics of your database workload * Preparing your application for multi-master * Detecting and dealing with transaction conflicts

multi-master clustermysql high availabilitymysql scalability
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBD

The document provides a history of high availability clustering technologies, including Heartbeat, OpenAIS/Corosync, Pacemaker, and DRBD. It discusses how these projects have evolved over time, with Heartbeat focusing only on messaging, OpenAIS/Corosync providing improved scalability, Pacemaker controlling resources and supporting multiple stacks, and DRBD enabling shared storage across nodes. It also outlines future directions like Pacemaker Cloud for cloud deployments and stretch clusters for multi-site scalability.

high availabilitycorosyncdrbd
Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)

Using Galera Cluster for MySQL in different architectures in LAN, WAN, load balancers and other solutions. Comes with several benchmarks.

mysqlhigh availabilitygalera
Data vs Connection
●   DATA :
•Replication
•DRBD
●   Connection
•LVS
•Proxy
•Heartbeat / Pacemaker
Shared Storage
●   1 MySQL instance
●   Monitor MySQL node
●   Stonith
●   $$$              1+1 <> 2
●   Storage = SPOF
●   Split Brain :(
DRBD
●   Distributed Replicated Block Device
●   In the Linux Kernel (as of very recent)
●   Usually only 1 mount
•Multi mount as of 8.X
•Requires GFS / OCFS2
●   Regular FS ext3 ...
●   Only 1 MySQL instance Active accessing data
●   Upon Failover MySQL needs to be started on
    other node
DRBD(2)
●   What happens when you pull the plug of a
    Physical machine ?
•Minimal Timeout
•Why did the crash happen ?
•Is my data still correct ?
•Innodb Consistency Checks ?
•Lengthy ?
•Check your BinLog size

Recommended for you

Galera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slidesGalera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slides

Slides for the webinar held on January 21st 2014 Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl. In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals. Agenda: Redundancy models for Galera, NDB and MongoDB/TokuMX Failover & Recovery (Automatic vs Manual) Zooming into Galera recovery procedures Split brains in multi-datacenter setups

mongodbmysqlmysql cluster nosql
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides

Galera is a MySQL replication technology that can simplify the design of a high availability application stack. With a true multi-master MySQL setup, an application can now read and write from any database instance without worrying about master/slave roles, data integrity, slave lag or other drawbacks of asynchronous replication. And that all sounds great until it’s time to go into production. Throw in a live migration from an existing database setup and devops life just got a bit more interesting ... So if you are in devops, then this webinar is for you! Operations is not so much about specific technologies, but about the techniques and tools you use to deploy and manage them. Monitoring, managing schema changes and pushing them in production, performance optimizations, configurations, version upgrades, backups; these are all aspects to consider – preferably before going live. Let us guide you through 9 key tips to consider before taking Galera Cluster into production.

mysql
Using galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wanUsing galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wan

We will show the advantages of having a geo-distributed database cluster and how to create one using Galera Cluster for MySQL. We will also discuss the configuration and status variables that are involved and how to deal with typical situations on the WAN such as slow, untrusted or unreliable links, latency and packet loss. We will demonstrate a multi-region cluster on Amazon EC2 and perform some throughput and latency measurements in real-time (video http://galeracluster.com/videos/using-galera-replication-to-create-geo-distributed-clusters-on-the-wan-webinar-video-3/)

wan clustergalera cloud clustergalera cluster
MySQL Cluster NDBD
●   Shared-nothing architecture
●   Automatic partitioning
●   Synchronous replication
●   Fast automatic fail-over of data nodes
●   In-memory indexes
●   Not suitable for all query patterns (multi-table
    JOINs, range scans)
Title
– Data
MySQL Cluster NDBD
●   All indexed data needs to be in memory
●   Good and bad experiences
•Better experiences when using the API
•Bad when using the MySQL Server
●   Test before you deploy
●   Does not fit for all apps
How replication works
● Master server keeps track of all updates in the
  Binary Log
•Slave requests to read the binary update log
•Master acts in a passive role, not keeping track
of what slave has read what data

●  Upon connecting the slaves do the following:
•The slave informs the master of where it left off
•It catches up on the updates
•It waits for the master to notify it of new
updates

Recommended for you

Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013

Codership provides high availability, no-data-loss, and scalable data replication and clustering solutions for open source databases to securely store customers' valuable data. They do this through solutions like Galera replication which allows for synchronous multi-master replication across MariaDB and MySQL database clusters.

mariadbgalera clustermysql
MySQL 5.6 Performance
MySQL 5.6 PerformanceMySQL 5.6 Performance
MySQL 5.6 Performance

This document provides an overview and agenda for a presentation on MySQL 5.6 performance tuning and best practices. The presentation covers analyzing MySQL workload and internals, performance improvements in MySQL 5.6 and 5.7, benchmark results, and pending issues. It emphasizes the importance of monitoring systems to understand performance bottlenecks and the need for an iterative process of monitoring, tuning, optimizing, and improving database performance over time.

mysql 5.6mysqlperformance
Performance Tuning Best Practices
Performance Tuning Best PracticesPerformance Tuning Best Practices
Performance Tuning Best Practices

The document provides best practices for performance tuning MySQL databases. It discusses benchmarking and profiling concepts, sources of performance problems like inefficient schemas and indexes, and SQL coding practices. It also recommends tuning server parameters and provides tools for benchmarking, profiling, and optimizing performance.

mysql
Buytaert kris my_sql-pacemaker
Two Slave Threads
● How does it work?
•The I/O thread connects to the master and asks for
the updates in the master’s binary log
•The I/O thread copies the statements to the relay
log
•The SQL thread implements the statements in the
relay log
Advantages
•Long running SQL statements don’t block log
downloading
•Allows the slave to keep up with the master better
•In case of master crash the slave is more likely to
have all statements
Replication commands
Slave commands
● START|STOP SLAVE

● RESET SLAVE

● SHOW SLAVE STATUS

● CHANGE MASTER TO…

● LOAD DATA FROM MASTER

● LOAD TABLE tblname FROM MASTER



Master commands
● SHOW MASTER STATUS

● PURGE MASTER LOGS…
Show slave statusG
  Slave_IO_State: Waiting for master to send event
             Master_Host: 172.16.0.1
             Master_User: repli
             Master_Port: 3306
           Connect_Retry: 60
         Master_Log_File: XMS-1-bin.000014
      Read_Master_Log_Pos: 106
          Relay_Log_File: XMS-2-relay.000033
           Relay_Log_Pos: 251
     Relay_Master_Log_File: XMS-1-bin.000014
         Slave_IO_Running: Yes
        Slave_SQL_Running: Yes
         Replicate_Do_DB: xpol
      Replicate_Ignore_DB:
       Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
 Replicate_Wild_Ignore_Table:
             Last_Errno: 0
             Last_Error:
            Skip_Counter: 0
      Exec_Master_Log_Pos: 106
         Relay_Log_Space: 547
         Until_Condition: None
          Until_Log_File:
           Until_Log_Pos: 0
       Master_SSL_Allowed: No
       Master_SSL_CA_File:
       Master_SSL_CA_Path:
         Master_SSL_Cert:
        Master_SSL_Cipher:
          Master_SSL_Key:
     Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
           Last_IO_Errno: 0
           Last_IO_Error:
          Last_SQL_Errno: 0
          Last_SQL_Error:
1 row in set (0.00 sec)

Recommended for you

Introduction to Galera
Introduction to GaleraIntroduction to Galera
Introduction to Galera

Introduction to Galera clustering with MySQL. Installation, configuration, startup, internal architecture, replication topologies, load balancers, benchmarks.

mysqlhigh availabilitygalera
Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016

In this webinar, we will describe how to get started with Galera Cluster and build a functional multi-master cluster. First, will show how to easily install the required packages using the new preferred installation method – the dedicated Galera package repository. Then we will discuss the important Galera configuration settings and how to select values for them. Finally, we will demonstrate how to bootstrap a 3-node Galera installation with the right sequence of steps. Once the nodes are up and running we will discuss how to monitor the health of the cluster and which status variables are important to watch. Galera Cluster is trusted by thousands of users. Galera Cluster powers Percona XtraDB Cluster and MariaDB Enterprise Cluster. This is a webinar presented by Codership, the developers and experts of Galera Cluster.

mysql clusteringpercona xtradb clustermysql cluster
максим бабич
максим бабичмаксим бабич
максим бабич
Row vs Statement
●   Pro                              ●  Pro
                                     •All changes can be replicated
•Proven (around since MySQL 3.23)
                                     •Similar technology used by other
•Smaller log files                   RDBMSes
                                     •Fewer locks required for some
•Auditing of actual SQL statements   INSERT, UPDATE or DELETE
                                     statements
•No primary key requirement for      ● Con
replicated tables                    •More data to be logged
●   Con                              •Log file size increases
                                     (backup/restore implications)
•Non-deterministic functions and     •Replicated tables require explicit
UDFs                                 primary keys
                                     •Possible different result sets on
                                     bulk INSERTs
Multi Master Replication
●   Replicating the same table data both ways can
    lead to race conditions
•Auto_increment, unique keys, etc.. could cause
problems If you write them 2x
●   Both nodes are master
●   Both nodes are slave
●   Write in 1 get updates on the other

                                    M|S       M|S
MySQL Proxy
●   Man in the middle
●   Decides where to connect to
•LUA
●   Write rules to
•Redirect traffic
•
Master Slave & Proxy
●   Split Read and Write Actions
●   No Application change required
●   Sends specific queries to a specific node
●   Based on
•Customer
•User
•Table
•Availability

Recommended for you

сергей спиридонов
сергей спиридоновсергей спиридонов
сергей спиридонов
чашкин иван
чашкин иванчашкин иван
чашкин иван
дыдыкин егор
дыдыкин егордыдыкин егор
дыдыкин егор

The document discusses touch and gesture handling in modern web applications. It covers touch events like touchstart, touchmove, touchend and their mouse event equivalents. It also describes how to implement swipe/scroll gestures, zooming functionality using touch inputs, and pull to refresh features using touchstart and touchmove events. Key aspects covered include detecting swipe direction, translating pages on swipe, handling touch and click events for links and taps, and transforming elements on zoom gestures.

MySQL Proxy
●   Your new SPOF
●   Make your Proxy HA too !
•Heartbeat OCF Resource
Breaking Replication
●   If the master and slave gets out of sync
●   Updates on slave with identical index id
•Check error log for disconnections and issues
with replication
Monitor your Setup
●   Not just connectivity
●   Also functional
•Query data
•Check resultset is correct
●   Check replication
•MaatKit
•OpenARK
Pulling Traffic
●   Eg. for Cluster, MultiMaster setups
•DNS
•Advanced Routing
•LVS


•Flipper / MMM

Recommended for you

сумин андрей
сумин андрейсумин андрей
сумин андрей
Jouravski kickstart1
Jouravski kickstart1Jouravski kickstart1
Jouravski kickstart1
MMM
●   Multi-Master Replication Manager
    for MySQL

•Perl scripts to perform
monitoring/failover and
management of MySQL master-
master replication configurations
●   Balance master / slave configs
    based on replication state

•Map Virtual IP to the Best Node
●   http://mysql-mmm.org/
Flipper
●   Flipper is a Perl tool for
    managing read and write
    access pairs of MySQL servers
●   master-master MySQL Servers
●   Clients machines do not
    connect "directly" to either
    node instead,
●   One IP for read,
●   One IP for write.
●   Flipper allows you to move
    these IP addresses between
    the nodes in a safe and
    controlled manner.
●   http://provenscaling.com/softw
    are/flipper/
Linux-HA PaceMaker
●   Plays well with others
●   Manages more than MySQL
●

●   ...v3 .. don't even think about the rest anymore
●

●   http://clusterlabs.org/
Heartbeat
●   Heartbeat v1
•Max 2 nodes
•No finegrained resources
•Monitoring using “mon”
●   Heartbeat v2
•XML usage was a consulting opportunity
•Stability issues
•Forking ?

Recommended for you

ярослав рабоволюк
ярослав рабоволюкярослав рабоволюк
ярослав рабоволюк
Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker

Linux High Availability provides information on configuring high availability clusters in Linux. It discusses: - Key components of HA clustering including one service taking over work of another if it fails, IP and service takeover. - The importance of high availability and costs of downtime for businesses. Statistics show significant downtime even at 99.9% availability levels. - Best practices for HA including keeping configurations simple, preparing for failures, and testing HA setups. Complexity increases reliability risks.

linux-hapacemakerukuug
Pacemaker Architecture
            ●   Stonithd : The Heartbeat fencing subsystem.
            ●   Lrmd : Local Resource Management Daemon.
                Interacts directly with resource agents (scripts).
            ●   pengine Policy Engine. Computes the next state of the
                cluster based on the current state and the configuration.
            ●   cib Cluster Information Base. Contains definitions of all
                cluster options, nodes, resources, their relationships to
                one another and current status. Synchronizes updates to
                all cluster nodes.
            ●   crmd Cluster Resource Management Daemon. Largely
                a message broker for the PEngine and LRM, it also
                elects a leader to co-ordinate the activities of the cluster.
            ●   openais messaging and membership layer.
            ●   heartbeat messaging layer, an alternative to OpenAIS.
            ●   ccm Short for Consensus Cluster Membership. The
                Heartbeat membership layer.
Pacemaker ?
●   Not a fork
●   Only CRM Code taken out of Heartbeat
●   As of Heartbeat 2.1.3
•Support for both OpenAIS / HeartBeat
•Different Release Cycles as Heartbeat
Heartbeat, OpenAis ?
●   Both Messaging Layers
●   Initially only Heartbeat
●   OpenAIS
●   Heartbeat got unmaintained
●   OpenAIS has heisenbugs :(
●   Heartbeat maintenance taken over by LinBit
●   CRM Detects which layer
Pacemaker




Heartbeat       or         OpenAIS




            Cluster Glue

Recommended for you

Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker

Linux High Availability provides concise summaries of key concepts: - High availability (HA) clustering allows services to take over work from others that go down, through IP and service takeover. It is designed for uptime, not performance or load balancing. - Downtime is expensive for businesses due to lost revenue and customer dissatisfaction. Statistics show significant drops in availability even at 99.9% uprates. - To achieve high availability, systems must be designed with simplicity, failure preparation, and reliability testing in mind. Complexity often undermines reliability. - Myths exist around technologies like virtualization and live migration providing complete high availability solutions. True HA requires eliminating all single points of

linux-haload11pacemaker
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions

A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011. A video recording of this presentation is available from Ustream: http://ustre.am/fyLk

highavailabilitymysqlha
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions

This document discusses various high availability solutions for MySQL databases. It begins with an overview of high availability concepts and considerations. It then summarizes MySQL replication, disk replication using DRBD, shared storage, and MySQL Cluster. Other high availability tools mentioned include Pacemaker, Galera replication, MMM, Tungsten Replicator, Red Hat Cluster Suite, Solaris Cluster, and Flipper. The document provides information on how these different techniques can be used to add redundancy and eliminate single points of failure for MySQL databases.

opensourcehighavailabilitymysql
Configuring Heartbeat
●   /etc/ha.d/ha.cf
Use crm = yes


●   /etc/ha.d/authkeys
Configuring Heartbeat
heartbeat::hacf {"clustername":

         hosts => ["host-a","host-b"],

         hb_nic => ["bond0"],

         hostip1 => ["10.0.128.11"],

         hostip2 => ["10.0.128.12"],

         ping => ["10.0.128.4"],

    }

heartbeat::authkeys {"ClusterName":

         password => “ClusterName ",

    }

http://github.com/jtimberman/puppet/tree/master/heartbeat/
Heartbeat Resources
●   LSB
●   Heartbeat resource (+status)
●   OCF (Open Cluster FrameWork) (+monitor)
●   Clones (don't use in HAv2)
●   Multi State Resources
A MySQL Resource
●   OCF
•Clone
•Where do you hook up the IP ?
•Multi State
•But we have Master Master replication
•Meta Resource
•Dummy resource that can monitor
•Connection
•Replication state

Recommended for you

Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02

This document discusses various high availability solutions for MySQL databases. It begins with an overview of high availability concepts and considerations. It then covers MySQL replication, disk replication using DRBD, MySQL Cluster, and other tools like Pacemaker, Galera replication, MMM, Tungsten Replicator, Red Hat Cluster Suite, Solaris Cluster, and Flipper. The document provides details on how each solution works and its advantages and disadvantages for providing redundancy and high availability for MySQL databases.

MySQL HA with Pacemaker
MySQL HA with  PacemakerMySQL HA with  Pacemaker
MySQL HA with Pacemaker

- MySQL HA can be achieved with solutions like shared storage (DRBD), replication, MySQL Cluster, or Linux HA/Pacemaker. - Linux HA/Pacemaker provides high availability by managing resources across nodes and ensuring that services are running on an available node if one fails. - It uses a central configuration (CIB) to define resources, constraints between them, and monitor their status to determine the optimal placement of resources across nodes.

opendbcampmysqlopensqlcamp
MySQL HA with PaceMaker
MySQL HA with  PaceMakerMySQL HA with  PaceMaker
MySQL HA with PaceMaker

Pacemaker is a high availability cluster resource manager that can be used to provide high availability for MySQL databases. It monitors MySQL instances and replicates data between nodes using replication. If the primary MySQL node fails, Pacemaker detects the failure and fails over to the secondary node, bringing the MySQL service back online without downtime. Pacemaker manages shared storage and virtual IP failover to ensure connections are direct to the active MySQL node. It is important to monitor replication state and lag to ensure data consistency between nodes.

clusterlabsmysqlha
CRM
                          configure
●   Cluster Resource      property $id="cib-bootstrap-
                          options" 
    Manager                   stonith-enabled="FALSE" 
                              no-quorum-policy=ignore 
●   Keeps Nodes in Sync       start-failure-is-fatal="FALSE" 
                          rsc_defaults $id="rsc_defaults-
                          options" 
●   XML Based                 migration-threshold="1" 
                              failure-timeout="1"
●   cibadm                primitive d_mysql ocf:local:mysql 
                              op monitor interval="30s" 
●   Cli manageable            params test_user="sure"
                          test_passwd="illtell"
                          test_table="test.table"
●   Crm                   primitive ip_db
                          ocf:heartbeat:IPaddr2 
                              params ip="172.17.4.202"
                          nic="bond0" 
                              op monitor interval="10s"
                          group svc_db d_mysql ip_db
                          commit
Adding MySQL to the
stack

                     Replication
  Service IP MySQL

  “MySQLd”                          “MySQLd”   Resource MySQL

                                                Cluster Stack
                      Pacemaker

                      HeartBeat
         Node A                    Node B      Hardware
Pitfalls & Solutions
●   Monitor,
•Replication state
•Replication Lag


●   MaatKit
●   OpenARK
Conclusion
●   Plenty of Alternatives
●   Think about your Data
●   Think about getting Queries to that Data
●   Complexity is the enemy of reliability
●   Keep it Simple
●   Monitor inside the DB

Recommended for you

Loadays MySQL
Loadays MySQLLoadays MySQL
Loadays MySQL

This document discusses strategies for maintaining very large MySQL tables that have grown too big. It recommends creating a new database server with different configuration settings like InnoDB file per table to reduce size, using tools like MySQLTuner and tuning-primer to analyze settings, archiving old historical data with ptArchiver to reduce table sizes, and considering partitioning or changing the MySQL version. Monitoring tools like InnoDB status, global status, cacti and innotop are recommended to analyze server performance.

mysql load
MariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale:  an Intelligent Database ProxyMariaDB MaxScale:  an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database Proxy

MariaDB MaxScale is a database proxy that abstracts database clusters to simplify application development and management. It isolates complexity, provides high availability, and improves performance. MaxScale classifies queries, monitors cluster nodes, routes reads and writes, and hides failures to present a single logical database interface. Key components include intelligent routing, query classification, cluster monitoring, and extensible filtering. It provides a unified way to handle different cluster types and seamlessly handle failures and topology changes.

maxscalemariadbdatabase
UKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL TuningUKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL Tuning

This document provides a summary of a presentation on practical MySQL tuning. It discusses measuring critical system resources like CPU, memory, I/O and network usage to identify bottlenecks. It also covers rough tuning of MySQL parameters like the InnoDB buffer pool size, log file size and key buffer size. Further tuning includes application optimizations like query tuning with EXPLAIN, index tuning, and schema design. The presentation also discusses scaling MySQL through approaches like caching, sharding, replication and optimizing architecture and data distribution. Regular performance monitoring is emphasized to simulate increased load and aid capacity planning.

tuningukougmysql
Contact
Kris Buytaert
Kris.Buytaert@inuits.be

Further Reading
@krisbuytaert
http://www.krisbuytaert.be/blog/
http://www.inuits.be/
                      •Or the upcoming slides


                             Inuits
                             't Hemeltje
                             Duboistraat 50
                             2060 Antwerpen
                             Belgium
                             891.514.231

                             +32 475 961221

More Related Content

What's hot

How to understand Galera Cluster - 2013
How to understand Galera Cluster - 2013How to understand Galera Cluster - 2013
How to understand Galera Cluster - 2013
Codership Oy - Creators of Galera Cluster
 
MySQL Replication Alternative: Pros and Cons
MySQL Replication Alternative: Pros and ConsMySQL Replication Alternative: Pros and Cons
MySQL Replication Alternative: Pros and Cons
Darpan Dinker
 
MariaDB Galera Cluster - Simple, Transparent, Highly Available
MariaDB Galera Cluster - Simple, Transparent, Highly AvailableMariaDB Galera Cluster - Simple, Transparent, Highly Available
MariaDB Galera Cluster - Simple, Transparent, Highly Available
MariaDB Corporation
 
Introducing Galera 3.0
Introducing Galera 3.0Introducing Galera 3.0
Maria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High AvailabilityMaria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High Availability
OSSCube
 
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Severalnines
 
DIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterDIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL Cluster
Ulf Wendel
 
Galera explained 3
Galera explained 3Galera explained 3
Galera explained 3
Marco Tusa
 
Highly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlndHighly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlnd
Jervin Real
 
Taking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master ClusterTaking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master Cluster
Codership Oy - Creators of Galera Cluster
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBD
Dan Frincu
 
Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)
Henrik Ingo
 
Galera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slidesGalera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slides
Severalnines
 
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
Severalnines
 
Using galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wanUsing galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wan
Codership Oy - Creators of Galera Cluster
 
Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013
MariaDB Corporation
 
MySQL 5.6 Performance
MySQL 5.6 PerformanceMySQL 5.6 Performance
MySQL 5.6 Performance
MYXPLAIN
 
Performance Tuning Best Practices
Performance Tuning Best PracticesPerformance Tuning Best Practices
Performance Tuning Best Practices
webhostingguy
 
Introduction to Galera
Introduction to GaleraIntroduction to Galera
Introduction to Galera
Henrik Ingo
 
Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016
Sakari Keskitalo
 

What's hot (20)

How to understand Galera Cluster - 2013
How to understand Galera Cluster - 2013How to understand Galera Cluster - 2013
How to understand Galera Cluster - 2013
 
MySQL Replication Alternative: Pros and Cons
MySQL Replication Alternative: Pros and ConsMySQL Replication Alternative: Pros and Cons
MySQL Replication Alternative: Pros and Cons
 
MariaDB Galera Cluster - Simple, Transparent, Highly Available
MariaDB Galera Cluster - Simple, Transparent, Highly AvailableMariaDB Galera Cluster - Simple, Transparent, Highly Available
MariaDB Galera Cluster - Simple, Transparent, Highly Available
 
Introducing Galera 3.0
Introducing Galera 3.0Introducing Galera 3.0
Introducing Galera 3.0
 
Maria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High AvailabilityMaria DB Galera Cluster for High Availability
Maria DB Galera Cluster for High Availability
 
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
Webinar slides: Introducing Galera 3.0 - Now supporting MySQL 5.6
 
DIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL ClusterDIY: A distributed database cluster, or: MySQL Cluster
DIY: A distributed database cluster, or: MySQL Cluster
 
Galera explained 3
Galera explained 3Galera explained 3
Galera explained 3
 
Highly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlndHighly Available MySQL/PHP Applications with mysqlnd
Highly Available MySQL/PHP Applications with mysqlnd
 
Taking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master ClusterTaking Full Advantage of Galera Multi Master Cluster
Taking Full Advantage of Galera Multi Master Cluster
 
Pacemaker+DRBD
Pacemaker+DRBDPacemaker+DRBD
Pacemaker+DRBD
 
Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)Using and Benchmarking Galera in different architectures (PLUK 2012)
Using and Benchmarking Galera in different architectures (PLUK 2012)
 
Galera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slidesGalera Cluster - Node Recovery - Webinar slides
Galera Cluster - Node Recovery - Webinar slides
 
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
9 DevOps Tips for Going in Production with Galera Cluster for MySQL - Slides
 
Using galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wanUsing galera replication to create geo distributed clusters on the wan
Using galera replication to create geo distributed clusters on the wan
 
Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013Galera cluster - SkySQL Paris Meetup 17.12.2013
Galera cluster - SkySQL Paris Meetup 17.12.2013
 
MySQL 5.6 Performance
MySQL 5.6 PerformanceMySQL 5.6 Performance
MySQL 5.6 Performance
 
Performance Tuning Best Practices
Performance Tuning Best PracticesPerformance Tuning Best Practices
Performance Tuning Best Practices
 
Introduction to Galera
Introduction to GaleraIntroduction to Galera
Introduction to Galera
 
Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016Codership's galera cluster installation and quickstart webinar march 2016
Codership's galera cluster installation and quickstart webinar march 2016
 

Viewers also liked

максим бабич
максим бабичмаксим бабич
максим бабич
kuchinskaya
 
сергей спиридонов
сергей спиридоновсергей спиридонов
сергей спиридонов
kuchinskaya
 
чашкин иван
чашкин иванчашкин иван
чашкин иван
kuchinskaya
 
дыдыкин егор
дыдыкин егордыдыкин егор
дыдыкин егор
kuchinskaya
 
сумин андрей
сумин андрейсумин андрей
сумин андрей
kuchinskaya
 
Jouravski kickstart1
Jouravski kickstart1Jouravski kickstart1
Jouravski kickstart1
kuchinskaya
 
ярослав рабоволюк
ярослав рабоволюкярослав рабоволюк
ярослав рабоволюк
kuchinskaya
 

Viewers also liked (9)

максим бабич
максим бабичмаксим бабич
максим бабич
 
сергей спиридонов
сергей спиридоновсергей спиридонов
сергей спиридонов
 
чашкин иван
чашкин иванчашкин иван
чашкин иван
 
дыдыкин егор
дыдыкин егордыдыкин егор
дыдыкин егор
 
сумин андрей
сумин андрейсумин андрей
сумин андрей
 
Balashov
BalashovBalashov
Balashov
 
Jouravski kickstart1
Jouravski kickstart1Jouravski kickstart1
Jouravski kickstart1
 
ярослав рабоволюк
ярослав рабоволюкярослав рабоволюк
ярослав рабоволюк
 
Kharkov
KharkovKharkov
Kharkov
 

Similar to Buytaert kris my_sql-pacemaker

Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker
Kris Buytaert
 
Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker
Kris Buytaert
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
Lenz Grimmer
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
Lenz Grimmer
 
Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02
Louis liu
 
MySQL HA with Pacemaker
MySQL HA with  PacemakerMySQL HA with  Pacemaker
MySQL HA with Pacemaker
Kris Buytaert
 
MySQL HA with PaceMaker
MySQL HA with  PaceMakerMySQL HA with  PaceMaker
MySQL HA with PaceMaker
Kris Buytaert
 
Loadays MySQL
Loadays MySQLLoadays MySQL
Loadays MySQL
lefredbe
 
MariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale:  an Intelligent Database ProxyMariaDB MaxScale:  an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database Proxy
Markus Mäkelä
 
UKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL TuningUKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL Tuning
FromDual GmbH
 
MariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database ProxyMariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database Proxy
Markus Mäkelä
 
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar AhmedPGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
Equnix Business Solutions
 
Proper Care and Feeding of a MySQL Database for Busy Linux Administrators
Proper Care and Feeding of a MySQL Database for Busy Linux AdministratorsProper Care and Feeding of a MySQL Database for Busy Linux Administrators
Proper Care and Feeding of a MySQL Database for Busy Linux Administrators
Dave Stokes
 
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
Dave Stokes
 
CPU Caches
CPU CachesCPU Caches
CPU Caches
shinolajla
 
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux AdminsLinuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
Dave Stokes
 
The Proper Care and Feeding of MySQL Databases
The Proper Care and Feeding of MySQL DatabasesThe Proper Care and Feeding of MySQL Databases
The Proper Care and Feeding of MySQL Databases
Dave Stokes
 
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
The Peoper Care and Feeding of a MySQL Server for Busy Linux AdminThe Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
Dave Stokes
 
MySQL HA Alternatives 2010
MySQL  HA  Alternatives 2010MySQL  HA  Alternatives 2010
MySQL HA Alternatives 2010
Kris Buytaert
 
The Accidental DBA
The Accidental DBAThe Accidental DBA
The Accidental DBA
PostgreSQL Experts, Inc.
 

Similar to Buytaert kris my_sql-pacemaker (20)

Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker
 
Linux-HA with Pacemaker
Linux-HA with PacemakerLinux-HA with Pacemaker
Linux-HA with Pacemaker
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
 
Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02Mysqlhacodebits20091203 1260184765-phpapp02
Mysqlhacodebits20091203 1260184765-phpapp02
 
MySQL HA with Pacemaker
MySQL HA with  PacemakerMySQL HA with  Pacemaker
MySQL HA with Pacemaker
 
MySQL HA with PaceMaker
MySQL HA with  PaceMakerMySQL HA with  PaceMaker
MySQL HA with PaceMaker
 
Loadays MySQL
Loadays MySQLLoadays MySQL
Loadays MySQL
 
MariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale:  an Intelligent Database ProxyMariaDB MaxScale:  an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database Proxy
 
UKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL TuningUKOUG 2011: Practical MySQL Tuning
UKOUG 2011: Practical MySQL Tuning
 
MariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database ProxyMariaDB MaxScale: an Intelligent Database Proxy
MariaDB MaxScale: an Intelligent Database Proxy
 
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar AhmedPGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
PGConf.ASIA 2019 Bali - Tune Your LInux Box, Not Just PostgreSQL - Ibrar Ahmed
 
Proper Care and Feeding of a MySQL Database for Busy Linux Administrators
Proper Care and Feeding of a MySQL Database for Busy Linux AdministratorsProper Care and Feeding of a MySQL Database for Busy Linux Administrators
Proper Care and Feeding of a MySQL Database for Busy Linux Administrators
 
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
The Proper Care and Feeding of a MySQL Database for Busy Linux Admins -- SCaL...
 
CPU Caches
CPU CachesCPU Caches
CPU Caches
 
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux AdminsLinuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
Linuxfest Northwest Proper Care and Feeding Of a MySQL for Busy Linux Admins
 
The Proper Care and Feeding of MySQL Databases
The Proper Care and Feeding of MySQL DatabasesThe Proper Care and Feeding of MySQL Databases
The Proper Care and Feeding of MySQL Databases
 
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
The Peoper Care and Feeding of a MySQL Server for Busy Linux AdminThe Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
The Peoper Care and Feeding of a MySQL Server for Busy Linux Admin
 
MySQL HA Alternatives 2010
MySQL  HA  Alternatives 2010MySQL  HA  Alternatives 2010
MySQL HA Alternatives 2010
 
The Accidental DBA
The Accidental DBAThe Accidental DBA
The Accidental DBA
 

More from kuchinskaya

Smirnov dependency-injection-techforum(1)
Smirnov dependency-injection-techforum(1)Smirnov dependency-injection-techforum(1)
Smirnov dependency-injection-techforum(1)
kuchinskaya
 
Smirnov reverse-engineering-techforum
Smirnov reverse-engineering-techforumSmirnov reverse-engineering-techforum
Smirnov reverse-engineering-techforum
kuchinskaya
 
Perepelitsa
PerepelitsaPerepelitsa
Perepelitsa
kuchinskaya
 
Kalugin balashov
Kalugin balashovKalugin balashov
Kalugin balashov
kuchinskaya
 
владимир габриелян
владимир габриелянвладимир габриелян
владимир габриелян
kuchinskaya
 
митасов роман
митасов романмитасов роман
митасов роман
kuchinskaya
 

More from kuchinskaya (20)

Zamyakin
ZamyakinZamyakin
Zamyakin
 
Panfilov
PanfilovPanfilov
Panfilov
 
Platov
PlatovPlatov
Platov
 
Rabovoluk
RabovolukRabovoluk
Rabovoluk
 
Smirnov dependency-injection-techforum(1)
Smirnov dependency-injection-techforum(1)Smirnov dependency-injection-techforum(1)
Smirnov dependency-injection-techforum(1)
 
Smirnov reverse-engineering-techforum
Smirnov reverse-engineering-techforumSmirnov reverse-engineering-techforum
Smirnov reverse-engineering-techforum
 
Zacepin
ZacepinZacepin
Zacepin
 
Zagursky
ZagurskyZagursky
Zagursky
 
Haritonov
HaritonovHaritonov
Haritonov
 
Chudov
ChudovChudov
Chudov
 
Bubnov
BubnovBubnov
Bubnov
 
A.pleshkov
A.pleshkovA.pleshkov
A.pleshkov
 
Zenovich
ZenovichZenovich
Zenovich
 
Romanenko
RomanenkoRomanenko
Romanenko
 
Perepelitsa
PerepelitsaPerepelitsa
Perepelitsa
 
Osipov
OsipovOsipov
Osipov
 
Kubasov
KubasovKubasov
Kubasov
 
Kalugin balashov
Kalugin balashovKalugin balashov
Kalugin balashov
 
владимир габриелян
владимир габриелянвладимир габриелян
владимир габриелян
 
митасов роман
митасов романмитасов роман
митасов роман
 

Recently uploaded

Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
Eric D. Schabell
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
Andrey Yasko
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
ScyllaDB
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Chris Swan
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
UiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs ConferenceUiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs Conference
UiPathCommunity
 
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Erasmo Purificato
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
Matthew Sinclair
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
Stephanie Beckett
 
Best Programming Language for Civil Engineers
Best Programming Language for Civil EngineersBest Programming Language for Civil Engineers
Best Programming Language for Civil Engineers
Awais Yaseen
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
Matthew Sinclair
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
Matthew Sinclair
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
Larry Smarr
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
BookNet Canada
 

Recently uploaded (20)

Observability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetryObservability For You and Me with OpenTelemetry
Observability For You and Me with OpenTelemetry
 
Comparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdfComparison Table of DiskWarrior Alternatives.pdf
Comparison Table of DiskWarrior Alternatives.pdf
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
UiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs ConferenceUiPath Community Day Kraków: Devs4Devs Conference
UiPath Community Day Kraków: Devs4Devs Conference
 
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...
 
20240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 202420240702 QFM021 Machine Intelligence Reading List June 2024
20240702 QFM021 Machine Intelligence Reading List June 2024
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
What's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptxWhat's New in Copilot for Microsoft365 May 2024.pptx
What's New in Copilot for Microsoft365 May 2024.pptx
 
Best Programming Language for Civil Engineers
Best Programming Language for Civil EngineersBest Programming Language for Civil Engineers
Best Programming Language for Civil Engineers
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
 
20240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 202420240704 QFM023 Engineering Leadership Reading List June 2024
20240704 QFM023 Engineering Leadership Reading List June 2024
 
The Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU CampusesThe Increasing Use of the National Research Platform by the CSU Campuses
The Increasing Use of the National Research Platform by the CSU Campuses
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
 

Buytaert kris my_sql-pacemaker

  • 1. MySQL HA with PaceMaker Kris Buytaert
  • 2. Kris Buytaert ● CTO and Open Source Consultant @inuits.eu ● „Infrastructure Architect“ ● I don't remember when I started using MySQL ● Specializing in Automated , Large Scale Deployments , Highly Available infrastructures, since 2008 also known as “the Cloud” th ● Surviving the 10 floor test ● Cofounded devopsdays.org
  • 3. In this presentation ● High Availability ? ● MySQL HA Solutions ● MySQL Replication ● Linux HA / Pacemaker
  • 4. What is HA Clustering ? ● One service goes down => others take over its work ● IP address takeover, service takeover, ● Not designed for high-performance ● Not designed for high troughput (load balancing)
  • 5. Does it Matter ? ● Downtime is expensive ● You mis out on $$$ ● Your boss complains ● New users don't return
  • 6. Lies, Damn Lies, and Statistics Counting nines (slide by Alan R) 99.9999% 30 sec 99.999% 5 min 99.99% 52 min 99.9% 9 hr 99% 3.5 day
  • 7. The Rules of HA ● Keep it Simple ● Keep it Simple ● Prepare for Failure ● Complexity is the enemy of reliability ● Test your HA setup
  • 8. You care about ? ● Your data ? •Consistent •Realitime •Eventual Consistent ● Your Connection •Always •Most of the time
  • 9. Eliminating the SPOF ● Find out what Will Fail •Disks •Fans •Power (Supplies) ● Find out what Can Fail •Network •Going Out Of Memory
  • 10. Split Brain ● Communications failures can lead to separated partitions of the cluster ● If those partitions each try and take control of the cluster, then it's called a split-brain condition ● If this happens, then bad things will happen •http://linux-ha.org/BadThingsWillHappen
  • 11. Historical MySQL HA ● Replication •1 read write node •Multiple read only nodes •Application needed to be modified
  • 12. Solutions Today ● BYO ● DRBD ● MySQL Cluster NDBD ● Multi Master Replication ● MySQL Proxy ● MMM / Flipper ● Galera ● Percona XtraDB Cluster
  • 13. Data vs Connection ● DATA : •Replication •DRBD ● Connection •LVS •Proxy •Heartbeat / Pacemaker
  • 14. Shared Storage ● 1 MySQL instance ● Monitor MySQL node ● Stonith ● $$$ 1+1 <> 2 ● Storage = SPOF ● Split Brain :(
  • 15. DRBD ● Distributed Replicated Block Device ● In the Linux Kernel (as of very recent) ● Usually only 1 mount •Multi mount as of 8.X •Requires GFS / OCFS2 ● Regular FS ext3 ... ● Only 1 MySQL instance Active accessing data ● Upon Failover MySQL needs to be started on other node
  • 16. DRBD(2) ● What happens when you pull the plug of a Physical machine ? •Minimal Timeout •Why did the crash happen ? •Is my data still correct ? •Innodb Consistency Checks ? •Lengthy ? •Check your BinLog size
  • 17. MySQL Cluster NDBD ● Shared-nothing architecture ● Automatic partitioning ● Synchronous replication ● Fast automatic fail-over of data nodes ● In-memory indexes ● Not suitable for all query patterns (multi-table JOINs, range scans)
  • 19. MySQL Cluster NDBD ● All indexed data needs to be in memory ● Good and bad experiences •Better experiences when using the API •Bad when using the MySQL Server ● Test before you deploy ● Does not fit for all apps
  • 20. How replication works ● Master server keeps track of all updates in the Binary Log •Slave requests to read the binary update log •Master acts in a passive role, not keeping track of what slave has read what data ● Upon connecting the slaves do the following: •The slave informs the master of where it left off •It catches up on the updates •It waits for the master to notify it of new updates
  • 22. Two Slave Threads ● How does it work? •The I/O thread connects to the master and asks for the updates in the master’s binary log •The I/O thread copies the statements to the relay log •The SQL thread implements the statements in the relay log Advantages •Long running SQL statements don’t block log downloading •Allows the slave to keep up with the master better •In case of master crash the slave is more likely to have all statements
  • 23. Replication commands Slave commands ● START|STOP SLAVE ● RESET SLAVE ● SHOW SLAVE STATUS ● CHANGE MASTER TO… ● LOAD DATA FROM MASTER ● LOAD TABLE tblname FROM MASTER Master commands ● SHOW MASTER STATUS ● PURGE MASTER LOGS…
  • 24. Show slave statusG Slave_IO_State: Waiting for master to send event Master_Host: 172.16.0.1 Master_User: repli Master_Port: 3306 Connect_Retry: 60 Master_Log_File: XMS-1-bin.000014 Read_Master_Log_Pos: 106 Relay_Log_File: XMS-2-relay.000033 Relay_Log_Pos: 251 Relay_Master_Log_File: XMS-1-bin.000014 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: xpol Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 547 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)
  • 25. Row vs Statement ● Pro ● Pro •All changes can be replicated •Proven (around since MySQL 3.23) •Similar technology used by other •Smaller log files RDBMSes •Fewer locks required for some •Auditing of actual SQL statements INSERT, UPDATE or DELETE statements •No primary key requirement for ● Con replicated tables •More data to be logged ● Con •Log file size increases (backup/restore implications) •Non-deterministic functions and •Replicated tables require explicit UDFs primary keys •Possible different result sets on bulk INSERTs
  • 26. Multi Master Replication ● Replicating the same table data both ways can lead to race conditions •Auto_increment, unique keys, etc.. could cause problems If you write them 2x ● Both nodes are master ● Both nodes are slave ● Write in 1 get updates on the other M|S M|S
  • 27. MySQL Proxy ● Man in the middle ● Decides where to connect to •LUA ● Write rules to •Redirect traffic •
  • 28. Master Slave & Proxy ● Split Read and Write Actions ● No Application change required ● Sends specific queries to a specific node ● Based on •Customer •User •Table •Availability
  • 29. MySQL Proxy ● Your new SPOF ● Make your Proxy HA too ! •Heartbeat OCF Resource
  • 30. Breaking Replication ● If the master and slave gets out of sync ● Updates on slave with identical index id •Check error log for disconnections and issues with replication
  • 31. Monitor your Setup ● Not just connectivity ● Also functional •Query data •Check resultset is correct ● Check replication •MaatKit •OpenARK
  • 32. Pulling Traffic ● Eg. for Cluster, MultiMaster setups •DNS •Advanced Routing •LVS •Flipper / MMM
  • 33. MMM ● Multi-Master Replication Manager for MySQL •Perl scripts to perform monitoring/failover and management of MySQL master- master replication configurations ● Balance master / slave configs based on replication state •Map Virtual IP to the Best Node ● http://mysql-mmm.org/
  • 34. Flipper ● Flipper is a Perl tool for managing read and write access pairs of MySQL servers ● master-master MySQL Servers ● Clients machines do not connect "directly" to either node instead, ● One IP for read, ● One IP for write. ● Flipper allows you to move these IP addresses between the nodes in a safe and controlled manner. ● http://provenscaling.com/softw are/flipper/
  • 35. Linux-HA PaceMaker ● Plays well with others ● Manages more than MySQL ● ● ...v3 .. don't even think about the rest anymore ● ● http://clusterlabs.org/
  • 36. Heartbeat ● Heartbeat v1 •Max 2 nodes •No finegrained resources •Monitoring using “mon” ● Heartbeat v2 •XML usage was a consulting opportunity •Stability issues •Forking ?
  • 37. Pacemaker Architecture ● Stonithd : The Heartbeat fencing subsystem. ● Lrmd : Local Resource Management Daemon. Interacts directly with resource agents (scripts). ● pengine Policy Engine. Computes the next state of the cluster based on the current state and the configuration. ● cib Cluster Information Base. Contains definitions of all cluster options, nodes, resources, their relationships to one another and current status. Synchronizes updates to all cluster nodes. ● crmd Cluster Resource Management Daemon. Largely a message broker for the PEngine and LRM, it also elects a leader to co-ordinate the activities of the cluster. ● openais messaging and membership layer. ● heartbeat messaging layer, an alternative to OpenAIS. ● ccm Short for Consensus Cluster Membership. The Heartbeat membership layer.
  • 38. Pacemaker ? ● Not a fork ● Only CRM Code taken out of Heartbeat ● As of Heartbeat 2.1.3 •Support for both OpenAIS / HeartBeat •Different Release Cycles as Heartbeat
  • 39. Heartbeat, OpenAis ? ● Both Messaging Layers ● Initially only Heartbeat ● OpenAIS ● Heartbeat got unmaintained ● OpenAIS has heisenbugs :( ● Heartbeat maintenance taken over by LinBit ● CRM Detects which layer
  • 40. Pacemaker Heartbeat or OpenAIS Cluster Glue
  • 41. Configuring Heartbeat ● /etc/ha.d/ha.cf Use crm = yes ● /etc/ha.d/authkeys
  • 42. Configuring Heartbeat heartbeat::hacf {"clustername": hosts => ["host-a","host-b"], hb_nic => ["bond0"], hostip1 => ["10.0.128.11"], hostip2 => ["10.0.128.12"], ping => ["10.0.128.4"], } heartbeat::authkeys {"ClusterName": password => “ClusterName ", } http://github.com/jtimberman/puppet/tree/master/heartbeat/
  • 43. Heartbeat Resources ● LSB ● Heartbeat resource (+status) ● OCF (Open Cluster FrameWork) (+monitor) ● Clones (don't use in HAv2) ● Multi State Resources
  • 44. A MySQL Resource ● OCF •Clone •Where do you hook up the IP ? •Multi State •But we have Master Master replication •Meta Resource •Dummy resource that can monitor •Connection •Replication state
  • 45. CRM configure ● Cluster Resource property $id="cib-bootstrap- options" Manager stonith-enabled="FALSE" no-quorum-policy=ignore ● Keeps Nodes in Sync start-failure-is-fatal="FALSE" rsc_defaults $id="rsc_defaults- options" ● XML Based migration-threshold="1" failure-timeout="1" ● cibadm primitive d_mysql ocf:local:mysql op monitor interval="30s" ● Cli manageable params test_user="sure" test_passwd="illtell" test_table="test.table" ● Crm primitive ip_db ocf:heartbeat:IPaddr2 params ip="172.17.4.202" nic="bond0" op monitor interval="10s" group svc_db d_mysql ip_db commit
  • 46. Adding MySQL to the stack Replication Service IP MySQL “MySQLd” “MySQLd” Resource MySQL Cluster Stack Pacemaker HeartBeat Node A Node B Hardware
  • 47. Pitfalls & Solutions ● Monitor, •Replication state •Replication Lag ● MaatKit ● OpenARK
  • 48. Conclusion ● Plenty of Alternatives ● Think about your Data ● Think about getting Queries to that Data ● Complexity is the enemy of reliability ● Keep it Simple ● Monitor inside the DB
  • 49. Contact Kris Buytaert Kris.Buytaert@inuits.be Further Reading @krisbuytaert http://www.krisbuytaert.be/blog/ http://www.inuits.be/ •Or the upcoming slides Inuits 't Hemeltje Duboistraat 50 2060 Antwerpen Belgium 891.514.231 +32 475 961221