MySQL HA with PaceMaker
- 5. Specializing in Automated , Large Scale Deployments , Highly Available infrastructures, since 2008 also known as “the Cloud”
- 12. What is HA Clustering ? One service goes down => others take over its work IP address takeover, service takeover,
- 34. If those partitions each try and take control of the cluster, then it's called a split-brain condition
- 35. If this happens, then bad things will happen http://linux-ha.org/BadThingsWillHappen
- 58. Usually only 1 mount Multi mount as of 8.X Requires GFS / OCFS2 Regular FS ext3 ...
- 74. Good and bad experiences Better experiences when using the API
- 77. How replication works Master server keeps track of all updates in the Binary Log Slave requests to read the binary update log
- 78. Master acts in a passive role, not keeping track of what slave has read what data Upon connecting the slaves do the following: The slave informs the master of where it left off
- 80. It waits for the master to notify it of new update s
- 82. Two Slave Threads How does it work? The I/O thread connects to the master and asks for the updates in the master’s binary log
- 84. The SQL thread implements the statements in the relay log Advantages Long running SQL statements don’t block log downloading
- 86. In case of master crash the slave is more likely to have all statements
- 94. Show slave status Slave_IO_State: Waiting for master to send event Master_Host: 172.16.0.1 Master_User: repli Master_Port: 3306 Connect_Retry: 60 Master_Log_File: XMS-1-bin.000014 Read_Master_Log_Pos: 106 Relay_Log_File: XMS-2-relay.000033 Relay_Log_Pos: 251 Relay_Master_Log_File: XMS-1-bin.000014 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: xpol Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 106 Relay_Log_Space: 547 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: 1 row in set (0.00 sec)
- 98. No primary key requirement for replicated tables Con Non-deterministic functions and UDFs Pro All changes can be replicated
- 104. Multi Master Replication Replicating the same table data both ways can lead to race conditions Auto_increment, unique keys, etc.. could cause problems If you write them 2x Both nodes are master
- 119. Updates on slave with identical index id Check error log for disconnections and issues with replication
- 128. MMM Multi-Master Replication Manager for MySQL Perl scripts to perform monitoring/failover and management of MySQL master-master replication configurations Balance master / slave configs based on replication state Map Virtual IP to the Best Node http://mysql-mmm.org/
- 129. Flipper Flipper is a Perl tool for managing read and write access pairs of MySQL servers
- 134. Flipper allows you to move these IP addresses between the nodes in a safe and controlled manner.
- 146. Lrmd : Local Resource Management Daemon. Interacts directly with resource agents (scripts).
- 147. pengine Policy Engine. Computes the next state of the cluster based on the current state and the configuration.
- 148. cib Cluster Information Base. Contains definitions of all cluster options, nodes, resources, their relationships to one another and current status. Synchronizes updates to all cluster nodes.
- 149. crmd Cluster Resource Management Daemon. Largely a message broker for the PEngine and LRM, it also elects a leader to co-ordinate the activities of the cluster.
- 150. openais messaging and membership layer.
- 151. heartbeat messaging layer, an alternative to OpenAIS.
- 152. ccm Short for Consensus Cluster Membership. The Heartbeat membership layer.
- 166. Configuring Heartbeat heartbeat::hacf {"clustername": hosts => ["host-a","host-b"], hb_nic => ["bond0"], hostip1 => ["10.0.128.11"], hostip2 => ["10.0.128.12"], ping => ["10.0.128.4"], } heartbeat::authkeys {"ClusterName": password => “ClusterName ", } http://github.com/jtimberman/puppet/tree/master/heartbeat/
- 172. The MySQL Resource OCF Clone Where do you hook up the IP ? Multi State But we have Master Master replication Meta Resource Dummy resource that can monitor Connection
- 180. Crm configure property $id="cib-bootstrap-options" stonith-enabled="FALSE" no-quorum-policy=ignore start-failure-is-fatal="FALSE" rsc_defaults $id="rsc_defaults-options" migration-threshold="1" failure-timeout="1" primitive d_mysql ocf:local:mysql op monitor interval="30s" params test_user="sure" test_passwd="illtell" test_table="test.table" primitive ip_db ocf:heartbeat:IPaddr2 params ip="172.17.4.202" nic="bond0" op monitor interval="10s" group svc_db d_mysql ip_db commit
- 181. Hardware Cluster Stack Resource MySQL Replication Adding MySQL to the stack Node A Node B HeartBeat Pacemaker “ MySQLd” “ MySQLd” Service IP MySQL
- 191. ` Kris Buytaert < [email_address] > Further Reading http://www.krisbuytaert.be/blog/ http://www.inuits.be/ http://www.virtualization.com/ http://www.oreillygmt.com/ ? !