Severalnines Self-Training: MySQL® Cluster - Part VII

MySQL Cluster Training
presented by severalnines.com

Address:
Contact: SeveralninesAB
Jean-Jérôme Schmidt c/o SICS, Box 1263
Email: services@severalnines.com Isafjordsgatan22
SE-164-29 Kista

Copyright 2011 Severalnines AB Control your database infrastructure 1

Introduction

• At Severalnines, we believe in sharing information
and knowledge; we all come from an open source
background
• We know a lot of things about MySQL Cluster and
think that MySQL Cluster is a great technology
• These free MySQL Cluster Training slides are a
contribution of ours to the knowledge and information
sharing that‟s common practice in the open source
community
• If you have any questions on these slides or would
like to book an actual training class, please contact
us at: services@severalnines.com


Training Slides - Concept

• Over the coming weeks we will be chronologically
releasing slides for the different sections of our
MySQL Cluster Training program on our website.
• The full agenda of the training with all of its modules
is outlined in the next slides so that you can see what
topics will be covered over the coming weeks.
• Particularly specialised topics such as Cluster/J or
NDB API are not fully covered in the slides. We
recommend our instructor-led training classes for
such topics.
• Please contact us for more details:
services@severalnines.com


Full Training Agenda (1/4)
• MySQL Cluster Introduction
– MySQL eco system
– Scale up, scale out, and sharding
– MySQL Cluster Architecture
– Use cases
– Features
– Node types and Roles
• Detailed Concepts
– Data Distribution
– Verifying data distribution
– Access Methods
– Partitioning
– Node failures and failure detection
– Network Partitioning
– Transactions and Locking
– Consistency Model
– Redo logging and Checkpointing
• Internals
– NDB Design Internals


Agenda (2/4)

• Installing MySQL Cluster
– Setting up MySQL Cluster
– Starting/stopping nodes
– Recovery and restarts
– Upgrading configuration
– Upgrading Cluster
• Performance Tuning (instructor-led only; contact us at services@severalnines.com)
– Differences compared to Innodb/MyISAM
– Designing efficient and fast applications
– Identifying bottlenecks
– Tweaking configuration (OS and MySQL Cluster)
– Query Tuning
– Schema Design
– Index Tuning


Agenda (3/4)

• Management and Administration
– Backup and Restore
– Geographical Replication
– Online and offline operations
– Ndbinfo tables
– Reporting
– Single user mode
– Scaling Cluster
• Disk Data
– Use cases
– Limitations
– Best practice configuration
• Designing a Cluster
– Capacity Planning and Dimensioning
– Hardware recommendations
– Best practice Configuration
– Storage calculations


Agenda (4/4)

• Resolving Issues
– Common problems
– Error logs and Tracefiles
– Recovery and Escalation procedures
• Connectivity Overview
– NDBAPI
– Cluster/J
– LDAP
• Severalnines Tools
– Monitoring and Management
– Benchmarking
– Sandboxes
– Configuration and capacity planning
• Conclusion


Agenda: Lab Exercises
(only applicable to instructor-led training classes)

• Lab Exercises
– Installing and Loading data into MySQL Cluster
– Starting/stopping nodes, recovery
– Query tuning
– Backup and Restore
– Configuration Upgrade

• Would you like to try something particular?
– This is possible too, speak with your instructor


Prerequisites
• Readers / Participants have understanding of SQL and basic database concepts.

• Laptops/PCs for hands-on exercises
• Linux: 1GB RAM
• Windows: 2GB RAM
• Approx. 20GB disk space and Virtualbox installed.
• Virtualbox can be downloaded for free at http://www.virtualbox.org/wiki/Downloads

• MySQL Cluster version 7.1 or later


7th Installment
Severalnines Cluster Self-Training

Part 6: Management and
Administration


Topics covered in Installment 7

• Backup and Restore
• Geographical Redundancy
• Online and offline operations
• Ndbinfo tables
• Reporting
• Single user mode
• Scaling Cluster


Management and Administration
Backup and Restore


Backup and Restore

• The following should be backed up in MySQL Cluster
– NDBCLUSTER tables
– MySQL System tables (stored in MYISAM)
– MYISAM/INNODB tables – optional
– Binary logs (optional)
– Configuration files (config.ini and my.cnf)


Backup and Restore

• NDBCLUSTER tables
– ONLINE BACKUP is built into MySQL Cluster
• Only tables stored in the NDBCLUSTER storage engine can be
backed up ONLINE.
• INNOB/MYISAM tables
– mysqldump
– Oracle Hotbackup
• Binary logs
– Copy/rsync
• Config files
– Copy/rsync


Online Backup
START BACKUP
• Backups are started from the
management client or mgmapi
– ndb_mgm –e “START BACKUP” Management server

• Each data node will backup its
STORAGE LAYER
primary partition
– Consistent at “backup complete” DATA DATA
“backup start” time, as you wish NODE 1 NODE 2

• Thus the complete backup is P0 P1
spread over several computers S1 S0


Online Backup
START BACKUP

• Backups are written to a backup directory: Management server

– Default : DATADIR
– BACKUP_DATADIR STORAGE LAYER

• BACKUP_DATADIR can be
a mount point on a SAN. DATA
NODE 1
DATA
NODE 2

• Each backup gets an ID
– Starting from 1 and always incrementing P0 P1
S1 S0
• The backup consists of three files:
– Data file (the records in data memory)
• Backup-1.data
– Log file (changes during the backup)
• Backup-1.log
– Meta data (indexes and tables) /data/mysqlcluster/BACKUP/BACKUP-1

• Backup-1.ctl

Backup and Restore

• Exercise – BACKUP


Restore

• The ndb_restoreprogram is used to a MySQL Cluster
backup.
• A backup taken from two data nodes will have two
sets of backup files
– A backup taken from four data nodes will have four sets of
backup files.
• All backup sets must be restored
– Ndb_restore can restore one backup set at a time.
– Exception – meta data needs to be restored only once.


Restore

• Two data nodes with id=2 and id=3, backup id=1
• To restore the meta data (needs to be executed only
once):
ndb_restore –c<connectstring> --restore_meta
--node-id=2 --backup-id=1 /path/to/backup/files
• To restore the data and logs – must be executed
once for each backup set:
ndb_restore –c<connectstring> --restore_data
ndb_restore –c<connectstring> --restore_data


Backup and Restore

• Exercise – Restore


Backup and Restore

• SeveralninesClusterControl allows you to:
– Backup data to a central location instead of having the
backup files spread out of X computers.
– Create backup schedules
– Check backup history
– Restore backups easily


Geographical Redundancy

• Uses the Normal MySQL Replication to transfer data
between a Master and a Slave.
– Binary logs
– Positions as offsets in the files.
• Added extra information to allow failover.



STORAGE LAYER • Application inserts data
• Committed changes from
DATA DATA
NODE 1 NODE 2 data nodes are sent to all
mysql servers with --log-
P0 P1
bin set.
S1 S0
– This means that several
mysql servers can have
MASTER – serverid=1
equivalent binary log
Injector
Thead

BINARY LOG



STORAGE LAYER

DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0


BINARY LOG



STORAGE LAYER • Every
TimeBetweenEpochs
DATA
NODE 1
DATA
NODE 2
(default 100ms) there is
a EPOCH_COMPLETE
P0 P1 message sent to the
S1 S0
Masters.
EPOCH_COMPLETE
12 • The Master binlog buffer
MASTER – serverid=1 will be flushed to the
Binary log
• Epochs contains 0 or
BINARY LOG
more transactions


NDB_BINLOG_INDEX
STORAGE LAYER
Epoch Binlog Pos Type
DATA DATA 12 b.01 4 I
NODE 1 NODE 2
12 b.01 28 I

P0 P1
12 b.01 32 U
S1 S0 15 b.01 255 U

• The NDB_BINLOG_INDEX
is updated.
• Stores a map between Epoch
Copy to Binary log
and Binlog/pos
12

4 28 32 255


NDB_APPLY_STATUS
STORAGE LAYER
Epoch Binlog Pos Type
DATA DATA 12 b.01 4 I
NODE 1 NODE 2
12 b.01 28 I

P0 P1
12 b.01 32 U
S1 S0 15 b.01 255 U

• The Master writes the unsent
MASTER binary log data to the
slave network

12

4 28 32 255


STORAGE LAYER STORAGE LAYER

DATA DATA DATA DATA
NODE 1 NODE 2 NODE 1 NODE 2

P0 P1 P0 P1

S1 S0 S1 S0

MASTER – serverid=1 SLAVE

IO THREAD receives data and put in RELAY LOG
Copy to Binary log

12 12

4 28 32 255 4 28 32 255


STORAGE LAYER

DATA DATA
NODE 1 NODE 2

P0 P1
S1 S0

SLAVE

SQL Thread reads from the RELAY LOG

12

4 28 32 255



STORAGE LAYER

DATA DATA
NDB_APPLY_STATUS
NODE 1 NODE 2

Server Id Last Epoch
P0 P1
1 12
S1 S0

• NDB_APPLY_STATUS
SLAVE
• Last applied epoch and
Applies the entire epoch and updates
NDB_APPLY_STATUS table where it came from.
12
• An epoch is applied as
4 28 32 255 one transaction.

Geographical Redundancy Failover
Ndb_ MASTER – serverid=1 SLAVE
Binlog
ndex

Copy to Binary log IO THREAD receives data and put in RELAY LOG

12 12

4 28 32 255 4 28 32 255


DATA DATA
DATA DATA
NODE 1 NODE 2
NODE 1 NODE 2
NDB_APPLY_STATUS

P0 P1
P0 P1
S1 S0
S1 S0

Ndb_
Binlog
ndex

Copy to Binary log

12

4 28 32 255 4 28 32 255

Geographical Redundancy Failover
Ndb_ MASTER – serverid=1 SLAVE
Binlog
ndex

Copy to Binary log

14 12

4 28 32 255 4 28 32 255


DATA DATA
DATA DATA
NODE 1 NODE 2
NODE 1 NODE 2
NDB_APPLY_STATUS
1 - 12
P0 P1
P0 P1
S1 S0
S1 S0

Ndb_
Binlog Failover –check what is the applied epoch.
ndex
Find corresponding binlog/pos.
Copy to Binary log Change master
Start replication
14

4 28 32 255 4 28 32 255

Staging the slave

• The slave cluster is staged by restoring a backup
from the Master Cluster
– ndb_restore … -e
• -e - restore epoch
• The –e is very important, otherwise the slave cluster will not get
the last applied epoch written in ndb_apply_status table.
• The last applied epoch will be the epoch when the backup
finished.



• It is also possible to have conflict detection and
resolution.
• Out of scope in this training.



• Exercise – setup Geo rep.


Offline and online operations


Online/offline Operations

• Online Operation
– There can still be traffic on the Cluster while the operation is
taking place
– No Service Interruption
• Offline Operation
– Traffic needs to be blocked in order to implement the
change.
– Service Interruption


Online operations

• Adding nodes
– Adding data nodes
– Adding mysql servers
• Changing configuration
– values except
• Hostnames
• NoOfReplicas
• Upgrade
– Software
– Hardware
– OS
• Schema modifications
– Add column (ALTER ONLINE TABLE… ADD COLUMN x
INT)
– Create Index / Drop Index

Offline operations

• NoOfReplicas=2  3
– Requires a backup and restore
• Schema Changes
– data types, column sizes, names
– The table in question must be locked:
• On all mysql servers except one: LOCK TABLE t1 READ;
• On the exception: ALTER TABLE t1 MODIFY b CHAR(255);
• UNLOCK TABLES;


Monitoring


Monitoring MySQL Cluster

• MySQL Cluster offers the following monitoring
interfaces
– NDBINFO - containing system tables stored accessible from
the mysql server
– LOGEVENTs – subscribe using the mgmapi
• SeveralninesClusterControl collects information from
both these sources.
– Web interface
– Command line tools


NDBINFO

• The NDBINFO (MySQL Cluster 7.1 and later) is a
schema containing statistical counters collected from
the data nodes.
– Memory usage (memoryusage)
– Statistical counters (counters)
– Redo log information (logspaces)
– Redo buffer information (logbuffers)
– Disk page buffer memory stats (diskpagebuffer)
• All this information is also collected by
SeveralninesClusterControl.


How do you spot issues?

• ClusterControl can assist
– Query Monitoring/Histogram
– Health Reports
– Configuration Management
• You can also watch our video describing how to find
configuration issues:
http://www.severalnines.com/resources/videos


Reporting


Reporting

• Best practice is to separate reporting from the real-
time database.
– Running reporting on a realtime database will affect
performance.
• From MySQL Cluster it is possible to replicate to an
InnodbMySQL server.
– Run the reporting/heavy queries on this server.


Reporting

MySQL Cluster

Application/realtime

Reporting, heavy queries
DATA Sync repl. DATA
NODE 0 NODE 1

Asynch. replication

Replication servers


Reporting

• How to set it up
– http://johanandersson.blogspot.com/2009/05/ha-mysql-write-
scaling-using-cluster-to.html
• If one master fails, the reporting slave can failover to
the other master
– No data loss
• If the reporting slave fails – restart it
– The reporting slave can reconnect and resume.


Single User Mode


Single User Mode

• MySQL Cluster has a Single User Mode that it is
possible to enter. In practice it is rarely used.
• ndb_mgm> ENTER SINGLE USER MODE 7
– This means that only an application with Nodeid=7 is
allowed to connect and operate on MySQL Cluster
– All other applications will be blocked
• ndb_mgm> EXIT SINGLE USER MODE
– Only useful if you have a dedicated adminstrative MySQL
Server.
– Then you “lock” the database and you can do offline ALTER
TABLEs from this MySQL Server.


Scaling MySQL Cluster


Scaling MySQL Cluster

• MySQL Cluster consists of two layers that can be
scaled
– Data/storage Layer
– Access Layer (mysql servers, application nodes)
• But how to know when it is time to scale?
• The following variables need to be studied
– Network
• Iftop
– Disk
• Iostat
– CPU
• top
• vmstat


SQL Layer/Access Layer

• To determine if the SQL Layer / Access Layer needs
to be scaled, it is sufficient to look at the CPU
utilization. (Network saturation is not very common).
– However if CPU utlization is low AND cpu utilization is low
on the data nodes the requests are spending time in the
network.
• Optimize those request (better indexes, force index etc)
• MySQL Servers can be added online, just install the
binaries, issue GRANTs and start the MySQL Server.
– Make sure you have connection slots available.
• With SeveralninesClusterControl, a new MySQL
Server can be provisioned with a single click.


Data Layer / Storage Layer

• Network
– Is the network fully utilized (this is very very seldom the
case)?
• CPU
– What is the %util of the TC, LQH or CMVMI threads?
– cat /data/mysqlcluster/ndb_X_out.log |greptid
• The PIDs of the different threads are printed out
– top –Hd1
• Is one of the threads mentioned above running at 100%
• If yes  add data node (one node group needs to be added at
a time).


Data Layer / Storage Layer

• Disk
– iostat –kx 1
– Is the ioutil at 100%?
• If yes  by better disks, checkpoint with a lower rate, increase
redo log size,
• It is also possible to add more data nodes in order to spread
out the write load across more instances.
• Check if the write back cache is enabled, if not, enable it.
– Note – E.g EC2 disks are inherently slow.
DiskCheckpointSpeed should be set lower, e.g 5MB/s.


Add Data Node Online

• There are two way of adding data nodes
– Dynamically / “Hard” - edit config.ini
• Cons: You must perform a rolling restart
– Statically / “Easy”: Plan ahead and allocate “slots” for data
nodes at configuration time
• Cons: You must know the hostnames from the beginning.
• Pros: Easy
• Start with 2 data nodes but configure for 8 data nodes
• It is not possible to shrink the number of data nodes.


Dynamic - Add Data Node Online

• Edit config.ini and add:
[NDBD]
Nodeid:X
Hostname=„newhost1‟
[NDBD]
Nodeid:Y
• Perform a rolling restart in the following order:
– Restart the management servers
– Restart the mysql servers
– Restart the already existing data nodes
– Start the data nodes on „newhost1‟ and „newhost2‟
ndbmtd --initial


“Static/plan ahead”- Add Data Node Online

• In config.ini before you start Cluster the first time
[NDBD]
Nodeid:X
Nodegroup: 65536 (magic number)
[NDBD]
Nodeid:Y
Nodegroup: 65536 (magic number)
• To add data nodes simply start the data nodes on
„newhost1‟ and „newhost2‟
– ndbmtd –initial
• Note, in some releases the magic number is 65535.

Add Data Node Online

• In any case the final step is to redistribute the data.
• Foreach table do:
– ALTER TABLE t1 REORGANIZE PARTITION
• This will rehash the table
• This is an ONLINE operation.
• It will take some time.
– http://johanandersson.blogspot.com/2009/03/online-add-
node-preliminary-numbers.html
– http://johanandersson.blogspot.com/2009/03/online-add-
node-preliminary-numbers-2.html
• However, it is not a common task to add data nodes.
Increasing/decreasing # mysql servers is more
common.

Coming next in Installment 8:

Disk Data


We hope these training slides are
useful to you!

Please visit our website to view the
next section of this training.

For any questions, comments, feedback or to
book a training class, please contact us at:

services@severalnines.com

Thank you!


Disclaimer

© Copyright 2011 Severalnines AB. All rights reserved.

Severalnines& the Severalnineslogo(s) are trademarks of Severalnines AB.

MySQL is a registered trademark of Oracle and/or its affiliates.

Other names may be trademarks of their respective owners.


Severalnines Self-Training: MySQL® Cluster - Part VII

Related slideshows

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

More Related Content

What's hot

What's hot (20)

Similar to Severalnines Self-Training: MySQL® Cluster - Part VII

Similar to Severalnines Self-Training: MySQL® Cluster - Part VII (20)

More from Severalnines

More from Severalnines (20)

Recently uploaded

Recently uploaded (20)

Severalnines Self-Training: MySQL® Cluster - Part VII

Editor's Notes