MySQL Cluster Performance Tuning - 2013 MySQL User Conference
•
3 likes•1,604 views
Slides from a presentation given at Percona Live MySQL Conference 2013 in Santa Clara, US. Topics include: - How to look for performance bottlenecks - Foreign Key performance in MySQL Cluster 7.3 - Sharding and table partitioning - efficient use of datatypes (e.g. BLOBS vs varbinary)
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to MySQL Cluster Performance Tuning - 2013 MySQL User Conference
Similar to MySQL Cluster Performance Tuning - 2013 MySQL User Conference (20)
More from Severalnines
More from Severalnines (20)
Recently uploaded
Recently uploaded (20)
MySQL Cluster Performance Tuning - 2013 MySQL User Conference
- 1. Performance Tuning of MySQL Cluster April 2013 Johan Andersson Severalnines AB johan@severalnines.com
- 2. Agenda 7.3 Feature Update OS Tuning Stability Tuning Application design Identifying bottlenecks Tuning tricks 2 Copyright 2011 Severalnines AB
- 3. 7.3 Feature Update Node.js Connector JavaScript (V8 engine) to access data directly in the Data nodes No SQL – bypasses the MySQL Server lower latency, high throughput for simple queries (like PK operations, simple scans from one table) 3 Copyright 2011 Severalnines AB
- 4. 7.3 Feature Update FOREIGN KEYs finally supported! Implemented at the Data Node level But.. ERROR 1506 (HY000): Foreign key clause is not yet supported in conjunction with partitioning Hopefully fixed for the GA release What about the performance penalty? 4 Copyright 2011 Severalnines AB
- 5. 7.3 Foreign Key Perf create table users_posts ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid), constraint fk_forum foreign key(fid) references forum(fid) on delete cascade, constraint fk_user foreign key(uid) references users(uid) on delete cascade ) engine=ndb; Compare INSERT performance with and w/o FKs With FK, must check that forum(fid) and users(uid) exists. Populate with 1M records 5 Copyright 2011 Severalnines AB
- 6. 7.3 Foreign Key Perf Bencher drivers the load: https://github.com/severalnines/bencher 4 threads, 2 data nodes, 4 cores, App mysqld data nodes FOREIGN KEYs enabled Summary: -------------------------- Average Throughput = 1274.58 tps (stdev=59.71) 6 Copyright 2011 Severalnines AB
- 7. 7.3 Foreign Key Perf Bencher drivers the load: https://github.com/severalnines/bencher 4 threads, 2 data nodes, 4 cores, App mysqld data nodes Not using FOREIGN KEYs Summary: -------------------------- Average Throughput = 1428.57 tps (stdev=57.10) Foreign keys gave ~11% drop in performance. 7 Copyright 2011 Severalnines AB
- 8. 7.2 7.3 Caveats Rolling upgrade from 7.2.10 to 7.3.1 works! A little gotcha: --engine-condition-pushdown no longer supported in MySQL 5.6 Mysqld will fail to start Take it out from my.cnf before upgrading! 8 Copyright 2011 Severalnines AB
- 9. Facts A single query will never run as fast as on Innodb (served from RAM) Network latency is a issue More data nodes does not speed up query execution time. 9 Copyright 2011 Severalnines AB
- 10. OS Tuning Disable NUMA in /etc/grub.conf echo ‘0’ > /proc/sys/vm/swappiness echo ‘vm.swappiness=0’ >> /etc/sysctl.conf Bind data node threads to CPUs/cores cat /proc/interrupts | grep eth cpu0 cpu1 cpu2 cpu3 44: 31 49432584 0 0 xen-dyn-event eth0" 45: 1633715292 0 0 0 xen-dyn-event eth1" 10 Copyright 2011 Severalnines AB Avoid! OK! In config.ini [ndbd default]: ThreadConfig=ldm={count=1,cpubind=1,2},main={cpubind=3} ..
- 11. Stability Tuning Tuning the REDO log is key FragmentLogFileSize=256M NoOfFragmentLogFiles=<4-6> X DataMemory in MB / 4 x FragmentLogFileSize RedoBuffer=64M for a write busy system Disk based data: SharedGlobalMemory=4096M In the LOGFILE GROUP: undo_buffer_size=128M Or higher (max is 600M) 11 Copyright 2011 Severalnines AB
- 12. Stability Tuning Make sure you don’t have more “execution threads” than cores You want to have Major page faults low Involuntary context switches low mysql> SELECT node_id, thr_no,thr_nm , os_ru_majflt, os_ru_nivcsw FROM threadstat; +---------+--------+--------+--------------+--------------+ | node_id | thr_no | thr_nm | os_ru_majflt | os_ru_nivcsw | +---------+--------+--------+--------------+--------------+ | 3 | 0 | main | 1 | 541719 | | 4 | 0 | main | 0 | 561769 | +---------+--------+--------+--------------+--------------+ 2 rows in set (0.01 sec) 12 Copyright 2011 Severalnines AB
- 13. Application Design Define the most typical Use Cases List all my friends, session management etc etc. Optimize everything for the typical use case Engineer schema to cater for the Use Cases Keep it simple Complex access patterns does not scale Simple access patterns do ( Primay key and Partitioned Index Scans ) Note! There is no parameter in config.ini that affects performance – only availability. Everything is about the Schema and the Queries. Tune the mysql servers (sort buffers etc) as you would for innodb. 13 Copyright 2011 Severalnines AB
- 14. Simple Access PRIMARY KEY lookups are HASH lookup O(1) INDEX searches a T-tree and takes O(log n) time. In 7.2 and later JOINs are ok, but in 7.1 you should try to avoid them. 14 Copyright 2011 Severalnines AB
- 15. Identifying Bottlenecks A lot of CPU is used on the data nodes Probably a lot of large index scans and full table scans are used. Check Slow query log or a query monitor A lot of CPU is used on the mysql servers Probably a lot of GROUP BY/DISTINCT or aggregate functions. Hardly no CPU is used on either mysql or data nodes Probably low load Time is spent on network (a lot of “ping pong” to satisfy a request). System is running slow in general Disks (io util), queries, swap (must never happen), network
- 16. Need To Add Data Nodes? (adding mysql servers is easy) top –Hd1 Is any of data nodes threads at 100%? Yes: add more data nodes (online) No: do nothing
- 17. Detecting Query Problems Here is a standard method for how to attack the problem. Performance tuning is a never-ending loop: BEGIN – Capture information – e.g, slow query log • Change long_query_time if needed – EXPLAIN the queries • What indexes are used? • Are tables JOINed in the correct order (small to big) – Re-run the optimized typical use cases using bencher/ mysqlslap GOTO BEGIN; END; Never tune unless you can measure and test! Don't optimize unless you have a problem!
- 18. Enable Logging Slow query log set global slow_query_log=1; set global long_query_time=0.01; set global log_queries_not_using_indexes=1; General log (if you don’t get enough info in the Slow Query Log) Activate for a very short period of time (30-60seconds) – intrusive Can fill up disk very fast – make sure you turn it off. set global general_log=1; Use Severalnines ClusterControl Includes a Cluster-wide Query Monitor. Query frequency, EXPLAINs, lock time etc. Performance Monitor and Manager.
- 19. Setup 19 Copyright 2011 Severalnines AB subid data 1 A 3 B 2 C 4 D subscriber Partition 0 Partition 1 NETWORK!
- 20. Sharding By default, all index scans hit all data nodes good if result set is big – you want as many CPUs as possible to help you. For smaller result sets (~a couple of hundred records) Partition Pruning is key for scalability. User-defined partitioning can help to improve equality index scans on part of a primary key. CREATE TABLE t1 (uid, fid, somedata, PRIMARY KEY(uid, fid)) PARTITION BY KEY(userid); All data belonging to a particular uid will be on the same partition. Great locality! select * from user where uid=1; Only one data node will be scanned (no matter how many nodes you have)
- 21. Sharding mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 0 | +-----------------------+-------+ CREATE TABLE t1( … ) PARTITION BY KEY (userid); An run query, and verify it works: select * from user where userid=1; mysql> show global status like 'ndb_pruned_scan_count’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Ndb_pruned_scan_count | 1 | +-----------------------+-------+
- 22. Sharding mysql> show global status like 'ndb%pruned%'; | Ndb_api_table_scan_count | 264 | | Ndb_api_range_scan_count | 18 | | Ndb_api_pruned_scan_count | 3 |
- 23. Sharding - EXAMPLE create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb partition by key(uid); NO PARTITIONING create table users_posts2 ( uid integer , fid integer , pid integer auto_increment, message varchar(1024), primary key(uid,fid, pid) ) engine=ndb; PARTITION BY KEY
- 24. Sharding – EXPLAIN PARTITIONS mysql> explain partitions select * from users_posts u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0,p1 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL With PARTITION BY KEY (UID) mysql> explain partitions select * from users_posts2 u where u.uid=1G id: 1 select_type: SIMPLE table: u partitions: p0 type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: const rows: 2699 Extra: NULL
- 25. Data Types BLOBs/TEXTs vs VARBINARY/VARCHAR BLOB/TEXT columns are stored in an external hidden table. First 256B are stored inline in main table Reading a BLOB/TEXT requires two reads One for reading the Main table + reading from hidden table Change to VARBINARY/VARCHAR if: Your BLOB/TEXTs can fit within an 14000 Bytes record (record size is currently 14000 Bytes) Reading/writing VARCHAR/VARBINARY is less expensive Note 1: BLOB/TEXT are also more expensive in Innodb as BLOB/TEXT data is not inlined with the table. Thus, two disk seeks are needed to read a BLOB. Note 2: Store images, movies etc outside the database on the filesystem.
- 26. Query Tuning MySQL Cluster 7.2 and later has pushed down joins joins are performed in the data nodes. OPTIMIZER in MySQL Cluster 7.1 and earlier is weak Statistics gathering is non-existing Optimizer thinks there are only 10 rows to examine in each table! FORCE INDEX / STRAIGHt_JOIN to get queries run the way you want
- 27. Query Tuning if you have two similar indexes: index(a) index(a,ts) on the following table CREATE TABLE `t1` ( `id` int(11) NOT NULL AUTO_INCREMENT, `a` bigint(20) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (`id`), KEY `idx_t1_a` (`a`), KEY `idx_t1_a_ts` (`a`,`ts`)) ENGINE=ndbcluster DEFAULT CHARSET=latin1
- 28. Query Tuning mysql> select count(id) from t1 where a=5; +-----------+ | count(id) | +-----------+ | 3072000 | +-----------+ 1 row in set (0.02 sec) mysql> select count(id) from t1 where a=5 and ts>'2013-04-18 14:34:08’; +-----------+ | count(id) | +-----------+ | 512 | +-----------+ 1 row in set (0.00 sec)
- 29. Query Tuning Pre 7.2 mysql> explain select * from t1 where a=2 and ts='2011-10-05 15:32:11'; +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ | 1 | SIMPLE | t1 | ref | idx_t1_a,idx_t1_a_ts | idx_t1_a | 9 | const | 10 | Using where | +----+-------------+-------+------+----------------------+----------+---------+-------+------+-------------+ Use FORCE INDEX(..) ... mysql> explain select * from t1 FORCE INDEX (idx_t1_a_ts) where a=2 and ts='2011-10-05 15:32:11; +| 1 | SIMPLE | t1 | ref | idx_t1_a_ts | idx_t1_a_ts | 13 | const,const | 10 | Using where | 1 row in set (0.00 sec) ..to ensure the correct index is picked! The difference can be 1 record read instead of any number of records!
- 30. Index Statistics explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a key_len: 9 ref: const Rows: 17 Extra: Using where with pushed condition
- 31. Index Statistics mysql> analyze table t1; +---------+---------+----------+----------+ | Table | Op | Msg_type | Msg_text | +---------+---------+----------+----------+ | test.t1 | analyze | status | OK | +---------+---------+----------+----------+ 1 row in set (3.40 sec)
- 32. Index Statistics Mysql> explain select * from t1 where a=5 and ts>'2013-04-18 14:34:08' G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: t1 type: range possible_keys: idx_t1_a,idx_t1_a_ts key: idx_t1_a_ts key_len: 13 ref: NULL rows: 253 Extra: Using where with pushed condition; Using MRR 1 row in set (0.00 sec)
- 33. Ndb_cluster_connection_pool Problem: A Sendbuffer on the connection between mysqld and the data nodes is protected by a Mutex. Connection threads in MySQL must acquire Mutex and the put data in SendBuffer. Many threads gives more contention on the mutex Must scale out with many MySQL Servers. Workaround: Ndb_cluster_connection_pool (in my.cnf) creates more connections from one mysqld to the data nodes Threads load balance on the connections gives less contention on mutex which in turn gives increased scalabilty Less MySQL Servers needed to drive load! www.severalnines.com/cluster-configurator allows you to specify the connection pool.
- 34. Ndb_cluster_connection_pool Gives atleast 70% better performance and a MySQL Server that can scale beyond four database connections. Set Ndb_cluster_connection_pool=2x<CPU cores> It is a good starting point One free [mysqld] slot is required in config.ini for each Ndb_cluster_connection. 4 mysql servers,each with Ndb_cluster_connection_pool=8 requires 32 [mysqld] in config.ini Note that also memcached and node.js, cluster/j etc also has the concept of the ndb_cluster_connection_pool.
- 36. Resources MySQL Cluster Configurator www.severalnines.com/config MySQL Cluster Management + Monitoring www.severalnines.com/cmon MySQL Cluster Training Slides www.severalnines.com/mysql-cluster-training My Blog johanandersson.blogspot.com
- 37. Keep in touch… Facebook www.facebook.com/severalnines Twitter @severalnines Linked in: www.linkedin.com/company/severalnines
- 38. Thank you for your time! johan@severalnines.com 38 Copyright 2011 Severalnines AB