Scylla Summit 2017: A Toolbox for Understanding Scylla in the Field

PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Scylla
Performance Toolbox
ScyllaDB
Avi Kivity

AND ON TWO LINES
First and last name
Position, company
Understanding environment
and application impact
on performance
CTO, ScyllaDB
Avi Kivity

AND ON TWO LINES
First and last name
Position, company
Avi Kivity
3
KVM hypervisor author and ex-maintainer
ScyllaDB co-founder and CTO

AND ON TWO LINES
First and last name
Position, company
Agenda
4
▪ Environment
▪ Tracing
▪ Metrics

AND ON TWO LINES
First and last name
Position, company
Environment

AND ON TWO LINES
First and last name
Position, company
Environment
▪ Networking
▪ Disk interrupts
▪ Disk write cache
▪ Virtualization and containers
6

AND ON TWO LINES
First and last name
Position, company
Networking model (multiqueue)
7
NIC
OS/HW
Core Core Core Core Core Core
Rx Queue

AND ON TWO LINES
First and last name
Position, company
Networking model (singlequeue)
8
NIC
OS/HW
Core Core Core Core Core Core
Rx Queue
S/W Rx Queue

AND ON TWO LINES
First and last name
Position, company
Networking model (hybrid)
▪ Each core group is assigned a single hardware queue
▪ One core in core group handles networking
▪ Useful when too few hardware queues
▪ Too difficult to draw
9

AND ON TWO LINES
First and last name
Position, company
How is the networking model configured?
▪ Determined by scylla_setup based on the hardware
▪ Stored in /etc/scylla.d/perftune.yaml
10
$ cat /etc/scylla.d/perftune.yaml
cpu_mask: '0x000000ff'
mode: mq
nic: eth0
tune:
- net

AND ON TWO LINES
First and last name
Position, company
Unbalanced networking
top - 11:40:29 up 3 min, 1 user, load average: 4.48, 4.36, 3.16
Tasks: 152 total, 8 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu0 : 34.3 us, 17.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 6.1 hi, 42.6 si, 0.0 st
KiB Mem : 62882836 total, 61356464 free, 1129072 used, 397300 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 61124456 avail Mem
11

AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back cache
Write-back cache
▪ Scylla writes to disk
▪ Disk places data in DRAM cache, and acknowledges
▪ Disk initiates data write to actual SSD in background
▪ Scylla asks disk to verify that the data made it to non-volatile
storage
▪ Disk waits until background write completes
o Potential stall
12

AND ON TWO LINES
First and last name
Position, company
STALL
Disk write cache - write back
13
Scylla
Disk controller
Media
Write
Media
access
FlushACK
Media
access
complete
ACK

AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back cache
Write-back cache
▪ Scylla writes to disk
▪ Disk places data in DRAM cache, and acknowledges
▪ Disk initiates data write to actual SSD in background
▪ Scylla asks disk to verify that the data made it to non-volatile
storage
▪ Disk does not wait until background write completes
o No stall
14

AND ON TWO LINES
First and last name
Position, company
Disk write cache - write back
15
Scylla
Disk controller
Media
Write
Media
access
Flush
ACK
Media
access
complete
ACK

AND ON TWO LINES
First and last name
Position, company
Beware of iowait
▪ iowait caused by pushing XFS out of its comfort zone
16
top - 11:40:29 up 3 min, 1 user, load average: 4.48, 4.36, 3.16
Tasks: 152 total, 8 running, 151 sleeping, 0 stopped, 0 zombie

AND ON TWO LINES
First and last name
Position, company
Tracing

AND ON TWO LINES
First and last name
Position, company
Types of tracing
▪ Single-shot
▪ Probabilistic
▪ Slow query
18

AND ON TWO LINES
First and last name
Position, company
Single-shot tracing
▪ Useful for gaining an understanding of a query during
development
▪ Issue from cqlsh
19

AND ON TWO LINES
First and last name
Position, company
Probabilistic tracing
▪ Useful to gain an insight about what the application is doing
▪ Controlled by nodetool
▪ Start with very low probability to avoid disturbing the workload
20
$ nodetool settraceprobability 0.000001

AND ON TWO LINES
First and last name
Position, company
Slow-query logging
▪ Catch that long (and slow) tail
▪ Caution: a slow query can interfere with fast queries
21

AND ON TWO LINES
First and last name
Position, company
Metrics

AND ON TWO LINES
First and last name
Position, company
Metrics overview
▪ Aggregated vs. Shard metrics
▪ CPU metrics
▪ I/O metrics
▪ Coordinator-side metrics
▪ Replica-side metrics
23

AND ON TWO LINES
First and last name
Position, company
Zooming into aggregated metrics
▪ Start with cluster-level view
▪ Look at individual nodes
o Cluster runs at speed of slowest node
▪ Look at individual shards
o Node runs at speed of slowest shard
24

AND ON TWO LINES
First and last name
Position, company
CPU metrics
▪ Utilization / load
o For throughput load, should achieve 100%
o If not
• Does one shard reach 100% and the others don’t?
– Hot partition
– Check networking environment
• Sufficient client concurrency?
25

AND ON TWO LINES
First and last name
Position, company
I/O Queue metrics
I/O by type of operation: query, compaction, commitlog
▪ Bandwidth, IOPS (and average size)
▪ Delay
▪ Correlates with iostat command output
26

AND ON TWO LINES
First and last name
Position, company
Coordinator-side metrics
▪ CQL requests per second
▪ CQL connections and their distribution
o High connection open rate?
o Sufficient connections per shard?
o Bad connection distribution?
▪ Statements prepared
o Is the client using prepared statements correctly?
▪ Foreground reads and writes
▪ Background reads and writes
▪ Reconciliation
27

AND ON TWO LINES
First and last name
Position, company
Replica-side metrics
▪ Reads and writes - hot shard, hot node
▪ Cache hits/misses - compare with expectations
▪ Cache total memory - watch for sudden drops
▪ Active SSTable reads - high value indicates weak I/O
▪ Queued SSTable reads - high value indicates weak I/O
▪ Current compactions
28

AND ON TWO LINES
First and last name
Position, company
Summary

AND ON TWO LINES
First and last name
Position, company
Summary
▪ Many moving parts
▪ Despite automation, things can go wrong
▪ Application may get things wrong
▪ Need combination of methodical approach and intuition
▪ Engage the developers so we can improve things
30

AND ON TWO LINES
First and last name
Position, company
THANK YOU
avi@scylladb.com
@AviKivity
Please stay in touch
Any questions?

Scylla Summit 2017: A Toolbox for Understanding Scylla in the Field

Related slideshows

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (9)

Similar to Scylla Summit 2017: A Toolbox for Understanding Scylla in the Field

Similar to Scylla Summit 2017: A Toolbox for Understanding Scylla in the Field (20)

More from ScyllaDB

More from ScyllaDB (20)

Recently uploaded

Recently uploaded (20)

Scylla Summit 2017: A Toolbox for Understanding Scylla in the Field