Ceph Tech Talk -- Ceph Benchmarking Tool
•Download as PPTX, PDF•
2 likes•913 views
Ceph Benchmarking Tool (CBT) is a Python framework for benchmarking Ceph clusters. It has client and monitor personalities for generating load and setting up the cluster. CBT includes benchmarks for RADOS operations, librbd, KRBD on EXT4, KVM with RBD volumes, and COSBench tests against RGW. Test plans are defined in YAML files and results are archived for later analysis using tools like awk, grep, and gnuplot.
Report
Share
Ceph Tech Talk -- Ceph Benchmarking Tool
- 1. Ceph Benchmarking Tool (CBT) Kyle BaderCeph Tech Talk May 26, 2016
- 2. INTRO TO CBT
- 3. • Benchmarking framework written in python • Began as a engineering benchmark tool for upstream developlment • Adopted for downstream performance and sizing • Used by many people in Ceph community • Red Hat • Intel / Samsung / SanDisk • Quanta QCT / Supermicro / Dell WHAT IS IT?
- 4. CBT PERSONALITIES HEAD • CBT checkout • Key based authentication to all other hosts • Including itself.. • PDSH packages • Space to store results archives • YAML testplans
- 5. CBT PERSONALITIES CLIENT • Generates load against the SUT • Ceph admin keyring readable by cbt user • Needs loadgen tools installed • FIO • COSbench • Should be a VM for kvmrbdfio • Can be containerized (good for rbdfio)
- 6. CBT PERSONALITIES MON • Nodes to setup monitors on OSD • Nodes to setup OSDs
- 7. • RADOS Bench • FIO with RBD engine • FIO on KRBD on EXT4 • FIO on KVM (vdb) on EXT4 • COSBench for S3/Swift against RGW CBT BENCHMARKS
- 8. • Cluster creation ( optional, use_existing: true ) • Cache tier configuration • Replicated and Erasure coded pools • Collects monitoring information from every node • Collectl – cpu/disk/net/etc. CBT EXTRAS
- 9. • SSH Key on head • Pub key in all hosts authorized_keys (including head) • Ceph packages on all hosts • PDSH packages on all hosts (for pdcp) • Collectl installed on all hosts BASIC SETUP
- 10. • Test network beforehand, bad network easily impairs performance • All-to-All iperf • Check network routes, interfaces • Bonding • Switches should use 5-tuple-hashing for LACP • Nodes should use LACP xmit_hash_policy=layer3+4 TEST METHODOLOGY
- 11. • Use multiple iterations for micro benchmarks • Use client sweeps to establish point of contention / max throughput • Client sweeps should always start with X(1) ~ 1 client • Should have 4-6 different increments of clients • Eg. client1, client[1-2], client[1-3], client[1-4] TEST METHODOLOGY
- 13. CBT CLUSTER CONFIGURATION cluster: head: "ceph@head” clients: ["ceph@client"] osds: ["ceph@osd"] mons: ["ceph@mon"] osds_per_node: 1 fs: xfs mkfs_opts: -f -i size=2048 mount_opts: -o inode64,noatime,logbsize=256k conf_file: /etc/ceph.conf ceph.conf: /etc/ceph/cepf.conf iterations: 3 rebuild_every_test: False tmp_dir: "/tmp/cbt" pool_profiles: replicated: pg_size: 4096 pgp_size: 4096 replication: 'replicated'
- 14. CLIENT SWEEPS cluster: head: "ceph@head” clients: ["ceph@client1"] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, ”ceph@client3”] osds: ["ceph@osd"] mons: ["ceph@mon"] cluster: head: "ceph@head” clients: ["ceph@client1”,”ceph@client2”, “ceph@client3”,”ceph@client4”] osds: ["ceph@osd"] mons: ["ceph@mon"]
- 15. • Spawns RADOS bench processes on each client • Establish raw RADOS throughput • Works against replicated or EC pools RADOS BENCH benchmarks: radosbench: op_size: [ 4194304, 524288, 4096 ] write_only: False time: 300 concurrent_ops: [ 128 ] concurrent_procs: 1 use_existing: True pool_profile: replicated osd_ra: [256]
- 16. • Spawns FIO proccesses on each client • Uses RBD ioengine • Establish raw librbd performance • No VM / container setup required FIO WITH RBD IO ENGINE benchmarks: librbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] procs_per_volume: [ 1 ] volumes_per_client: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100 use_existing_volumes: true
- 17. • Maps KRBD volume to each client • Creates EXT4 filesystem on KRBD • Mounts filesystem • Spawns FIO process per client • Uses AIO IO Engine on filesystem • Client can be container or bare metal • Establishes KRBD performance potential FIO WITH KRBD ON EXT4 benchmarks: rbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
- 18. • Create KVM instances outside CBT • KVM instances listed as clients • Creates EXT4 filesystem on /dev/vdb • Mounts filesystem • Spanws FIO process per client • Uses AIO IO Engine • Establish RBD performance with QEMU IO susbsystems FIO WITH KVM (VDB) ON EXT4 benchmarks: kvmrbdfio: time: 900 vol_size: 65536 mode: [ randwrite, randread, randrw ] rwmixread: 70 op_size: [ 4096, 16384 ] concurrnet_procs: [ 1 ] iodepth: [ 16 ] osd_ra: [ 128 ] cmd_path: '/home/ceph-admin/fio/fio’ pool_profile: 'rbd’ log_avg_msec: 100
- 19. • Install COSBench on head/clients outside CBT • Install / Configure RGW outside CBT • Translates CBT YAML to COSBench XML • Runs COSBench COSBENCH benchmarks: cosbench: cosbench_dir: /root/0.4.1.0 cosbench_xml_dir: /home/ceph-admin/plugin/cbt/conf/cosbench/ controller: client01 auth: config: username=cosbench:operator;password=intel2012;url=… obj_size: [128KB] template: [default] mode: [write] ratio: [100] …. Example at cbt/docs/cosbench.README
- 20. Running CBT # Loop through each test plan for clients in $(seq 1 6);do cbt/cbt –archive=/tmp/${clients}-clients-results path/to/test.yaml done
- 21. ANALYZING DATA
- 22. • No robust tools for analysis • Nested archive directory based on YAML options • Archive/000000/Librbdfio/osd_ra-00000128… • Usually awk/grep/cut-fu to csv • Plot charts with gnplot, Excel, R ANALYZING DATA
- 23. THANK YOU!