SlideShare a Scribd company logo

Fragging Rights: A Tale of a Pathological Storage Workload

•
•
Eric Sproul
Eric Sproul

This document summarizes Eric Sproul's presentation about pathological storage workloads on ZFS. It describes how their workload of small, frequent appends to columns in files caused heavy fragmentation in ZFS's metaslabs over time. This led to degraded performance as allocation times increased. Analysis showed the problem was heavy fragmentation, not disk saturation. Updating the workload to use larger, sequential writes and an in-memory database helped avoid this issue going forward.

Related slideshows

Large scale data-parsing with Hadoop in Bioinformatics
Large scale data-parsing with Hadoop in BioinformaticsLarge scale data-parsing with Hadoop in Bioinformatics
Large scale data-parsing with Hadoop in Bioinformatics
 
SVs hackathon group report
SVs hackathon group reportSVs hackathon group report
SVs hackathon group report
 
FOSDEM 2015: Distributed Tile Processing with GeoTrellis and Spark
FOSDEM 2015: Distributed Tile Processing with GeoTrellis and SparkFOSDEM 2015: Distributed Tile Processing with GeoTrellis and Spark
FOSDEM 2015: Distributed Tile Processing with GeoTrellis and Spark
 
1 of 25
Download to read offline
A Tale of a Pathological Storage Workload Eric Sproul, Circonus ZFS User Conference March 2017 Fragging Rights
Fragging Rights | Eric Sproul | March 17, 2017 Circonus Release Engineer Wearer of other hats as well (ops, sales eng., support) ZFS user since Solaris 10u2 (>10 years ago now??) Helped bring OmniOS to the world @ OmniTI Eric Sproul @eirescot esproul
• How ZFS manages free space • Our workload • Problem manifesta=on • Lessons learned and future direc=on We’ll talk about free space, COW, and the hole we dug for ourselves: Fragging Rights | Eric Sproul | March 17, 2017 ZFS handled it well… unFl it didn’t. A Tale of Surprise and Pain
Fragging Rights | Eric Sproul | March 17, 2017 ZFS tracks free space with space maps Time-ordered log of allocaNons and frees Top-level vdevs divided into a few hundred metaslabs, each with its own space map At metaslab load, the map is read into AVL tree in memory Free Space in ZFS For details, see Jeff Bonwick’s excellent explanaFon of space maps:
 hOp://web.archive.org/web/20080616104156/hOp://blogs.sun.com/bonwick/entry/space_maps
Fragging Rights | Eric Sproul | March 17, 2017 Never overwrite exisNng data. “UpdaNng a file” means new bits wriXen to previously free space, followed by freeing of old chunk. Copy-on-Write
Scalable =me-series data store Compact on-disk format while not sacrificing granularity Enable rich analysis of historical telemetry data Store value plus 6 other pieces of info: • variance of value (stddev) • deriva=ve (change over =me) and stddev of deriva=ve • counter (non-nega=ve deriva=ve) and stddev of counter • count of samples that were averaged to produce value Fragging Rights | Eric Sproul | March 17, 2017 “It seemed like a good idea at the Fme…” The Workload
Fragging Rights | Eric Sproul | March 17, 2017 File format is columnar and offset-based, with 32-byte records. Common case is appending to the end (most recent measurement/rollup). No synchronous semanNcs used for updaNng data files (this is a cluster and we have write-ahead logs). 5x 1m records = 32 * 5 = 160 bytes append 1x 5m rollup = 32 bytes append 1x 3h rollup = 32 bytes append, then update w/ recalculated rollup The Workload
Fragging Rights | Eric Sproul | March 17, 2017 With copy-on-write, every Circonus record append or overwrite modifies a ZFS block, freeing the old copy of that block. ZFS has variable block sizes: minimum is a single disk sector (512b/4K), max is recordsize property (we used 8K). When the tail block reaches 8K, it stops gedng updated and a new tail block starts. The Workload
Fragging Rights | Eric Sproul | March 17, 2017 ZFS sees tail block updated every 5 minutes for: 1m files: 8192/160 = 51 Nmes (~4 hours) 5m files: 8192/32 = 256 Nmes (~21 hours) 3h files: tragic • last record wriXen, then rewriXen 35 more Nmes
 (as 3h rollup value gets recalculated) • 256 records to fill 8K, ZFS sees 256*36 = 9216 block updates (32 days) Alas, we also use compression, so it’s actually ~2x worse than this. The Workload
Fragging Rights | Eric Sproul | March 17, 2017 Aher “some Nme”, depending on ingesNon volume and pool/vdev size, performance degrades swihly. TXGs take longer and longer to process, stalling the ZIO pipeline. ZIO latency bubbles up to userland; applicaNon sees increasing write syscall latency, lowering throughput. Customers are sad. 😔 The Problem
Fragging Rights | Eric Sproul | March 17, 2017 Start with what the app sees: DTrace syscalls (us) syscall: pwrite bytes: 32 value ------------- Distribution ------------- count 4 | 0 8 | 878 16 |@@@@@@@@@@@@ 27051 32 |@@@@@@@@@@@@@@@ 34310 64 |@@@@@@ 13361 128 |@ 2586 256 | 148 512 | 53 1024 | 39 2048 | 33 4096 | 82 8192 | 534 16384 |@@@@ 8614 32768 | 474 65536 | 22 131072 | 0 262144 | 0 524288 | 0 1048576 | 36 2097152 | 335 4194304 | 72 8388608 | 0 Troubleshoo=ng lolwut?!? bad App VFS ZFS (disks) userland kernel syscall
Fragging Rights | Eric Sproul | March 17, 2017 Slow writes must mean disks are saturated, right? extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b data 918.3 2638.5 4644.3 32217.0 1362.7 11.8 386.4 21 40 rpool 0.9 4.4 3.5 22.3 0.0 0.0 2.0 0 0 sd0 0.9 4.4 3.5 22.3 0.0 0.0 0.5 0 0 sd6 67.6 175.1 347.9 2299.8 0.0 0.6 2.6 0 13 sd7 64.4 225.5 335.9 2300.7 0.0 0.7 2.3 0 14 sd8 67.3 167.8 314.5 2300.4 0.0 0.6 2.6 0 13 sd9 65.3 173.8 326.7 2299.8 0.0 0.6 2.6 0 13 sd10 66.1 226.6 332.2 2300.7 0.0 0.6 2.2 0 13 sd11 67.2 153.9 338.8 2301.4 0.0 0.4 2.0 0 11 sd12 69.4 154.6 345.5 2301.4 0.0 0.4 2.0 0 11 sd13 64.0 162.0 321.9 2300.8 0.0 0.4 2.0 0 11 sd14 65.5 163.7 328.7 2300.8 0.0 0.4 2.0 0 11 sd15 64.5 221.4 343.9 2303.5 0.0 0.8 2.7 0 15 sd16 61.1 222.6 318.1 2303.5 0.0 0.8 2.8 0 15 sd17 63.5 211.3 338.1 2303.2 0.0 0.7 2.7 0 14 sd18 63.8 213.0 330.6 2303.2 0.0 0.7 2.6 0 14 sd19 68.7 170.0 321.9 2300.4 0.0 0.6 2.5 0 13 Troubleshoo=ng App VFS ZFS (disks) userland kernel iostat -x
Fragging Rights | Eric Sproul | March 17, 2017 We know the problem is in the write path, but it's not the disks. What Now? TL;DR is that ZFS internals are very complicated and difficult to reason about, especially with legacy ZFS code (OmniOS r151006, circa 2013). We flailed around for a while, using DTrace to generate kernel flame graphs, to get a sense of what the kernel was up to.
Fragging Rights | Eric Sproul | March 17, 2017 Flame graphs! Kernel Stack Profiling System mostly idle (not shown), but when not, it's ZFS. hXps://github.com/brendangregg/FlameGraph
Fragging Rights | Eric Sproul | March 17, 2017 >1s to find free space?!? metaslab_alloc time (nsec) value ------------- Distribution ------------- count 1024 | 0 2048 | 1166 4096 |@@@@@ 41714 8192 |@@@@@@@@@@@@@@@@ 134407 16384 |@@@@@@@@@@@@@ 107140 32768 |@@ 17603 65536 |@ 10066 131072 |@ 10315 262144 |@ 8144 524288 | 3715 1048576 | 1598 2097152 | 581 4194304 | 75 8388608 | 49 16777216 | 0 33554432 | 0 67108864 | 0 134217728 | 0 268435456 | 0 536870912 | 0 1073741824 | 1276 2147483648 | 0 Slab Alloca=on App VFS ZFS (disks) userland kernel metaslab_alloc
• pwrite(2) syscall is slow. • Slow because ZFS takes so long to allocate new space for writes. • We don't know precisely why these alloca=ons are bogged down,
 but it likely involves the nature of the pool's free space. StarNng with applicaNon-perceived latency: Fragging Rights | Eric Sproul | March 17, 2017 What do we know now? Troubleshoo=ng: Recap
Fragging Rights | Eric Sproul | March 17, 2017 Aggregate free space isn't everything Needed to visualize exisNng metaslab allocaNons: Quan=ty vs. Quality # zdb -m data Metaslabs: vdev 0 metaslabs 116 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 38 free 2.35G metaslab 1 offset 100000000 spacemap 153 free 2.19G metaslab 2 offset 200000000 spacemap 156 free 1.70G metaslab 3 offset 300000000 spacemap 158 free 639M metaslab 4 offset 400000000 spacemap 160 free 1.11G
Fragging Rights | Eric Sproul | March 17, 2017 Visualizing Metaslabs Credit for the idea: hOps://www.delphix.com/blog/delphix-engineering/zfs-write-performance-impact-fragmentaFon Color indicates fullness; Green = empty Red = full Percentage is remaining free space.
Fragging Rights | Eric Sproul | March 17, 2017 Visualizing Metaslabs Aher data rewrite, allocaNons are Nghter. Performance problem is gone. Did we just create "ZFS Defrag"?
Fragging Rights | Eric Sproul | March 17, 2017 From one slab, on one vdev, using `zdb -mmm`: Spacemap Detail Metaslabs: vdev 0 metaslabs 116 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 39 offset 9c00000000 spacemap 357 free 13.2G segments 1310739 maxsize 168K freepct 82% [ 0] ALLOC: txg 7900863, pass 1 [ 1] A range: 9c00000000-9c00938800 size: 938800 [ 2] A range: 9c00939000-9c0093d200 size: 004200 ... [4041229] FREE: txg 7974611, pass 1 [4041230] F range: 9d79d10600-9d79d10800 size: 000200 [4041231] F range: 9e84efe400-9e84efe600 size: 000200 [4041232] FREE: txg 7974612, pass 1 [4041233] F range: 9e72ba4600-9e72ba5400 size: 000e00 90th %ile freed size: ~9K 31% of frees are 512b-1K >4M records in one spacemap...
 Costly to load, only to discover you can't allocate from it!
 Probably contributes to those long metaslab_alloc() =mes.
• Spacemap histograms • Visible via zdb (-mm) and mdb (::spa -mh) • Metaslab fragmenta=on metrics • Allocator changes to account for fragmenta=on • New tuning knobs* for write thro_le Since our iniNal foray into this issue, new features have come out: Fragging Rights | Eric Sproul | March 17, 2017 We weren't alone! OpenZFS Improvements * hOp://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/
Fragging Rights | Eric Sproul | March 17, 2017 metaslab 2 offset 400000000 spacemap 227 free 7.99G On-disk histogram: fragmentation 90 9: 632678 **************************************** 10: 198275 ************* 11: 342565 ********************** 12: 460625 ****************************** 13: 213397 ************** 14: 82860 ****** 15: 9774 * 16: 137 * 17: 1 * Spacemap Histogram Key is power-of-2 range size FragmentaNon metric is based on this distribuNon
• Performance is fine un=l some percentage of metaslabs are "spoiled", even though overall pool used space is low. • Once in this state, only solu=on is bulk data rewrite. • Happens sooner if you have fewer/smaller slabs. • Happens sooner if you increase inges=on rate. Our workload is (someNmes) pathologically bad at scale Fragging Rights | Eric Sproul | March 17, 2017 "Doctor, it hurts when I do <this>..."
 "Then don't do that." What We Learned
• Use in-memory DB to accumulate incoming data • Batch-update columnar files with large, sequen=al writes • Eventually replace columnar files with some other DB format Avoid those single-column updates Fragging Rights | Eric Sproul | March 17, 2017 Follow doctor's orders What We're Doing About It
QuesFons? Eric Sproul, Circonus ZFS User Conference March 2017 Thanks for listening! @eirescot

More Related Content

Fragging Rights: A Tale of a Pathological Storage Workload

  • 1. A Tale of a Pathological Storage Workload Eric Sproul, Circonus ZFS User Conference March 2017 Fragging Rights
  • 2. Fragging Rights | Eric Sproul | March 17, 2017 Circonus Release Engineer Wearer of other hats as well (ops, sales eng., support) ZFS user since Solaris 10u2 (>10 years ago now??) Helped bring OmniOS to the world @ OmniTI Eric Sproul @eirescot esproul
  • 3. • How ZFS manages free space • Our workload • Problem manifesta=on • Lessons learned and future direc=on We’ll talk about free space, COW, and the hole we dug for ourselves: Fragging Rights | Eric Sproul | March 17, 2017 ZFS handled it well… unFl it didn’t. A Tale of Surprise and Pain
  • 4. Fragging Rights | Eric Sproul | March 17, 2017 ZFS tracks free space with space maps Time-ordered log of allocaNons and frees Top-level vdevs divided into a few hundred metaslabs, each with its own space map At metaslab load, the map is read into AVL tree in memory Free Space in ZFS For details, see Jeff Bonwick’s excellent explanaFon of space maps:
 hOp://web.archive.org/web/20080616104156/hOp://blogs.sun.com/bonwick/entry/space_maps
  • 5. Fragging Rights | Eric Sproul | March 17, 2017 Never overwrite exisNng data. “UpdaNng a file” means new bits wriXen to previously free space, followed by freeing of old chunk. Copy-on-Write
  • 6. Scalable =me-series data store Compact on-disk format while not sacrificing granularity Enable rich analysis of historical telemetry data Store value plus 6 other pieces of info: • variance of value (stddev) • deriva=ve (change over =me) and stddev of deriva=ve • counter (non-nega=ve deriva=ve) and stddev of counter • count of samples that were averaged to produce value Fragging Rights | Eric Sproul | March 17, 2017 “It seemed like a good idea at the Fme…” The Workload
  • 7. Fragging Rights | Eric Sproul | March 17, 2017 File format is columnar and offset-based, with 32-byte records. Common case is appending to the end (most recent measurement/rollup). No synchronous semanNcs used for updaNng data files (this is a cluster and we have write-ahead logs). 5x 1m records = 32 * 5 = 160 bytes append 1x 5m rollup = 32 bytes append 1x 3h rollup = 32 bytes append, then update w/ recalculated rollup The Workload
  • 8. Fragging Rights | Eric Sproul | March 17, 2017 With copy-on-write, every Circonus record append or overwrite modifies a ZFS block, freeing the old copy of that block. ZFS has variable block sizes: minimum is a single disk sector (512b/4K), max is recordsize property (we used 8K). When the tail block reaches 8K, it stops gedng updated and a new tail block starts. The Workload
  • 9. Fragging Rights | Eric Sproul | March 17, 2017 ZFS sees tail block updated every 5 minutes for: 1m files: 8192/160 = 51 Nmes (~4 hours) 5m files: 8192/32 = 256 Nmes (~21 hours) 3h files: tragic • last record wriXen, then rewriXen 35 more Nmes
 (as 3h rollup value gets recalculated) • 256 records to fill 8K, ZFS sees 256*36 = 9216 block updates (32 days) Alas, we also use compression, so it’s actually ~2x worse than this. The Workload
  • 10. Fragging Rights | Eric Sproul | March 17, 2017 Aher “some Nme”, depending on ingesNon volume and pool/vdev size, performance degrades swihly. TXGs take longer and longer to process, stalling the ZIO pipeline. ZIO latency bubbles up to userland; applicaNon sees increasing write syscall latency, lowering throughput. Customers are sad. 😔 The Problem
  • 11. Fragging Rights | Eric Sproul | March 17, 2017 Start with what the app sees: DTrace syscalls (us) syscall: pwrite bytes: 32 value ------------- Distribution ------------- count 4 | 0 8 | 878 16 |@@@@@@@@@@@@ 27051 32 |@@@@@@@@@@@@@@@ 34310 64 |@@@@@@ 13361 128 |@ 2586 256 | 148 512 | 53 1024 | 39 2048 | 33 4096 | 82 8192 | 534 16384 |@@@@ 8614 32768 | 474 65536 | 22 131072 | 0 262144 | 0 524288 | 0 1048576 | 36 2097152 | 335 4194304 | 72 8388608 | 0 Troubleshoo=ng lolwut?!? bad App VFS ZFS (disks) userland kernel syscall
  • 12. Fragging Rights | Eric Sproul | March 17, 2017 Slow writes must mean disks are saturated, right? extended device statistics device r/s w/s kr/s kw/s wait actv svc_t %w %b data 918.3 2638.5 4644.3 32217.0 1362.7 11.8 386.4 21 40 rpool 0.9 4.4 3.5 22.3 0.0 0.0 2.0 0 0 sd0 0.9 4.4 3.5 22.3 0.0 0.0 0.5 0 0 sd6 67.6 175.1 347.9 2299.8 0.0 0.6 2.6 0 13 sd7 64.4 225.5 335.9 2300.7 0.0 0.7 2.3 0 14 sd8 67.3 167.8 314.5 2300.4 0.0 0.6 2.6 0 13 sd9 65.3 173.8 326.7 2299.8 0.0 0.6 2.6 0 13 sd10 66.1 226.6 332.2 2300.7 0.0 0.6 2.2 0 13 sd11 67.2 153.9 338.8 2301.4 0.0 0.4 2.0 0 11 sd12 69.4 154.6 345.5 2301.4 0.0 0.4 2.0 0 11 sd13 64.0 162.0 321.9 2300.8 0.0 0.4 2.0 0 11 sd14 65.5 163.7 328.7 2300.8 0.0 0.4 2.0 0 11 sd15 64.5 221.4 343.9 2303.5 0.0 0.8 2.7 0 15 sd16 61.1 222.6 318.1 2303.5 0.0 0.8 2.8 0 15 sd17 63.5 211.3 338.1 2303.2 0.0 0.7 2.7 0 14 sd18 63.8 213.0 330.6 2303.2 0.0 0.7 2.6 0 14 sd19 68.7 170.0 321.9 2300.4 0.0 0.6 2.5 0 13 Troubleshoo=ng App VFS ZFS (disks) userland kernel iostat -x
  • 13. Fragging Rights | Eric Sproul | March 17, 2017 We know the problem is in the write path, but it's not the disks. What Now? TL;DR is that ZFS internals are very complicated and difficult to reason about, especially with legacy ZFS code (OmniOS r151006, circa 2013). We flailed around for a while, using DTrace to generate kernel flame graphs, to get a sense of what the kernel was up to.
  • 14. Fragging Rights | Eric Sproul | March 17, 2017 Flame graphs! Kernel Stack Profiling System mostly idle (not shown), but when not, it's ZFS. hXps://github.com/brendangregg/FlameGraph
  • 15. Fragging Rights | Eric Sproul | March 17, 2017 >1s to find free space?!? metaslab_alloc time (nsec) value ------------- Distribution ------------- count 1024 | 0 2048 | 1166 4096 |@@@@@ 41714 8192 |@@@@@@@@@@@@@@@@ 134407 16384 |@@@@@@@@@@@@@ 107140 32768 |@@ 17603 65536 |@ 10066 131072 |@ 10315 262144 |@ 8144 524288 | 3715 1048576 | 1598 2097152 | 581 4194304 | 75 8388608 | 49 16777216 | 0 33554432 | 0 67108864 | 0 134217728 | 0 268435456 | 0 536870912 | 0 1073741824 | 1276 2147483648 | 0 Slab Alloca=on App VFS ZFS (disks) userland kernel metaslab_alloc
  • 16. • pwrite(2) syscall is slow. • Slow because ZFS takes so long to allocate new space for writes. • We don't know precisely why these alloca=ons are bogged down,
 but it likely involves the nature of the pool's free space. StarNng with applicaNon-perceived latency: Fragging Rights | Eric Sproul | March 17, 2017 What do we know now? Troubleshoo=ng: Recap
  • 17. Fragging Rights | Eric Sproul | March 17, 2017 Aggregate free space isn't everything Needed to visualize exisNng metaslab allocaNons: Quan=ty vs. Quality # zdb -m data Metaslabs: vdev 0 metaslabs 116 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 38 free 2.35G metaslab 1 offset 100000000 spacemap 153 free 2.19G metaslab 2 offset 200000000 spacemap 156 free 1.70G metaslab 3 offset 300000000 spacemap 158 free 639M metaslab 4 offset 400000000 spacemap 160 free 1.11G
  • 18. Fragging Rights | Eric Sproul | March 17, 2017 Visualizing Metaslabs Credit for the idea: hOps://www.delphix.com/blog/delphix-engineering/zfs-write-performance-impact-fragmentaFon Color indicates fullness; Green = empty Red = full Percentage is remaining free space.
  • 19. Fragging Rights | Eric Sproul | March 17, 2017 Visualizing Metaslabs Aher data rewrite, allocaNons are Nghter. Performance problem is gone. Did we just create "ZFS Defrag"?
  • 20. Fragging Rights | Eric Sproul | March 17, 2017 From one slab, on one vdev, using `zdb -mmm`: Spacemap Detail Metaslabs: vdev 0 metaslabs 116 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 39 offset 9c00000000 spacemap 357 free 13.2G segments 1310739 maxsize 168K freepct 82% [ 0] ALLOC: txg 7900863, pass 1 [ 1] A range: 9c00000000-9c00938800 size: 938800 [ 2] A range: 9c00939000-9c0093d200 size: 004200 ... [4041229] FREE: txg 7974611, pass 1 [4041230] F range: 9d79d10600-9d79d10800 size: 000200 [4041231] F range: 9e84efe400-9e84efe600 size: 000200 [4041232] FREE: txg 7974612, pass 1 [4041233] F range: 9e72ba4600-9e72ba5400 size: 000e00 90th %ile freed size: ~9K 31% of frees are 512b-1K >4M records in one spacemap...
 Costly to load, only to discover you can't allocate from it!
 Probably contributes to those long metaslab_alloc() =mes.
  • 21. • Spacemap histograms • Visible via zdb (-mm) and mdb (::spa -mh) • Metaslab fragmenta=on metrics • Allocator changes to account for fragmenta=on • New tuning knobs* for write thro_le Since our iniNal foray into this issue, new features have come out: Fragging Rights | Eric Sproul | March 17, 2017 We weren't alone! OpenZFS Improvements * hOp://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/
  • 22. Fragging Rights | Eric Sproul | March 17, 2017 metaslab 2 offset 400000000 spacemap 227 free 7.99G On-disk histogram: fragmentation 90 9: 632678 **************************************** 10: 198275 ************* 11: 342565 ********************** 12: 460625 ****************************** 13: 213397 ************** 14: 82860 ****** 15: 9774 * 16: 137 * 17: 1 * Spacemap Histogram Key is power-of-2 range size FragmentaNon metric is based on this distribuNon
  • 23. • Performance is fine un=l some percentage of metaslabs are "spoiled", even though overall pool used space is low. • Once in this state, only solu=on is bulk data rewrite. • Happens sooner if you have fewer/smaller slabs. • Happens sooner if you increase inges=on rate. Our workload is (someNmes) pathologically bad at scale Fragging Rights | Eric Sproul | March 17, 2017 "Doctor, it hurts when I do <this>..."
 "Then don't do that." What We Learned
  • 24. • Use in-memory DB to accumulate incoming data • Batch-update columnar files with large, sequen=al writes • Eventually replace columnar files with some other DB format Avoid those single-column updates Fragging Rights | Eric Sproul | March 17, 2017 Follow doctor's orders What We're Doing About It
  • 25. QuesFons? Eric Sproul, Circonus ZFS User Conference March 2017 Thanks for listening! @eirescot