Date-tiered Compaction Policy for Time-series Data
•
1 like•1,060 views
Clara Xiong (Flurry/Yahoo!) With petabytes of data on thousands of nodes replicated across multiple data centers, growing at an accelerating rate, we have been running a workload at scale with a bottleneck of IO bandwidth. This talk covers a new compaction policy to improve efficiency for time-range scans of various look-back windows by structuring and maintaining a date-tiered store file layout for time-series data with infrequent updates and deletes.
Report
Share
Date-tiered Compaction Policy for Time-series Data
- 1. Date Tiered Compaction Clara Xiong
- 2. ▪Not a time-series system(row key is independent of time) but similar read/write pattern. ▪Exponential data growth pushes disk IO to the limit. ▪Date tiered compaction partitions the storage by time. ▪Resulted in drastically lower latency and massive reduction of disk IO consumption. Why Date Tiered Compaction for Flurry?
- 3. ▪ Inspired by Cassandra’s Date Tiered Compaction. ▪When new window appears, older windows get merged into exponentially larger windows. What is Date Tiered Compaction? Base window = 1 hour, windows per tier = 4 From https://labs.spotify.com/2014/12/18/date-tiered-compaction/
- 4. ▪Exponential sizing balances the tradeoffs between optimal scan performance of recent data and total file count. ▪Out of range time windows will no longer be merged for efficient data retention management What is Date Tiered Compaction? (cont’d)
- 5. ▪Use maxTimestamp for file selection to favor late-arriving data. ▪Normalize timestamp by the order of sequence id. ▪Plugged-in per- window compaction policy to reduce wasteful compaction. File Selection
- 6. A Date Tiered Layout in Production 0 50 100 150 200 250 300 350 400 450 File Size (20MB) Window (Days)
- 7. ▪HDFS Read Bytes ▪Mapper Run Time Minutes Days After DTC Was On 0.00E+00 2.00E+14 4.00E+14 6.00E+14 8.00E+14 1.00E+15 1.20E+15 1.40E+15 1.60E+15 1.80E+15 2.00E+15 0 2 4 6 8 10 12 7-day job 1-day job 0 10 20 30 40 50 60 70 0 2 4 6 8 10 12 7-day job(avg) 7-day job(max) 1-day job(avg) 1-day job(max)
- 8. Reduction on HDFS IO (%) Notes 1-day look back job 79% 7-day look back job 97% Minor Compaction 85% Still provisional IO Reduction Summary
- 9. ▪ Date Tiered Compactor (Duo Zhang @xiaomi) ● Outputs store files for given time boundaries. ● Allows us to split the data by window boundaries ▪ Date tiered major compaction ● Maintains tiered layout while processing deletes, updates and locality. ● Restores layout skewed by out-of-order writes, bulk load and etc. ▪ Window factory and Minor compaction for archiving (Duo Zhang) Enhancements
- 10. ▪What will benefit most? ●Most writes have cell timestamp near the write time and most reads are for limited recent data out of the full data set ▪What will not? ●Random GET without a time range ●Frequent out of order data writes creating long tails, especially writes with future timestamps ●Frequent bulk load of heavily overlapping time-ranges Is Date Tiered Compaction for You?
- 11. ▪ Don’t turn it on cluster wide: Meta should not be on Date Tiered Compaction. Limit all the related configuration changes by table /column family ▪ Increase blocking file count and max compaction file count settings generously ▪ Put in pressure aware compaction throughput throttler to limit the impact of synchronized large minor compaction Best Practice Recommendations
- 12. ▪ Design specification https://docs.google.com/document/d/1_AmlNb2N8Us1xICsTeGDLK IqL6T-oHoRLZ323MG_uy8 ▪ Email claraxiong@yahoo-inc.com Questions?