InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxDB IOx
•
1 like•436 views
This document discusses the components and architecture of InfluxDB IOx for replication, durability, and subscriptions. It describes the write buffer, how writes are routed and distributed across shards, replication between buffers to ensure durability, and how subscriptions are handled for querying data.
Report
Share
InfluxDB IOx Tech Talks: Replication, Durability and Subscriptions in InfluxDB IOx
- 1. Paul Dix InfluxData – CTO & co-founder paul@influxdata.com @pauldix Replication, durability and subscriptions in InfluxDB IOx
- 2. InfluxDB IOx components • Write Buffer • Write segment persistence • Subscriptions • In-memory database • Parquet persistence • Catalog • Configuration, partition summaries
- 4. Routing Layer • Parse LP • Split into shards • Split into partitions • Write one Entry per shard
- 5. Possible Destinations • Single server or multiple servers • Write Buffer • In-memory DB
- 6. The Write Buffer • Subscriptions • Persistence • Replication
- 7. The Write Buffer • Router to Buffer
- 8. The Write Buffer • Buffer assigns clock value & writer id (unique) • Makes request to peers • Note the request from Router to Buffer still open
- 9. The Write Buffer • Buffer 2 takes the write, updates its clock to max • Buffer 2 responds with its clock value • Buffer 1 updates its clock to the max of the two
- 10. The Write Buffer • Buffer 1 sends success response to routing layer • Routing layer responds to the client (after all shards succeed)
- 11. The Write Buffer • Continues to retry
- 12. The Write Buffer • Consistency checks
- 13. The Write Buffer • Ask peers for anything missing • Logical clock + entry id • High water mark
- 14. The Write Buffer • Persistence to object storage • Entries in segments • Time or size
- 15. Subscriptions • Everything • Table (or regex) • Predicate • Push or Pull • Or poll object storage for segments
- 16. Subscriptions • Scale query tier
- 17. Catalog & Parquet • Persist in larger blocks
- 19. Thank You