Alluxio
•
0 likes•374 views
This document discusses Alluxio, an open-source memory-centric distributed storage system that serves data from memory, SSDs, and HDDs to analytics and AI/ML frameworks. It provides a unified namespace and supports various storage systems. The architecture includes a master node for metadata and workflow management and worker nodes. Resiliency is enabled through an active-passive master configuration and lineage-based recovery of lost data. The system has been used in production with over 100 nodes managing 1+ PB of data and providing 30x performance improvements.
Report
Share
Alluxio
- 3. Berkeley Data Analytics Stack
- 4. Supported Storages and Framework
- 5. Memory-Centric distributed storage system Storage (hdfs,s3,...) AlluxioBlock 1 Block 2 Block 3 Spark Memory Spark Job A Block 1 Block 2 Spark Memory Spark Job B Block 1 Block 3
- 6. Architecture ● Metadata ● Workflow ManagerMaster Worker Worker Worker
- 7. Tiered Storage Master Worker Worker Worker Memory SSD HDD
- 8. Unified and Transparent Namespace
- 9. Resiliency: Master Master Worker Worker Master Worker Active Passive Write Read Journal
- 10. Resiliency: Lineage File Set A File Set C Spark Job File Set B File Set D Spark Job MapReduce Job File Set E X
- 11. Code
- 12. Bigger Case ● Mem+HDD ● 100+ nodes ● 1 PB+ managed space ● 30x Perf improvement
- 13. Where is the code?
- 14. Thanks!