Apache Hudi

Unlocking Advanced Analytics with AWS Glue and Apache Hudi Discover how you can elevate your data analytics with AWS Glue and Apache Hudi. Ever wondered how to efficiently track and analyze Athena query metrics for enhanced insights? Look no further! Our latest blog post dives deep into setting up Apache Hudi tables to store comprehensive Athena query metrics. 📊 Here’s a sneak peek at the powerful insights you can derive: 🔍 Total Data Scanned by WorkGroup in Athena Queries: Understand resource utilization across different workgroups. 📈 Count of Queries Executed by WorkGroup in Athena: Gauge workload distribution and query frequency. 📊 Athena Query Performance Analysis by WorkGroup: Evaluate average execution times and optimize query performance. 🚀 Performance Analysis of Athena Queries: Dive into detailed metrics like data scanned, execution times, and more. 📋 Database and Table Usage Metrics in Athena Queries: Track usage patterns to optimize database resources. This is just the beginning! Uncover more actionable insights and streamline your analytics workflows. Ready to elevate your data analytics game? Check out our detailed blog for a deep dive into AWS Glue, Apache Hudi, and Athena query metrics. 👉 Read the full blog here Start making data-driven decisions today! 🌟 #AWSGlue #ApacheHudi #DataAnalytics #AthenaQueries #BigDataAnalytics #AWS #DataInsights

📊 Impressive Growth in Data Lakes Powered by Apache Hudi In just one year, our data landscape has seen remarkable growth. From a modest 30TB in July 2023, our data lakes have expanded to a substantial 130TB by July 2024. This represents a staggering accumulation of 100TB of data in just twelve months, underscoring our commitment to handling large-scale data with efficiency and scalability. 🔧 Infrastructure Overview Apache Hudi: The backbone of our data management strategy, ensuring efficient and reliable data ingestion and processing. AWS Glue: Serving as our Hive metastore, crucial for cataloging and organizing data within Hudi tables. AWS Glue Jobs: Powering over 4,000 jobs daily, with an average of 11,000 jobs per month, to manage our Hudi operations seamlessly. Storage: Leveraging S3, the cornerstone of our data storage solution, for its scalability, durability, and cost-effectiveness. Athena: Enabling ad-hoc querying capabilities, providing quick insights into our vast dataset. AWS DMS: Facilitating data ingestion from SQL Server and PostgreSQL, ensuring a smooth flow of data into our Hudi-powered environment. DynamoDB Streams: Integrating real-time data from DynamoDB, enriching our datasets with up-to-date information. 🚀 Future Directions Currently, we are exploring DataHub for enhanced metadata management, aiming to further streamline our data operations and ensure comprehensive governance as our data volumes continue to grow. 🔗 Learn More Curious about how we optimized our job targets and migrated to a serverless architecture with templated Glue jobs? Check out our detailed journey and insights on our blog. https://lnkd.in/eJtP3fH6

Concurrency Control in Lakehouse. Open table formats such as Apache Hudi, Apache Iceberg & Delta Lake support concurrent access to data by multiple transactions. This is one of the most important problem tackled by a lakehouse architecture as opposed to data lakes. To ensure transactional guarantees during multiple reads & writes, there needs to be some mechanism to deal with. Concurrency control defines how different writers/readers coordinate access to a table. Iceberg uses Optimistic concurrency control to deal with concurrent writes. So, transactions operate without initial locks, updating data freely. At commit time, they check for conflicts; if found, conflicting transaction is rolled back. Apache Hudi differentiates itself from other formats by distinctly separating its processes into three categories: - writer processes (handling user upserts & deletes) - table services (managing data and metadata for optimization & bookkeeping) - readers (executing queries) It ensures snapshot isolation among these categories, ensuring each operates on a consistent table snapshot. Hudi uses: ✅ optimistic concurrency control (OCC) among writers ✅ lock-free, non-blocking approach using Multiversion Concurrency Control (MVCC) between writers and table services, as well as among various table services With MVCC, transactions see a consistent past state of the database, allowing read access without blocking, regardless of subsequent data changes. On top of that, with v1.0, Hudi has introduced "Non Blocking Concurrency Control (NBCC)" which allows multiple writers to write simultaneously and conflicts can be resolved later in query or via compaction. NBCC offers the same set of guarantees as OCC but without explicit locks for serializing the writes. Reading link in comments. #dataengineering #softwareengineering

New HFile Reader in Apache Hudi 0.15.0. The metadata table in Hudi hosts various index types containing table metadata. These indexes are stored as "partitions" in the metadata table. Internally, the metadata table is a Hudi Merge-on-Read (MoR) table. There are different indices currently available: - files index: Stored as files partition. Contains file information such as file name, size, etc. - column_stats index: Stored as column_stats partition. Contains the statistics info of columns, such as min and max values, total values, null counts, size, etc., - bloom_filter index: Stored as bloom_filter partition in the metadata table. Stores bloom filters of all data files centrally to avoid scanning the footers directly from individual files. - record_index: Stored as record_index partition in the metadata table. Contains the mapping of the record key to location. Now for efficient lookups & needle-in-a-haystack type queries, compute engines must avoid scanning the entire index, as large datasets can have index sizes in TBs. This is why the 'HFile format' is chosen as the preferred file format for the metadata table. The HFile format is great for point & range lookups and allows us to quickly access the required metadata without needing to scan large portions of the table. So, What's happening? ✅ Apache Hudi 0.15.0 introduces a new "HFile Reader" implemented in Java for accessing metadata table & faster lookups. ✅ This reader is independent of HBase or Hadoop dependencies. ✅ Hudi connector (for engines like Trino, Presto) will use this HFile reader to support reading different indexes from the metadata table. ✅ technically, the readers & writers are language-independent (Rust, C++) as long as it aligns with the defined HFile format spec. 📄 HFile Format Spec: https://lnkd.in/dciGntpz 📙 Release notes: https://lnkd.in/dgUTP6_a 📱Chat about it in our Slack: https://lnkd.in/gZSZzdX4 #dataengineering #softwareengineering

5 Strategies for optimal query performance in a Lakehouse. One aspect that is particularly tricky to master when dealing with large amount of data is "performance". Querying larger data volumes demands certain tuning & optimization. Because your queries with any performant engine might be faster today, but over time it may not. Because over time: - Your query patterns may change - New query patterns may be added - Queries may be slow because of unorganized/small files It is therefore important to be aware of some of the techniques that can help prune & organize data effectively. Because the less data you read, the faster your queries could be. Here are 5 methods to think about for performance in a lakehouse with table formats like Apache Hudi, Apache Iceberg & Delta Lake. ✅ Partitioning: Organize and store related data together when writing to storage. ✅ File Sizing: Combine smaller files into a larger, more optimal size to read less number of files. ✅ Linear Sorting: Arrange data in order based on a unique range of values to improve query efficiency. ✅ Metadata Index: Use statistics from file formats, Bloom filters, etc to enhance data retrieval with structures like indexes (collated to a metadata table). For Iceberg & Delta there are no indexes but file-level stats are aggregated in the manifest file/transaction log. ✅ Z-order/Hilbert curves: Position related data points closely within the storage to enhance read operations involving multiple dimensions. #dataengineering #softwareengineering

Awesome to see what Apache Hudi with #Presto can do 👏

REMINDER 🚨 The next #UnityCatalog Community Meetup is tomorrow, July 11th at 8:30AM PST / 11:30AM EST! Agenda: 🌟 Opening & Welcome 🌟 Apache Hudi/UC demo with Kyle Weller from Onehouse 🌟 Localhost setup with Matthew Powers, CFA from Databricks 🌟 Roadmap 🌟 Q&A, discussion Register here ➡ https://lnkd.in/eAhFjG-r #opensource #linuxfoundation #oss #lfaidata

Amazon Web Services (AWS) team shows how to get started with Apache XTable on AWS and how to use it in a batch pipeline orchestrated with Amazon Managed Workflows for Apache Airflow. This is an amazing hands on blog for anyone looking at orchestrating metadata translation between lakehouse table formats (Apache Hudi, Apache Iceberg & Delta Lake) with XTable. Blog: https://lnkd.in/daGfR9mx Credits: Matthias Rudolph, Stephen said #dataengineering #softwareengineering

Apache Hudi 0.15.0 introduces two new abstractions. These abstractions enhance integration with query engines like Trino, which uses its own native File System APIs. 1. HoodieStorage Abstraction - Hadoop-independent file system & storage APIs for readers/writers - extendible with Hadoop File system and TrinoFileSystem (for Trino-Hudi connector) 2. HoodieIOFactory Abstraction - provides APIs to create readers/writers for I/O without the need to depend on Hadoop class ✅ With this new way, the `hudi-common` module (core implementation of Hudi spec) & core reader logic is Hadoop-independent.  This simplifies integration with engines like Trino, making it easier and more maintainable by allowing custom storage and IO factory implementations to be plugged in. Read details in the release note - https://lnkd.in/dnybdngQ #dataengineering #softwareengineering

The Onehouse-Hudi June edition newsletter is NOW OUT! 🎉 Check out all the amazing stuff that has happened in the Hudi & Data community in the past month. 📰 Newsletter: https://lnkd.in/dPSZ_9wY #dataengineering #softwareengineering