Rebuilding Web Tracking Infrastructure for Scale

Modern data ecosystems require new paradigms to address diverse data sources and user needs. Traditional assumptions about data originating from internal systems and a single data warehouse no longer apply. A new model called "Data Regions" establishes multiple environments for different data usage scenarios, including source onboarding, exploration, reporting, analytics and more. By supporting varied access, structures, domains and integrity across regions, Data Regions can address today's complex data challenges and modernize companies' data ecosystems.

Learn how Pure Storage engineering manages streaming 190B log events per day and makes use of that deluge of data in our continuous integration (CI) pipeline. Our test infrastructure runs over 70,000 tests per day creating a large triage problem that would require at least 20 triage engineers. Instead, Spark's flexible computing platform allows us to write a single application for both streaming and batch jobs to understand the state of our CI pipeline for our team of 3 triage engineers. Using encoded patterns, Spark indexes log data for real-time reporting (Streaming), uses Machine Learning for performance modeling and prediction (Batch job), and finds previous matches for newly encoded patterns (Batch job). Resource allocation in this mixed environment can be challenging; a containerized Spark cluster deployment, and disaggregated compute and storage layers allow us to programmatically shift compute resources between the streaming and batch applications.. This talk will go over design decisions to meet SLAs of streaming and batching in hardware, data layout, access patterns, and containers strategy. We will also go over the challenges, lessons learned, and best practices for similar data pipelines. Speaker Joshua Robinson, JOSHUA ROBINSON Founding Engineer Pure Storage

My talk on building an EDW from Apache Hive, Ranger, Atlas, and other Apache projects. This talk was initially delivered at Dataworks Summit EU 2017

Introduction to Apache NiFi 1.10 Parameters, Stateless, RetryFlowFile, Backpressure prediction, parquetreader, parquetwriter, postslack, remote inputport in process group. Dec 2019, Timothy Spann, Field Engineer, Data in Motion Princeton Meetup 10-dec-2019 https://www.meetup.com/futureofdata-princeton/events/266496424/ Hosted By PGA Fund at: https://pga.fund/coworking-space/ Princeton Growth Accelerator 5 Independence Way, 4th Floor, Princeton, NJ

1) The document discusses using Apache MXNet for industrial IoT applications. MiniFi ingests camera images and sensor data at the edge and runs Apache MXNet to recognize objects in images. The data is then stored in Hadoop. 2) It describes using Apache MXNet on edge devices like the Raspberry Pi and Nvidia Jetson TX1 to perform tasks like image recognition from cameras and sensors. 3) The document provides information on setting up Apache MXNet on various IoT devices and edge servers to enable machine learning and deep learning capabilities for industrial IoT applications.

Danny Chen presented on Uber's use of HBase for global indexing to support large-scale data ingestion. Uber uses HBase to provide a global view of datasets ingested from Kafka and other data sources. To generate indexes, Spark jobs are used to transform data into HFiles, which are loaded into HBase tables. Given the large volumes of data, techniques like throttling HBase access and explicit serialization are used. The global indexing solution supports requirements for high throughput, strong consistency and horizontal scalability across Uber's data lake.

The document discusses Hive LLAP (Live Long and Process) as a high performance and cost-effective alternative to traditional Massively Parallel Processing (MPP) databases for querying large datasets on Hadoop. It describes Walmart's implementation of Hive LLAP on their data lake to improve query performance for business users. A proof-of-concept found Hive LLAP queries were up to 50% faster when using 15 nodes instead of 10, and it performed comparably or better than two MPP databases with similar or larger infrastructures. Walmart plans to further evaluate Hive LLAP on newer Hadoop distributions and technologies to improve availability and workload management.

This document discusses using Pivotal's Big Data Suite to build a real-time analytics solution for processing taxi trip data streams. It presents an architecture that uses Spring XD for data ingestion, Spark Streaming for in-memory analytics on 10-second windows, Gemfire for fast data retrieval, and Pivotal HD for long-term storage. The solution demonstrates filtering inconsistent data, finding top traffic areas, and available taxis in real-time. The document highlights how the Big Data Suite provides a complete toolset for data-driven enterprises through its optimized Hadoop distribution, in-memory processing, stream processing, and low-latency data stores.

The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the Big Data ecosystem together; it enables users to "run-anything-anywhere". This talk will briefly cover the capabilities of the Beam model for data processing, as well as the current state of the Beam ecosystem. We'll discuss Beam architecture and dive into the portability layer. We'll offer a technical analysis of the Beam's powerful primitive operations that enable true and reliable portability across diverse environments. Finally, we'll demonstrate a complex pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Spark on Amazon Web Services, Apache Flink on Google Cloud, Apache Apex on-premise), and give a glimpse at some of the challenges Beam aims to address in the future.

Companies who use Kafka today struggle with monitoring and managing Kafka clusters. Kafka is a key backbone of IoT streaming analytics applications. The challenge is understanding what is going on overall in the Kafka cluster including performance, issues and message flows. No open source tool caters to the needs of different users that work with Kafka: DevOps/developers, platform team, and security/governance teams. See how the new Hortonworks Streams Messaging Manager enables users to visualize their entire Kafka environment end-to-end and simplifies Kafka operations. In this session learn how SMM visualizes the intricate details of how Apache Kafka functions in real time while simultaneously surfacing every nuance of tuning, optimizing, and measuring input and output. SMM will assist users to quickly understand and operate Kafka while providing the much-needed transparency that sophisticated and experienced users need to avoid all the pitfalls of running a Kafka cluster. Speaker: Andrew Psaltis, Principal Solution Engineer, Hortonworks

Apache Hadoop YARN is the resource and application manager for Apache Hadoop. In the past, YARN only supported launching containers as processes. However, as containerization has become extremely popular, more and more users wanted support for launching Docker containers. With recent changes, YARN now supports running Docker containers alongside process containers. Coupled with the newly added support for long-running services on YARN, this allows a host of new possibilities. In this talk, we'll present how to run a container cloud on YARN. Leveraging the support in YARN for Docker and long-running services, we can allow users to easily spin up sets of Docker containers for their applications. These containers can be self contained or wired up to form more complex applications. We will go over some of the lessons we learned as part of our experiences handling issues such as resource management, debugging application failures, running Docker, service discovery, etc. Speaker Billie Rinaldi, Principal Software Engineer I, Hortonworks

In my talk I will discuss and show examples of using Apache Hadoop, Apache Hive, Apache MXNet, Apache OpenNLP, Apache NiFi and Apache Spark for deep learning applications. As part of my talk I will walk through using Apache NXNet Pre-Built Models, MXNet's New Model Server with Apache NiFi, executing MXNet with Apache NiFi and running Apache MXNet on edge nodes utilizing Python and Apache MiniFi. This talk is geared towards Data Engineers interested in the basics of Deep Learning with open source Apache tools in a Big Data environment. I will walk through source code examples available in github and run the code live on an Apache Hadoop / YARN / Apache Spark cluster. This will be an introduction to executing Deep Learning Pipelines in an Apache Big Data environment. My talk at Data Works Summit Sydney was listed in top 7 -> https://hortonworks.com/blog/7-sessions-dataworks-summit-sydney-see/ Also have speak at and run Future of Data Princeton and at Oracle Code NYC. Ref: https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html https://community.hortonworks.com/articles/146704/edge-analytics-with-nvidia-jetson-tx1-running-apac.html https://dzone.com/refcardz/introduction-to-tensorflow Speaker Timothy Spann, Solutions Engineer, Hortonworks

MiNiFi is a recently started sub-project of Apache NiFi that is a complementary data collection approach which supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation. Simply, MiNiFi agents take the guiding principles of NiFi and pushes them to the edge in a purpose built design and deploy manner. This talk will focus on MiNiFi's features, go over recent developments and prospective plans, and give a live demo of MiNiFi. The config.yml is available here: https://gist.github.com/JPercivall/f337b8abdc9019cab5ff06cb7f6ff09a

This document discusses ways to troubleshoot slow Hadoop jobs using metrics, logging, and tracing. It describes how to use the Ambari metrics system and Grafana dashboards to monitor metrics for clusters. It also explains how to leverage Hadoop logs and the YARN Application Timeline Service for logging and correlation across workloads. Finally, it presents Apache Zeppelin and analyzers for Hive, Tez, and YARN as tools for ad-hoc analysis to diagnose issues.

The document discusses the past, present, and future of Apache Hadoop YARN. It describes how YARN started as a sub-project of Hadoop to improve its resource management capabilities. Today, YARN is central to modern data architectures, providing centralized resource management and scheduling. Going forward, YARN aims to better support containers, simplified APIs, treating services as first-class citizens, and enhance its user experience.

This document discusses streaming data ingestion and processing options. It provides an overview of common streaming architectures including Kafka as an ingestion hub and various streaming engines. Spark Streaming is highlighted as a popular and full-featured option for processing streaming data due to its support for SQL, machine learning, and ease of transition from batch workflows. The document also briefly profiles StreamSets Data Collector as a higher-level tool for building streaming data pipelines.

The document discusses Apache Hive and Apache Druid for fast SQL on big data. It provides performance benchmarks showing Hive LLAP is faster than Presto and Spark SQL for TPC-DS queries. It describes features of Hive LLAP including in-memory caching, query result caching, and metadata caching. It also discusses new Hive 3 features like materialized views and optimizer improvements. The document then provides an overview of Apache Druid's capabilities for real-time ingestion and querying of streaming data before discussing how Hive and Druid can work together, with Hive able to push down queries to Druid.

The truth about SQL and Data Warehousing on Hadoop