SlideShare a Scribd company logo

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

•
•
Cloudera, Inc.
Cloudera, Inc.

This session will provide an executive overview of the Apache Hadoop ecosystem, its basic concepts, and its real-world applications. Attendees will learn how organizations worldwide are using the latest tools and strategies to harness their enterprise information to solve business problems and the types of data analysis commonly powered by Hadoop. Learn how various projects make up the Apache Hadoop ecosystem and the role each plays to improve data storage, management, interaction, and analysis. This is a valuable opportunity to gain insights into Hadoop functionality and how it can be applied to address compelling business challenges in your agency.

1 of 36
1© Cloudera, Inc. All rights reserved. 5th Annual
2© Cloudera, Inc. All rights reserved. The Essentials of Apache Hadoop The What, Why and How to Meet Agency Objectives Sarah Sproehnle, Vice President, Customer Success
3© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Introduction
4© Cloudera, Inc. All rights reserved. What is Apache Hadoop? • Hadoop is a software framework for storing, processing, and analyzing “big data” • Distributed • Scalable • Fault-tolerant • Open source
5© Cloudera, Inc. All rights reserved. A Large (and Growing) Ecosystem Impala
6© Cloudera, Inc. All rights reserved. About Cloudera • The leader in Apache Hadoop-based software and services • Founded in 2008 by leading experts on Hadoop • Over 1000 employees • Global operations spanning over 20 countries • Provides support, consulting, training, and certification for Hadoop users • Employs committers to virtually every significant Hadoop-related project • Many authors of industry standard books on Apache Hadoop projects • Tom White, Lars George, Kathleen Ting, etc.
7© Cloudera, Inc. All rights reserved. • CDH (Cloudera’s Distribution, including Apache Hadoop) • 100% open source, enterprise-ready distribution of Hadoop and related projects • The most complete, tested, and widely-deployed distribution of Hadoop • Integrates all the key Hadoop ecosystem projects CDH
8© Cloudera, Inc. All rights reserved. Vendor Integration Platform & Cloud System Integration Data Systems Software and OEM
9© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. How it Works
10© Cloudera, Inc. All rights reserved. Traditional Large-Scale Computation Traditionally, computation has been processor-bound • Relatively small amounts of data • Lots of complex processing The early solution: bigger computers • Faster processor, more memory • But even this couldn’t keep up
11© Cloudera, Inc. All rights reserved. Distributed Systems The better solution: more computers • Distributed systems – use multiple machines for a single job “In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, we didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers.” – Grace Hopper
12© Cloudera, Inc. All rights reserved. Distributed Systems: The Data Bottleneck (1) • Traditionally, data is stored in a central location • Data is copied to processors at runtime • Fine for limited amounts of data
13© Cloudera, Inc. All rights reserved. Distributed Systems: The Data Bottleneck (2) • Modern systems have much more data • terabytes+ a day • petabytes+ total • We need a new approach…
14© Cloudera, Inc. All rights reserved. The Origins of Hadoop • Hadoop is based on work done at Google in the late 1990s/early 2000s • Google’s problem: • Indexing the entire web requires massive amounts of storage • A new approach was required to process such large amounts of data • Google’s solution: • GFS, the Google File System - described in a paper released in 2003 • Distributed MapReduce - described in a paper released in 2004 • Doug Cutting and others read these papers and implemented a similar, open source solution • This is what would become Hadoop
15© Cloudera, Inc. All rights reserved. What is Hadoop? • Hadoop is a distributed data storage and processing platform • Stores massive amounts of data in a very resilient way • Distributes the processing to where the data is stored • Tools built around Hadoop (the ‘Hadoop ecosystem’) can be configured/extended to handle many different tasks • Extract Transform Load (ETL) • BI environment • Predictive analytics • Statistical analysis • Machine learning
16© Cloudera, Inc. All rights reserved. Hadoop is Scalable • Adding nodes (machines) adds capacity proportionally • Increasing load results in a graceful decline in performance • Not failure of the system Number of Nodes Capacity
17© Cloudera, Inc. All rights reserved. Hadoop is Fault Tolerant • Node failure is inevitable • What happens? • System continues to function • Master re-assigns work to a different node • Data replication means there is no loss of data • Nodes which recover rejoin the cluster automatically
18© Cloudera, Inc. All rights reserved. The Hadoop Ecosystem (1) • Many tools have been developed around ‘Core Hadoop’ • Known as the Hadoop ecosystem • Designed to make Hadoop easier to use, or to extend its functionality • All are open source • The ecosystem is growing all the time
19© Cloudera, Inc. All rights reserved. The Hadoop Ecosystem (2) • Examples of Hadoop ecosystem projects (all included in CDH): Project What does it do? Spark In-memory and streaming processing framework HBase NoSQL database built on HDFS Hive SQL processing engine designed for batch workloads Impala SQL query engine designed for BI workloads Parquet Very efficient columnar data storage format Sqoop Data movement to/from RDBMSs Flume, Kafka Streaming data ingestion Solr Powerful text search functionality Hue Web-based user interface for Hadoop Sentry Authorization tool, providing security for Hadoop
20© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Hadoop in the Real World
21© Cloudera, Inc. All rights reserved. Five Really Popular Use Cases • ETL Processing • Business Intelligence • Predictive Analytics • Enterprise Data Hub • Low-cost Storage of Large Data Volumes
22© Cloudera, Inc. All rights reserved. • ETL: Extract, Transform, Load • Challenges: • Too much data • Takes too long • Too costly # 1 – Traditional ETL Processing Networked Storage ETL Grid Data Warehouse Transactional Systems Data Stream BI Tools Networked Storage Tape Storage Tape Storage
23© Cloudera, Inc. All rights reserved. # 1 – ETL Processing with Hadoop • Hadoop cluster is used for ETL • Often now ELT: Extract, Load, then Transform • Structured and unstructured data is moved into the cluster • Once processed, data can be analyzed in Hadoop or moved to the EDW Networked Storage Data Stream Enterprise Data Warehouse ETL Analytics Real-time
24© Cloudera, Inc. All rights reserved. # 1 – Vendor Integration - ETL • For more information visit: http://www.cloudera.com/partners/partners- listing.html
25© Cloudera, Inc. All rights reserved. # 2 – Traditional Business Intelligence • BI traditionally takes place at the data warehouse layer • Problem: EDW can’t keep up with growing data volumes • Performance declines • Increasing capacity can be very expensive • Archived data is not available for analysis Networked Storage ETL Grid Data Warehouse Transactional Systems Data Stream BI Tools Tape Storage Tape Storage
26© Cloudera, Inc. All rights reserved. # 2 – Business Intelligence with Hadoop • BI tools can use Hadoop for much of their work • Analyze all the data • Use the EDW for the tasks for which it is best suited Networked Storage Data Stream Enterprise Data Warehouse ETL Analytics Real-time BI Tools
27© Cloudera, Inc. All rights reserved. # 2 – Vendor Integration - BI • For more information visit: http://www.cloudera.com/partners/partners- listing.html
28© Cloudera, Inc. All rights reserved. # 3 – Predictive Analytics • Predictive Analytics (Eckerson Group definition) • The use of statistical or machine learning models to discover patterns and relationships in data that can help business people predict future behavior or activity • The Hadoop platform can run analytic workloads on large volumes of diverse data • Statistical models can be created and run inside the Hadoop environment • Entire data sets can be used to create models • There is no need to sample data • Hadoop provides an environment that makes self-service analytics possible • No need for ETL developers to stage data for data scientists
29© Cloudera, Inc. All rights reserved. # 3 - Predictive Analytics: Cerner Corporation Cerner Corporation • Healthcare IT space • Solutions and Services - Used by 54,000 medical facilities around the world The problem • Healthcare data is fragmented and lives in silos • The data was used for historical reporting The solution • Build a comprehensive view of population health using a single platform • Use predictive analytics to • Improve patient outcomes • Increase efficiency / Reduce costs
31© Cloudera, Inc. All rights reserved. # 3 – Vendor Integration – Predictive Analytics • For more information visit: http://www.cloudera.com/partners/partners- listing.html
32© Cloudera, Inc. All rights reserved. # 4 – The Need for the Enterprise Data Hub Thousands of Employees & Lots of Inaccessible Information Heterogeneous Legacy IT Infrastructure Silos of Multi- Structured Data Difficult to Integrate ERP, CRM, RDBMS, Machines Files, Images, Video, Logs, Clickstreams External Data Sources Data Archives EDWs Marts SearchServers Document Stores Storage
33© Cloudera, Inc. All rights reserved. # 4 – The Enterprise Data Hub: One Unified System Information & data accessible by all for insight using leading tools and apps Enterprise Data Hub Unified Data Management Infrastructure Ingest All Data Any Type Any Scale From Any Source EDWs Marts Storage SearchServers Documents ERP, CRM, RDBMS, Machines Files, Images, Video, Logs, Clickstreams External Data Sources EDH Archives
34© Cloudera, Inc. All rights reserved. # 5 - Low-cost Data Storage (1) • Hadoop combines industry standard hardware and a fault tolerant architecture • This combination provides a very cost effective data storage platform • The data stored on Hadoop is protected from loss by HDFS • Data replication ensures that no data is lost • The self-healing nature of Hadoop ensures that data is available when you need it • Hadoop enables users to store data which was previously discarded due to the cost of saving it • Transactional • Social media • Sensor • Click stream
35© Cloudera, Inc. All rights reserved. # 5 - Low-cost Data Storage (2) • The low cost of HDFS storage enables the following use-cases: • Enterprise Data Hub (EDH) • Active data archive • Staging area for data warehouses • Staging area for analytics store • Sandbox for data discovery • Sandbox for analytics
36© Cloudera, Inc. All rights reserved. In Summary, Why Do You Need Hadoop? • More data is coming • Internet of things • Sensor data • Streaming • More data means bigger questions, better answers • Hadoop easily scales to store and handle all of your data • Hadoop is cost-effective • Provides a significant cost-per-terabyte saving over traditional, legacy systems • Hadoop integrates with your existing datacenter components • Answer questions that you previously could not ask
37© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Thank You

More Related Content

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

  • 1. 1© Cloudera, Inc. All rights reserved. 5th Annual
  • 2. 2© Cloudera, Inc. All rights reserved. The Essentials of Apache Hadoop The What, Why and How to Meet Agency Objectives Sarah Sproehnle, Vice President, Customer Success
  • 3. 3© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Introduction
  • 4. 4© Cloudera, Inc. All rights reserved. What is Apache Hadoop? • Hadoop is a software framework for storing, processing, and analyzing “big data” • Distributed • Scalable • Fault-tolerant • Open source
  • 5. 5© Cloudera, Inc. All rights reserved. A Large (and Growing) Ecosystem Impala
  • 6. 6© Cloudera, Inc. All rights reserved. About Cloudera • The leader in Apache Hadoop-based software and services • Founded in 2008 by leading experts on Hadoop • Over 1000 employees • Global operations spanning over 20 countries • Provides support, consulting, training, and certification for Hadoop users • Employs committers to virtually every significant Hadoop-related project • Many authors of industry standard books on Apache Hadoop projects • Tom White, Lars George, Kathleen Ting, etc.
  • 7. 7© Cloudera, Inc. All rights reserved. • CDH (Cloudera’s Distribution, including Apache Hadoop) • 100% open source, enterprise-ready distribution of Hadoop and related projects • The most complete, tested, and widely-deployed distribution of Hadoop • Integrates all the key Hadoop ecosystem projects CDH
  • 8. 8© Cloudera, Inc. All rights reserved. Vendor Integration Platform & Cloud System Integration Data Systems Software and OEM
  • 9. 9© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. How it Works
  • 10. 10© Cloudera, Inc. All rights reserved. Traditional Large-Scale Computation Traditionally, computation has been processor-bound • Relatively small amounts of data • Lots of complex processing The early solution: bigger computers • Faster processor, more memory • But even this couldn’t keep up
  • 11. 11© Cloudera, Inc. All rights reserved. Distributed Systems The better solution: more computers • Distributed systems – use multiple machines for a single job “In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, we didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers.” – Grace Hopper
  • 12. 12© Cloudera, Inc. All rights reserved. Distributed Systems: The Data Bottleneck (1) • Traditionally, data is stored in a central location • Data is copied to processors at runtime • Fine for limited amounts of data
  • 13. 13© Cloudera, Inc. All rights reserved. Distributed Systems: The Data Bottleneck (2) • Modern systems have much more data • terabytes+ a day • petabytes+ total • We need a new approach…
  • 14. 14© Cloudera, Inc. All rights reserved. The Origins of Hadoop • Hadoop is based on work done at Google in the late 1990s/early 2000s • Google’s problem: • Indexing the entire web requires massive amounts of storage • A new approach was required to process such large amounts of data • Google’s solution: • GFS, the Google File System - described in a paper released in 2003 • Distributed MapReduce - described in a paper released in 2004 • Doug Cutting and others read these papers and implemented a similar, open source solution • This is what would become Hadoop
  • 15. 15© Cloudera, Inc. All rights reserved. What is Hadoop? • Hadoop is a distributed data storage and processing platform • Stores massive amounts of data in a very resilient way • Distributes the processing to where the data is stored • Tools built around Hadoop (the ‘Hadoop ecosystem’) can be configured/extended to handle many different tasks • Extract Transform Load (ETL) • BI environment • Predictive analytics • Statistical analysis • Machine learning
  • 16. 16© Cloudera, Inc. All rights reserved. Hadoop is Scalable • Adding nodes (machines) adds capacity proportionally • Increasing load results in a graceful decline in performance • Not failure of the system Number of Nodes Capacity
  • 17. 17© Cloudera, Inc. All rights reserved. Hadoop is Fault Tolerant • Node failure is inevitable • What happens? • System continues to function • Master re-assigns work to a different node • Data replication means there is no loss of data • Nodes which recover rejoin the cluster automatically
  • 18. 18© Cloudera, Inc. All rights reserved. The Hadoop Ecosystem (1) • Many tools have been developed around ‘Core Hadoop’ • Known as the Hadoop ecosystem • Designed to make Hadoop easier to use, or to extend its functionality • All are open source • The ecosystem is growing all the time
  • 19. 19© Cloudera, Inc. All rights reserved. The Hadoop Ecosystem (2) • Examples of Hadoop ecosystem projects (all included in CDH): Project What does it do? Spark In-memory and streaming processing framework HBase NoSQL database built on HDFS Hive SQL processing engine designed for batch workloads Impala SQL query engine designed for BI workloads Parquet Very efficient columnar data storage format Sqoop Data movement to/from RDBMSs Flume, Kafka Streaming data ingestion Solr Powerful text search functionality Hue Web-based user interface for Hadoop Sentry Authorization tool, providing security for Hadoop
  • 20. 20© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Hadoop in the Real World
  • 21. 21© Cloudera, Inc. All rights reserved. Five Really Popular Use Cases • ETL Processing • Business Intelligence • Predictive Analytics • Enterprise Data Hub • Low-cost Storage of Large Data Volumes
  • 22. 22© Cloudera, Inc. All rights reserved. • ETL: Extract, Transform, Load • Challenges: • Too much data • Takes too long • Too costly # 1 – Traditional ETL Processing Networked Storage ETL Grid Data Warehouse Transactional Systems Data Stream BI Tools Networked Storage Tape Storage Tape Storage
  • 23. 23© Cloudera, Inc. All rights reserved. # 1 – ETL Processing with Hadoop • Hadoop cluster is used for ETL • Often now ELT: Extract, Load, then Transform • Structured and unstructured data is moved into the cluster • Once processed, data can be analyzed in Hadoop or moved to the EDW Networked Storage Data Stream Enterprise Data Warehouse ETL Analytics Real-time
  • 24. 24© Cloudera, Inc. All rights reserved. # 1 – Vendor Integration - ETL • For more information visit: http://www.cloudera.com/partners/partners- listing.html
  • 25. 25© Cloudera, Inc. All rights reserved. # 2 – Traditional Business Intelligence • BI traditionally takes place at the data warehouse layer • Problem: EDW can’t keep up with growing data volumes • Performance declines • Increasing capacity can be very expensive • Archived data is not available for analysis Networked Storage ETL Grid Data Warehouse Transactional Systems Data Stream BI Tools Tape Storage Tape Storage
  • 26. 26© Cloudera, Inc. All rights reserved. # 2 – Business Intelligence with Hadoop • BI tools can use Hadoop for much of their work • Analyze all the data • Use the EDW for the tasks for which it is best suited Networked Storage Data Stream Enterprise Data Warehouse ETL Analytics Real-time BI Tools
  • 27. 27© Cloudera, Inc. All rights reserved. # 2 – Vendor Integration - BI • For more information visit: http://www.cloudera.com/partners/partners- listing.html
  • 28. 28© Cloudera, Inc. All rights reserved. # 3 – Predictive Analytics • Predictive Analytics (Eckerson Group definition) • The use of statistical or machine learning models to discover patterns and relationships in data that can help business people predict future behavior or activity • The Hadoop platform can run analytic workloads on large volumes of diverse data • Statistical models can be created and run inside the Hadoop environment • Entire data sets can be used to create models • There is no need to sample data • Hadoop provides an environment that makes self-service analytics possible • No need for ETL developers to stage data for data scientists
  • 29. 29© Cloudera, Inc. All rights reserved. # 3 - Predictive Analytics: Cerner Corporation Cerner Corporation • Healthcare IT space • Solutions and Services - Used by 54,000 medical facilities around the world The problem • Healthcare data is fragmented and lives in silos • The data was used for historical reporting The solution • Build a comprehensive view of population health using a single platform • Use predictive analytics to • Improve patient outcomes • Increase efficiency / Reduce costs
  • 30. 31© Cloudera, Inc. All rights reserved. # 3 – Vendor Integration – Predictive Analytics • For more information visit: http://www.cloudera.com/partners/partners- listing.html
  • 31. 32© Cloudera, Inc. All rights reserved. # 4 – The Need for the Enterprise Data Hub Thousands of Employees & Lots of Inaccessible Information Heterogeneous Legacy IT Infrastructure Silos of Multi- Structured Data Difficult to Integrate ERP, CRM, RDBMS, Machines Files, Images, Video, Logs, Clickstreams External Data Sources Data Archives EDWs Marts SearchServers Document Stores Storage
  • 32. 33© Cloudera, Inc. All rights reserved. # 4 – The Enterprise Data Hub: One Unified System Information & data accessible by all for insight using leading tools and apps Enterprise Data Hub Unified Data Management Infrastructure Ingest All Data Any Type Any Scale From Any Source EDWs Marts Storage SearchServers Documents ERP, CRM, RDBMS, Machines Files, Images, Video, Logs, Clickstreams External Data Sources EDH Archives
  • 33. 34© Cloudera, Inc. All rights reserved. # 5 - Low-cost Data Storage (1) • Hadoop combines industry standard hardware and a fault tolerant architecture • This combination provides a very cost effective data storage platform • The data stored on Hadoop is protected from loss by HDFS • Data replication ensures that no data is lost • The self-healing nature of Hadoop ensures that data is available when you need it • Hadoop enables users to store data which was previously discarded due to the cost of saving it • Transactional • Social media • Sensor • Click stream
  • 34. 35© Cloudera, Inc. All rights reserved. # 5 - Low-cost Data Storage (2) • The low cost of HDFS storage enables the following use-cases: • Enterprise Data Hub (EDH) • Active data archive • Staging area for data warehouses • Staging area for analytics store • Sandbox for data discovery • Sandbox for analytics
  • 35. 36© Cloudera, Inc. All rights reserved. In Summary, Why Do You Need Hadoop? • More data is coming • Internet of things • Sensor data • Streaming • More data means bigger questions, better answers • Hadoop easily scales to store and handle all of your data • Hadoop is cost-effective • Provides a significant cost-per-terabyte saving over traditional, legacy systems • Hadoop integrates with your existing datacenter components • Answer questions that you previously could not ask
  • 36. 37© Cloudera, Inc. All rights reserved.© Cloudera, Inc. All rights reserved. Thank You

Editor's Notes

  1. All this said, Hadoop alone is not sufficient to help you succeed. You have existing investments in technology, vendors you already rely upon. Cloudera partners more broadly and deeply across the Hadoop ecosystem than any other vendor. With more than 1,900 partners across the entire stack, customers can choice wherever and however they want to deploy their big data platform. With integrations to existing technologies, we are able to provide our customers with compatibility with your existing tools and skills. Our special relationship with Intel, through their strategic investment in Cloudera, delivers unique advantages to our customers: Innovation: Hadoop is evolving at a pace like never seen before. New projects are constantly being added, sometimes at the expense of performance, security & quality. Aligning hardware and software roadmaps accelerates the pace of innovation, without compromising quality.  The most secure and performance new innovations are delivered faster. Ecosystem: Data architectures are complicated, requiring multiple products from multiple vendors. Often times, stitching together solutions can lead to one of projects that limit value to the business. Building modern architectures requires economies of scale, the ability to align independent providers along common goals for enterprise readiness. Intel and Cloudera have deep partnerships with leading vendors so you can build solutions with confidence. Reliability: New technology segments are hard to predict. Bet on the wrong one you slow your business down trying to integrate with existing investments or worse, run the risk of having to start from scratch, losing both time and money. Building modern architectures requires economies of scale, the ability to align independent providers along common goals for enterprise readiness.  Intel and Cloudera have deep partnerships with leading vendors so you can build solutions with confidence.