SlideShare a Scribd company logo

Oct 2011 CHADNUG Presentation on Hadoop

•Download as PPTX, PDF•
•
Josh Patterson
Josh Patterson

Josh Patterson gave a presentation on Hadoop and how it has been used. He discussed his background working on Hadoop projects including for the Tennessee Valley Authority. He outlined what Hadoop is, how it works, and examples of use cases. This includes how Hadoop was used to store and analyze large amounts of smart grid sensor data for the openPDC project. He discussed integrating Hadoop with existing enterprise systems and tools for working with Hadoop like Pig and Hive.

Related slideshows

Lessons Learned from Modernizing USCIS Data Analytics Platform
Lessons Learned from Modernizing USCIS Data Analytics PlatformLessons Learned from Modernizing USCIS Data Analytics Platform
Lessons Learned from Modernizing USCIS Data Analytics Platform
 
Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx
Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptxDowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx
Dowling buso-feature-store-logical-clocks-spark-ai-summit-2020.pptx
 
Infrastructure Agnostic Machine Learning Workload Deployment
Infrastructure Agnostic Machine Learning Workload DeploymentInfrastructure Agnostic Machine Learning Workload Deployment
Infrastructure Agnostic Machine Learning Workload Deployment
 
1 of 33
October 2011 – CHADNUG – Chattanooga, TNWhat is Hadoop?Josh Patterson | Sr Solution Architect
Who is Josh Patterson?josh@cloudera.comTwitter: @jpatanoogaMaster’s Thesis: self-organizing mesh networks Published in IAAI-09: TinyTermite: A Secure Routing AlgorithmConceived, built, and led Hadoop integration for openPDC project at Tennessee Valley Authority (TVA)Led team which designed classification techniques for time series and Map ReduceOpen source work at http://openpdc.codeplex.comhttps://github.com/jpatanoogaTodaySr. Solutions Architect at Cloudera
OutlineLet’s Set the StageStory Time: Hadoop and the SmartgridThe Enterprise and HadoopUse Cases and Tools3
“After the refining process, one barrel of crude oil yielded more than 40% gasoline and only 3% kerosene, creating large quantities of waste gasoline for disposal.”--- Excerpt from the book “The American Gas Station”Data Today: The Oil Industry Circa 19004
DNA Sequencing TrendsCost of DNA Sequencing Falling Very Fast5
Unstructured Data Explosion6Complex, UnstructuredRelational 2,500 exabytes of new information in 2012 with Internet as primary driver
Digital universe grew by 62% last year to 800K petabytes and will grow to 1.2 “zettabytes” this yearObstacles to Leveraging DataCopyright 2010 Cloudera Inc. All rights reserved7Data comes in many shapes and sizes: relational tuples, log files, semistructured textual data (e.g., e-mail)
Sometimes makes the data unwieldy
Customers are not creating schemas for all of their data
Yet still may want to join data sets
Customers are moving some of it to tape or cold storage, throwing it away because “it doesn’t fit”
They are throwing data away because its too expensive to hold
Similar to the oil industry in 1900A Need for a Platform in an Evolving LandscapeNeed ability to look at true distribution of dataPreviously impossible due to scaleNeed lower cost of analysisAd Hoc analysis now more open and flexibleNeed Greater Flexibility, “BI Agility”Less restrictive than SQL-only systemsSpeed @ Scale is the new Killer AppResults in that previously took 1 day to process can gain new value when created in 10 minutes.Copyright 2010 Cloudera Inc. All rights reserved8
Story Time: Hadoop and the Smartgrid9“We’re gonna need a bigger boat.”--- Roy Scheider, “Jaws”
NERC Sensor Data CollectionopenPDC PMU Data Collection circa 2009 120 Sensors
30 samples/second
4.3B Samples/day
Housed in HadoopNERC Wanted High-Res Smartgrid DataStarted openPDC project @ TVAhttp://openpdc.codeplex.com/We used Hadoop to store and process smartgrid (PMU) time series datahttps://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/Copyright 2011 Cloudera Inc. All rights reserved
Major Themes From openPDCVelocity of incoming dataWhere to put the data?Wanted to scale out, not upWanted linear scalability in cost vs sizeWanted system robust in the face of HW failureNot fans of vendor lock-inWhat can we realistically expect from analysis and extraction at this scale?How long does it take to scan a Petabyte @ 40MB/s?
Apache HadoopOpen Source Distributed Storage and Processing EngineConsolidates Mixed Data
Move complex and relational data into a single repository
Stores Inexpensively
Keep raw data always available
Use industry standard hardware
Processes at the Source
Eliminate ETL bottlenecks
Mine data first, govern later MapReduceHadoop Distributed File System (HDFS)
What Hadoop doesNetworks industry standard hardware nodes together to combine compute and storage into scalable distributed system
Scales to petabytes without modification
Manages fault tolerance and data replication automatically
Processes semi-structured and unstructured data easily
Supports MapReduce natively to analyze data in parallelWhat Hadoop does not doNo random access or transactions
Hadoop is not a database

More Related Content

Oct 2011 CHADNUG Presentation on Hadoop

  • 1. October 2011 – CHADNUG – Chattanooga, TNWhat is Hadoop?Josh Patterson | Sr Solution Architect
  • 2. Who is Josh Patterson?josh@cloudera.comTwitter: @jpatanoogaMaster’s Thesis: self-organizing mesh networks Published in IAAI-09: TinyTermite: A Secure Routing AlgorithmConceived, built, and led Hadoop integration for openPDC project at Tennessee Valley Authority (TVA)Led team which designed classification techniques for time series and Map ReduceOpen source work at http://openpdc.codeplex.comhttps://github.com/jpatanoogaTodaySr. Solutions Architect at Cloudera
  • 3. OutlineLet’s Set the StageStory Time: Hadoop and the SmartgridThe Enterprise and HadoopUse Cases and Tools3
  • 4. “After the refining process, one barrel of crude oil yielded more than 40% gasoline and only 3% kerosene, creating large quantities of waste gasoline for disposal.”--- Excerpt from the book “The American Gas Station”Data Today: The Oil Industry Circa 19004
  • 5. DNA Sequencing TrendsCost of DNA Sequencing Falling Very Fast5
  • 6. Unstructured Data Explosion6Complex, UnstructuredRelational 2,500 exabytes of new information in 2012 with Internet as primary driver
  • 7. Digital universe grew by 62% last year to 800K petabytes and will grow to 1.2 “zettabytes” this yearObstacles to Leveraging DataCopyright 2010 Cloudera Inc. All rights reserved7Data comes in many shapes and sizes: relational tuples, log files, semistructured textual data (e.g., e-mail)
  • 8. Sometimes makes the data unwieldy
  • 9. Customers are not creating schemas for all of their data
  • 10. Yet still may want to join data sets
  • 11. Customers are moving some of it to tape or cold storage, throwing it away because “it doesn’t fit”
  • 12. They are throwing data away because its too expensive to hold
  • 13. Similar to the oil industry in 1900A Need for a Platform in an Evolving LandscapeNeed ability to look at true distribution of dataPreviously impossible due to scaleNeed lower cost of analysisAd Hoc analysis now more open and flexibleNeed Greater Flexibility, “BI Agility”Less restrictive than SQL-only systemsSpeed @ Scale is the new Killer AppResults in that previously took 1 day to process can gain new value when created in 10 minutes.Copyright 2010 Cloudera Inc. All rights reserved8
  • 14. Story Time: Hadoop and the Smartgrid9“We’re gonna need a bigger boat.”--- Roy Scheider, “Jaws”
  • 15. NERC Sensor Data CollectionopenPDC PMU Data Collection circa 2009 120 Sensors
  • 18. Housed in HadoopNERC Wanted High-Res Smartgrid DataStarted openPDC project @ TVAhttp://openpdc.codeplex.com/We used Hadoop to store and process smartgrid (PMU) time series datahttps://openpdc.svn.codeplex.com/svn/Hadoop/Current%20Version/Copyright 2011 Cloudera Inc. All rights reserved
  • 19. Major Themes From openPDCVelocity of incoming dataWhere to put the data?Wanted to scale out, not upWanted linear scalability in cost vs sizeWanted system robust in the face of HW failureNot fans of vendor lock-inWhat can we realistically expect from analysis and extraction at this scale?How long does it take to scan a Petabyte @ 40MB/s?
  • 20. Apache HadoopOpen Source Distributed Storage and Processing EngineConsolidates Mixed Data
  • 21. Move complex and relational data into a single repository
  • 23. Keep raw data always available
  • 24. Use industry standard hardware
  • 26. Eliminate ETL bottlenecks
  • 27. Mine data first, govern later MapReduceHadoop Distributed File System (HDFS)
  • 28. What Hadoop doesNetworks industry standard hardware nodes together to combine compute and storage into scalable distributed system
  • 29. Scales to petabytes without modification
  • 30. Manages fault tolerance and data replication automatically
  • 31. Processes semi-structured and unstructured data easily
  • 32. Supports MapReduce natively to analyze data in parallelWhat Hadoop does not doNo random access or transactions
  • 33. Hadoop is not a database
  • 35. Hadoop is batch oriented
  • 38. This aspect is part of Cloudera’s value addHDFS: Hadoop Distributed File SystemBlock Size = 64MBReplication Factor = 3Cost/GB is a few ¢/month vs $/month
  • 39. MapReduceIn simple terms, it’s an application with 2 functionsMap FunctionThink massively parallel “group by”Reduce FunctionThink “aggregation + processing”Not hard to writeCan be challenging to refactor existing algorithmsDesigned to work hand-in-hand with HDFSMinimizes disk seeks, operates at “transfer rate” of disk
  • 40. Speed @ ScaleScenario1 million sensors, collecting sample / 5 min5 year retention policyStorage needs of 15 TBProcessingSingle Machine: 15TB takes 2.2 DAYS to scanMapReduce@ 20 nodes: Same task takes 11 Minutes
  • 41. MapReduce ToolsPigProcedural language compiled into MRHiveSQL-like language compiled into MRMahoutCollection of data mining algorithms for HadoopStreamingAbility to write MR with tools such as python, etc
  • 42. “What if you don’t know the questions?” --- Forrester ReportThe Enterprise and Hadoop20
  • 43. Apache Hadoop in Production©2011 Cloudera, Inc. All Rights Reserved.21How Apache Hadoop fitsinto your existing infrastructure.CUSTOMERSANALYSTSBUSINESS USERSOPERATORSENGINEERSWeb ApplicationManagement ToolsIDE’sBI / AnalyticsEnterprise ReportingEnterprise Data WarehouseOperational Rules EnginesLogsFilesWeb DataRelational Databases
  • 44. What the Industry is DoingMicrosoft ships CTP of Hadoop Connectors for SQL Server and Parallel Data Warehouse(based on Cloudera’sSqoop)http://blogs.technet.com/b/dataplatforminsider/archive/2011/08/25/microsoft-ships-ctp-of-hadoop-connectors-for-sql-server-and-parallel-data-warehouse.aspxOracleAnnouncments at Oracle Open WorldConnector to Hadoop allowing data flow into OracleHadoop Accelerator for ExalogicIn-memory processing for MapReduceETL using HadoopIntegrated analytics on Oracle and HadoopCopyright 2010 Cloudera Inc. All rights reserved22
  • 45. Integrating With the Enterprise IT Ecosystem23Drivers, language enhancements, testingFile System MountUI FrameworkSDKFUSE-DFSHUEHUE SDKWorkflowSchedulingMetadataSqoop* frame-work, adaptersAPACHE OOZIEAPACHE OOZIEAPACHE HIVEMore coming…Data IntegrationFast Read/Write AccessLanguages / CompilersAPACHE PIG, APACHE HIVEAPACHE FLUME, APACHE SQOOPAPACHE HBASECoordinationAPACHE ZOOKEEPERPackaging, testing*Sqoop supports JDBC, MySQL, Postgres, HSQLDB
  • 46. Forester Report World Economic Forum Declared that data is a new asset classBig data is an applied science project in most companiesMajor potential constraint is not the cost of the computing technology but the skilled people needed to carry out these projectsthe data scientistsWhat if you don’t know the questions? Big data is all about exploration without preconceived notionsNeed tools to ask questions to understand the right questions to askMuch of the software is based on open-source Hadoophttp://blogs.forrester.com/brian_hopkins/11-09-30-big_data_will_help_shape_your_markets_next_big_winnersCopyright 2010 Cloudera Inc. All rights reserved24
  • 47. Ever been recommended a friend on Facebook?Ever been recommended a product on Amazon?Ever used the homepage at Yahoo?What Can Hadoop Do For Me?25
  • 48. Problems Addressed With HadoopText MiningIndex BuildingSearch EnginesGraph CreationTwitter, FacebookPattern RecognitionNaïve Bayes ClassificationRecommendation EnginesPredictive ModelsRisk AssessmentCopyright 2010 Cloudera Inc. All rights reserved26
  • 49. A Few Named ExamplesAnalyze search terms and subsequent user purchase decisions to tune search results, increase conversion ratesDigest long-term historical trade data to identify fraudulent activity and build real-time fraud preventionModel site visitor behavior with analytics that deliver better recommendations for new purchasesContinually refine predictive models for advertising response rates to deliver more precisely targeted advertisementsReplace expensive legacy ETL system with more flexible, cheaper infrastructure that is 20 times fasterCorrelate educational outcomes with programs and student histories to improve resultsCopyright © 2011, Cloudera, Inc. All Rights Reserved.27
  • 50. Packages For HadoopDataFuFrom Linkedinhttp://sna-projects.com/datafu/UDFs in Pigused at LinkedIn in many of off-line workflows for data derived products"People You May Know”"Skills”TechniquesPageRankQuantiles (median), variance, etc.SessionizationConvenience bag functionsConvenience utility functions28
  • 51. Integration with LibsMix MapReduce with Machine Learning LibsWEKAKXENCPLEXMap side “groups data”Reduce side processes groups of data with Lib in parallelInvolves tricks in getting K/V pairs into libPipes, tmp files, task cache dir, etc29
  • 52. Ask the right questions up front..Is the job disk bound?What is the latency requirement on the job?Does it need a sub-minute latency? (not good for Hadoop!)Does the job look at a lot or all of the data at the same time?Hadoop is good at looking at all data, complex/fuzzy joinsIs large amounts of ETL processing needed before analysis?Hadoop is good at ETL pre-processing workCan the analysis be converted into MR / Pig / Hive?Copyright 2010 Cloudera Inc. All rights reserved30
  • 53. What Hadoop Not Good At in Data MiningAnything highly iterativeAnything that is extemely CPU bound and not disk boundAlgorithms that can’t be inherently parallelizedExamplesStochastic Gradient Descent (SGD)Support Vector Machines (SVM)Doesn’t mean they arent great to use
  • 54. Questions? (Thank You!)Hadoop World 2011http://www.hadoopworld.com/Cloudera’s Distribution including Apache Hadoop (CDH):http://www.cloudera.comResourceshttp://www.slideshare.net/cloudera/hadoop-as-the-platform-for-the-smartgrid-at-tvahttp://gigaom.com/cleantech/the-google-android-of-the-smart-grid-openpdc/http://news.cnet.com/8301-13846_3-10393259-62.htmlhttp://gigaom.com/cleantech/how-to-use-open-source-hadoop-for-the-smart-grid/Timeseries blog articlehttp://www.cloudera.com/blog/2011/03/simple-moving-average-secondary-sort-and-mapreduce-part-1/32
  • 55. More?Look at www.cloudera.com/training to learn more about Hadoop
  • 57. Lots of great use cases.
  • 58. Check out the downloads page at
  • 60. Get your own copy of Cloudera Distribution for Apache Hadoop (CDH)
  • 61. Grab Demo VMs, Connectors, other useful tools.
  • 62. Contact Josh with any questions at
  • 63. josh@cloudera.comCopyright 2010 Cloudera Inc. All rights reserved33

Editor's Notes

  1. Its all about the love, baby.
  2. Theme: they through away a lot of valuable gas and oil just like we through away data today
  3. Example of data production trends
  4. But what if some constraints changed?
  5. Talk about changing market dynamics of storage costWhat if some of the previously held constraints changed? Enter hadoop
  6. Let’s set the stage in the context of story, why we were looking at big data for time series.
  7. Ok, so how did we get to this point?Older SCADA systems take 1 data point per 2-4 seconds --- PMUs --- 30 times a sec, 120 PMUs, Growing by 10x factor
  8. Data was sampled 30 times a secondNumber of sensors (Phasor Measurement Units / PMU) was increasing rapidly was 120, heading towards 1000 over next 2 years, currently taking in 4.3 billion samples per dayCost of SAN storage became excessiveLittle analysis possible on SAN due to poor read rates on large amounts (TBs) of data
  9. Pool commodity servers in a single hierarchical namespace.Designed for large files that are written once and read many times.Example here shows what happens with a replication factor of 3, each data block is present in at least 3 separate data nodes.Typical Hadoop node is eight cores with 16GB ram and four 1TB SATA disks.Default block size is 64MB, though most folks now set it to 128MB
  10. Note: these are not simple queries, they are DEEP COMPLEX SCANS
  11. Note: these are not simple queries, they are DEEP COMPLEX SCANS
  12. Theme: they through away a lot of valuable gas and oil just like we through away data today
  13. Apache Hadoop is a new solution in your existing infrastructure.It does not replace any existing major existing investment.Apache brings data that you’re already generating into context and integrates it with your business.You get access to key information about how your business is operating but pulling togetherWeb and application logsUnstructured filesWeb dataRelational dataHadoop is used by your team to analyze this data and deliver it to business users directly and via existing data management technologies
  14. Allows vendor specific solutions to build on topCertain custom code would be written for the platform as well
  15. Theme: they through away a lot of valuable gas and oil just like we through away data today
  16. Transition into “hadoop as a platform”, but vendors are building on top of it for specific vertical challenges
  17. This is probably a better slide 30
  18. Check this against the Mahout impl