The Hidden Value of Hadoop Migration
•
1 like•656 views
Many organizations focus on the licensing cost of Hadoop when considering migrating to a cloud platform. But other costs should be considered, as well as the biggest impact, which is the benefit of having a modern analytics platform that can handle all of your use cases. This session will cover lessons learned in assisting hundreds of companies to migrate from Hadoop to Databricks.
Report
Share
The Hidden Value of Hadoop Migration
- 1. Modernize Your Data Analytics Architecture with a Unified Approach to Data + AI Anand Venugopal Global Leader - Industry Solutions (Migrations) Databricks
- 2. Topics Why migrate from Hadoop to Databricks ? Success stories, technical and business benefits How can you migrate fast with low costs & low risk ?
- 3. Legacy On-Prem Analytics Architectures Are Not Keeping Up Hadoop costs rising when costs need to be cut Innovation hinges on ML and predictive insights Business agility requires real-time data This is preventing teams from driving high-impact business outcomes
- 4. Why Migrate to Databricks ? Forrester study finds 417% ROI for companies switching to Databricks 47% Cost-savings from retiring legacy infrastructure 5% Increase in revenue 25% Data team productivity increase
- 5. DEVOPS INTENSIVE RIGID AND INELASTIC Hadoop is Costly, Complex and Ineffective Hadoop ecosystem is complex and hard to manage that is prone to failures Low Productivity 24/7 HDFS clusters that need to be built for peak use and costly to upgrade Cost Prohibitive LACKS AI CAPABILITIES No out-of-box Hadoop support for ML/AI and separate environments for data and AI Slow Innovation X
- 6. Enterprises Need a Modern Data Analytics Architecture CRITICAL REQUIREMENTS Cost-effective scale and performance in the cloud Easy to manage and highly reliable for diverse data Predictive and real-time insights to drive innovation
- 7. Enhanced Productivity Lower Cost at Scale New Insights Faster Building a Modern Cloud Analytics Architecture with Databricks Data Science Workspace EASY TO MANAGE MASSIVE SCALE AI-ENABLED INNOVATION Managed cloud platform that can reliably handle all types of data On-demand, elastic autoscale clusters with optimized Apache Spark Unified and collaborative notebooks with built-in ML capabilities
- 8. Databricks Unified Data Analytics High performance query engineDELTA ENGINE One platform for every use caseStreaming Analytics BI Data Science Machine Learning Data Lake for all your data Structured, Semi-Structured and Unstructured Data Structured transactional layer
- 9. Powering Innovation with Modern Data Analytics Customers that migrated from Hadoop
- 10. Business value: What did they do with us? “The Un-carrier strategy is an approach that seeks to listen to the customer, address their pain points, bring innovation to the industry and improve the wireless experience for all.” Situation ○ Every network interaction (call, website load, text, app) logged in 1,600 node HDP data lake (30PB). ○ 4-5 “large scale” pipelines, with hundreds of downstream pipelines feeding the business ● PCMD (network measurement data), CDR (call records), EDR (DNS (website)), LSR (Location) ○ Process call data to get critical network insights: call-failure reasons and network outages. ○ PCMD – Per Call Measurement Data ● Provides insights on call failures at a granular level ● Best source to determine the outage cause and effect ● Provides rich information about the Sprint customers roaming in T-Mobile network
- 11. Solution: Holistic transformation instead of ‘lift & shift’ Overview ● Migration and transformation of streaming data analytics from Apache Storm and Hive on Hortonworks to Azure Databricks ● The Data was streaming in at an average of 2M records per second, 375GB per batch, 23 TB per day (uncompressed) Results Accelerating key insights e.g. hourly dashboards protecting revenue and customer churn. 78.5xPerformance gain versus on-prem operation BEFORE (with Hive on Tez): 47 mins for 15k cores to do the job AFTER: 35 mins for 256 cores to do the same job KPI computations took 1/4th of the time enabling new hourly dashboards (w/out optimizations e.g. warm pool and others still in process) 40%Reduction in use of 1600 node on-prem cluster
- 12. Supply Chain decisions Apply ML to 5000+ stores data Impact • 70% reduction in operational costs • Accelerated Business growth Demand Forecasting 500K stores, 2TB, 250 pipelines Impact • 10X more capacity • 2X faster data pipelines Predict Bakery food spoilage 10+ Large Hadoop clusters Impact • $100M in fresh food spoilage saved • $900K costs down, Time: 7 hr → 40m Optimize programming • Could not process 90 days of data with large Hadoop cluster Impact • 26% Team productivity increase • More Data, lower costs, low devops
- 13. Databricks Drives New Business Value at 3 Levels Databricks Value Framework The Data Platform Business Outcome More value Less value $$$ $$ $ BUSINESS IMPACTING USE CASES PRODUCTIVITY INFRASTRUCTURE Databricks accelerates and expands the realization of value from business-oriented use cases that use net-new capabilities vs. Hadoop Higher productivity among data scientists & data engineers eliminating manual tasks Reduced infrastructure spend with the performance of the Databricks runtime 3 1 2
- 14. $12.8M in value delivered with Databricks Value of Databricks ■ Removed Cloudera licensing ■ No need to add expensive new hardware for additional capacity ■ Avoided data center costs ■ Avoided Hadoop administration costs Cloudera costs vs. Databricks value & investment Units: $ Cumulative PV over 3 years Potential value with Databricks Cloudera - Cost of inaction Investment - Databricks, migration & cloud Net impact Includes cost of both solutions during migration $13.8 M -$18.7 M -$4.9 M $12.8 M Cloudera costs ■ Data center, Hadoop administration, new hardware, licensing Databricks investment ■ Databricks usage & support ■ Migration ■ Cloud compute Databricks customer example: Large U.S. Telco, 156 node cluster Source: Databricks value model Value of Databricks ■ Avoided Cloudera licensing ■ No need to add expensive new hardware for additional capacity ■ Avoided data center costs ■ Avoided Hadoop administration costs
- 15. Work with us for a Tailored Value Case for Your Migration Tailored Financial Analysis Tailored business case to be produced by answering 4 core questions: 1. How many nodes in your Hadoop environment? 2. How many people support your Hadoop environment? 3. When is your Cloudera renewal? 4. How do you expect your data needs to grow over time? Customer example
- 16. Proven Migration Strategy:Reduce Risk, Costs Databricks Expert Team System IntegratorsTools, ISV Partners AUTOMATION, TOOLS AND PROVEN METHODOLOGIES Cloud Partners COMPONENTS TO MIGRATE SUCCESSFUL MIGRATION Data + Metadata Workloads/ Jobs Security & governance Other tools, integrations Strategy Options: Lift & shift (faster, automatable) Transformation (higher impact)
- 17. Automated conversion for most workload types Data Migration Metastore Migration SQL Migration Security Scheduled Data pulls Orchestration HDFS Hive Databases / Tables / Views Impala Databases / Tables/ Views HDFS Hive Queries Spark Queries Sentry permissions /Ranger policies HDFS access permissions Sqoop statements Oozie Jobs Azure ADLS Gen 2, AWS S3 Databricks Tables Databricks Tables Spark Sql Databricks Notebooks Spark Sql Databricks Notebooks Databricks Notebooks Databricks permissions AWS IAM, ADLS ACLs Databricks compatible PySpark code Airflow DAGs & Databricks Jobs
- 18. Typically, customers save 55-66 % in costs and see a reduction of 2-3x in timelines by using Automation tool Data MigrationAssessment & Design Manual Migration Workloads Migration, Validation Cutover Operations 17- 20 Weeks 8 Weeks Using Automation Accelerated Data & Workloads Migration, Validation Accelerated Assessment & Design Cutover Operations * Typical implementation scenario ~ 4 PB of Data and 3000 jobs with mixed workloads considered
- 19. Our Partner Ecosystem will Accelerate Migrations ISV Partners and Migration Tools Security Governance Consulting & SI Partners Databricks Migration SWAT team + CS Packaged Services For Migration Cloud
- 20. Customized Hadoop Migration Success Plan with a Free Expert-led Assessment 1 2 3 Pre-questionnaire + Discovery, education workshops led by experts ▪ Learn about how Databricks works and how your current workloads, tools and processes map and transform in the future state in cloud Proposal and Recommendations for path forward ▪ The expert team will summarize all the findings and walk through the proposed costs, business value summary and recommended migration plan Technical, Use-case and Business Value analysis ▪ High level current and future state architecture, discuss use-cases and prioritize them, understand how $$ value is driven with the migration
- 21. Databricks Experts Know Hadoop ▪ More than 100 years of combined experience in Hadoop ▪ Practitioners, Architects, Engineers, and Consultants, Open Source Contributors and Committers ▪ Expertise with all Hadoop ecosystem components and distributions IMG IMG IMG IMG IMGIMG IMG
- 22. Hadoop migration to Databricks - recap Why - Costs, Productivity, Innovation → Business Impact Your competitors and market leaders are doing it NOW Databricks experts and automation strategy can help you migrate faster, with much lower cost and risk
- 23. Thank you Please visit databricks.com/migration