Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziyev and Olha Hrytsay
•
56 likes•10,358 views
BI architecture drivers have to change to satisfy new requirements in format, volume, latency, hosting, analysis, reporting, and visualization. In this presentation delivered at the 2014 SATURN conference, SoftServe`s Serhiy and Olha showcased a number of reference architectures that address these challenges and speed up the design and implementation process, making it more predictable and economical: - Traditional architecture based on an RDMBS data warehouse but modernized with column-based storage to handle a high load and capacity - NoSQL-based architectures that address Big Data batch and stream-based processing and use popular NoSQL and complex event-processing solutions - Hybrid architecture that combines traditional and NoSQL approaches to achieve completeness that would not be possible with either alone The architectures are accompanied by real-life projects and case studies that the presenters have performed for multiple companies, including Fortune 100 and start-ups.
Report
Share
Big Data Analytics: Reference Architectures and Case Studies by Serhiy Haziyev and Olha Hrytsay
- 1. BIG DATA ANALYTICS REFERENCE ARCHITECTURES AND CASE STUDIES BY SERHIY HAZIYEV AND OLHA HRYTSAY
- 2. Agenda 2 Big Data Challenges Big Data Reference Architectures Case Studies 10 tips for Designing Big Data Solutions
- 3. Big Data Challenges 3 UNSTRUCTURED STRUCTURED HIGH MEDIUM LOW Archives Docs Business Apps Media Social Networks Public Web Data Storages Machine Log Data Sensor Data Data Storages RDBMS, NoSQL, Hadoop, file systems etc. Machine Log Data Application logs, event logs, server data, CDRs, clickstream data etc. Sensor Data Smart electric meters, medical devices, car sensors, road cameras etc. Archives Scanned documents, statements, medical records, e-mails etc.. Docs XLS, PDF, CSV, HTML, JSON etc. Business Apps CRM, ERP systems, HR, project management etc. Social Networks Twitter, Facebook, Google+, LinkedIn etc. Public Web Wikipedia, news, weather, public finance etc Media Images, video, audio etc. Velocity Variety VolumeComplexity
- 4. Big Data Analytics 4 Traditional Analytics (BI) Big Data Analytics Focus on Data Sets Supports • Descriptive analytics • Diagnosis analytics • Limited data sets • Cleansed data • Simple models • Large scale data sets • More types of data • Raw data • Complex data models • Predictive analytics • Data Science Causation: what happened, and why? Correlation: new insight More accurate answers vs
- 5. Big Data Analytics Use Cases 5 Data Discovery Business Reporting Real Time Intelligence Data Quality Self Service Business Users Intelligent AgentsConsumers Low Latency Reliability Volume Performance Data Scientists/ Analysts
- 6. Big Data Analytics Reference Architectures 6 Architecture Drivers: Reference Architectures: ▪ Volume ▪ Sources ▪ Throughput ▪ Latency ▪ Extensibility ▪ Data Quality ▪ Reliability ▪ Security ▪ Self-Service ▪ Cost ▪ Extended Relational ▪ Non-Relational ▪ Hybrid
- 7. Relational Reference Architecture 7 Web Services Mobile Devices Native Desktop Web Browsers Advanced Analytics OLAP Cubes Query & Reporting Operational Data Stores Data Marts Data Warehouses Replication API/ODBC Messaging ETL Unstructured Semi- Structured Data Sources Integration Data Storages Analytics Presentation Structured
- 8. 8 Extended Relational Reference Architecture Web Services Mobile Devices Native Desktop Web Browsers Advanced Analytics OLAP Cubes Query & Reporting Operational Data Stores Data Marts Data Warehouses Replication API/ODBC Messaging ETL Unstructured Semi- Structured Data Sources Integration Data Storages Analytics Presentation Structured Key components affected with Big Data challenges
- 9. Non-Relational Reference Architecture 9 Web Services Mobile Devices Native Desktop Web Browsers Advanced Analytics Map Reduce Query & Reporting Search Engines Distributed File Systems NoSQL Databases API Messaging ETL Unstructured Semi- Structured Data Sources Integration Data Storages Analytics Presentation Structured Key components introduced with non-relational movement
- 10. Extended Relational vs. Non-Relational Architecture 10 Architecture Drivers Extended Relational Non-Relational Large data volume Self-service (ad-hoc reporting) Unstructured data processing High data model extensibility High data quality and consistency Extensive security Reliability and fault-tolerance Low latency (near-real time) Low cost Skills availability
- 11. Extended Relational vs. Non-Relational Architecture 11 Architecture Drivers Extended Relational Non-Relational Large data volume Self-service (ad-hoc reporting) Unstructured data processing High data model extensibility High data quality and consistency Extensive security Reliability and fault-tolerance Low latency (near-real time) Low cost Skills availability
- 12. Relational vs. Non-Relational Architecture 12 Relational Non-Relational • Rational • Predictable • Traditional • Agile • Flexible • Modern
- 13. Data Discovery Business Reporting Real Time Intelligence Big Data Analytics Use Cases 13 Business Users Intelligent AgentsConsumers Performance Volume Data Scientists
- 14. Data Discovery: Non-Relational Architecture 14 Web Services Mobile Devices Native Desktop Web Browsers Advanced Analytics Map Reduce Query & Reporting Search Engines Distributed File Systems NoSQL Databases API Messaging ETL Unstructured Semi- Structured Data Sources Integration Data Storages Analytics Presentation Structured
- 15. Data Discovery Business Reporting Real Time Intelligence Big Data Analytics Use Cases 15 Intelligent AgentsConsumers Data Scientists Data Quality Self Service Business Users
- 16. Business Reporting: Hybrid Architecture 16 Web Services Mobile Devices Native Desktop Web Browsers Map Reduce SQL Query & Reporting Distributed File Systems API Messaging ETL Unstructured Semi- Structured Data Sources Integration Data Storages Analytics Presentation Structured Relational DWH/DM Advanced Analytics Search Engines Extended Relational components Non-relational components
- 17. Data Discovery Business Reporting Real Time Intelligence Big Data Analytics Use Cases 17 Data Scientists Business Users Intelligent AgentsConsumers Low Latency Reliability
- 19. 19 Business Goals: Provide development environment for building custom mobile applications Charge customers for the platform they use with pay-as-you-go model Business Area: Cloud based platform for building, deploying, hosting and managing mobile applications Case Study #1: Usage & Billing Analysis
- 20. Architectural Decisions 20 ▪ Volume (> 10 TB) ▪ Sources (Semi-structured - JSON) ▪ Throughput (> 10K/sec) ▪ Latency (2 min) ▪ Extensibility (Custom metrics) ▪ Data Quality (Consistency) ▪ Reliability (24/7) ▪ Security (Multitenancy) ▪ Self-Service (Ad-Hoc reports) ▪ Cost (The less the better ) ▪ Constraints (Public Cloud) Architecture Drivers: Trade-off: // Extended Relational Non-Relational Extensibility - + Data Quality + - Self-Service + - Extended Relational Architecture Extensibility via Pre-allocated Fields pattern
- 21. Solution Architecture 21 Technologies: • Amazon Redshift • Amazon SQS • Amazon S3 • Elastic Beanstalk • Jaspersoft BI Professional • Python
- 22. 22 Business Goals: Build in-house Analytics Platform for ROI measurement and performance analysis of every product and feature delivered by the e-commerce platform; Provide the ability to understand how end-users are interacting with service content, products, and features on sites; Do clickstream analysis; Perform A/B Testing Business Area: Retail. A platform for e-commerce and collecting feedbacks from customers Case Study #2: Clickstream for retail website
- 23. // Extended Relational Non- Relational Volume/Scalability +/- + Throughput + + Self-Service + +/- Extensibility - + Architectural Decisions 23 ▪ Volume (45 TB) ▪ Sources (Semi-structured - JSON) ▪ Throughput (> 20K/sec) ▪ Latency (1 hour) ▪ Extensibility (Custom tags) ▪ Data Quality (Not critical) ▪ Reliability (24/7) ▪ Security (Multitenancy) ▪ Self-Service (Canned reports, Data science) ▪ Cost (The less the better ) ▪ Constraints (Public Cloud) Architecture Drivers: Trade-off: Non-Relational Architecture Reporting via Materialized View pattern
- 24. Solution Architecture 24 Technologies: • Amazon S3 • Flume • Hadoop/HDFS, MapReduce • HBase • Oozie • Hive Node 1 Node 2 Node N
- 25. 10 Tips for Designing Big Data Solutions 25 Understand data users and sources Discover architecture drivers Select proper reference architecture Do trade-off analysis, address cons Map reference architecture to technology stack Prototype, re-evaluate architecture Estimate implementation efforts Set up devops practices from the very beginning Advance in solution development through “small wins” Be ready for changes, big data technologies are evolving rapidly
- 26. 26 ▪ Leading global Product and Application Development partner founded in 1993 ▪ 3,300+ employees across North America, Ukraine and Western Europe ▪ Thousands of successful outsourcing projects! SaaS/Cloud Solutions . Mobility Solutions . UX/UI BI/Analytics/Big Data . Software Architecture . Security Clients include:
- 27. Thank You! 27 SoftServe US Office One Congress Plaza, 111 Congress Avenue, Suite 2700 Austin, TX 78701 Tel: 512.516.8880 Contacts Serhiy Haziyev: shaziyev@softserveinc.com Olha Hrytsay: ohrytsay@softserveinc.com
Editor's Notes
- Big Data – data that is too large complex and dynamic for any conventional data tools to capture, store, manage and analyze.
- More details can be found here: link to our case study