Azure Data Engineering.pptx
•Download as PPTX, PDF•
1 like•829 views
This document provides an overview of using Azure Data Factory (ADF) for ETL workflows. It discusses the components of modern data engineering, how to design ETL processes in Azure, an overview of ADF and its components. It also previews a demo on creating an ADF pipeline to copy data into Azure Synapse Analytics. The agenda includes discussions of data ingestion techniques in ADF, components of ADF like linked services, datasets, pipelines and triggers. It concludes with references, a Q&A section and a request for feedback.
Report
Share
Azure Data Engineering.pptx
- 1. #AzConfDev Explore ETL using Azure Data Factory Azure Data Factory Walkthrough
- 2. #GlobalAzure Priyatharshini Sundaram Software Engineer III @PayPal, MCT 5X Azure Certified, 1X GCP Certified, Azure Architect ,Developer & DevOps Engineer Expert , Data Engineer https://www.linkedin.com/in/priyatharshini-s-75b002110/
- 3. Agenda Components of a modern data Engineering & Warehousing High-level design of ETL in Azure Overview of Data Factory & Components Demo - Create ADF Pipeline to Copy data into Synapse Analytics
- 4. Components of a modern data Engineering & Warehousing Azure Data Factory Azure Data Lake Azure Databricks Azure HDInsight Azure Synapse Analytics
- 5. High-level design of ETL in Azure
- 6. Azure Data Factory • Code free ETL/ELT orchestrator for hybrid data ingestion. • Currently, It has more than 85 pre-built connectors, which helps user to establish a connection with source and destination from various platforms like other cloud & on- premise. • Work is defined as a pipeline operation – runs continuously as CD. • Can either run in cloud or self-hosted network. • You can perform data processing/transformation using ADF though recommended only for small/medium size data.
- 7. Data Ingestion using Azure Data Factory Types of data ingestion in Azure ADF Heterogenous PolyBase 70 100 80 File based SSIS Heterogenous
- 8. Data Ingestion using Azure Data Factory (Cont…) components of Azure Data Factory Linked Service Data Lake Store Azure Databricks Activities Pipeline Triggers Dataset @ Parameters IR Integration Runtime CF Control Flow
- 10. Demo – Create ADF Pipeline to Copy data into Synapse Analytics Create ADF pipeline to perform ETL and store data into Synapse Analytics
- 11. Reference If you’re interested to explore more ,Please check out below topics @Microsoft Learn Modules (docs.microsoft.com/Learn) • /certifications/azure-data-fundamentals • /azure-data-fundamentals-explore-data-warehouse-analytics • /azure-for-the-data-engineer • /azure/data-factory/v1/data-factory-samples • / azure/data-factory/v1/data-factory-build-your-first-pipeline
- 12. Q & A
- 13. Feedback