This document discusses using Azure services for big data analytics and data insights. It provides an overview of Azure services like Azure Batch, Azure Data Lake, Azure HDInsight and Power BI. It then describes a demo solution that uses these Azure services to analyze job posting data, including collecting data using a .NET application, storing in Azure Data Lake Store, processing with Azure Data Lake Analytics and Azure HDInsight, and visualizing results in Power BI. The presentation includes architecture diagrams and discusses implementation details.
AI for an intelligent cloud and intelligent edge: Discover, deploy, and manag...
Discover, manage, deploy, monitor – rinse and repeat. In this session we show how Azure Machine Learning can be used to create the right AI model for your challenge and then easily customize it using your development tools while relying on Azure ML to optimize them to run in hardware accelerated environments for the cloud and the edge using FPGAs and Neural Network accelerators. We then show you how to deploy the model to highly scalable web services and nimble edge applications that Azure can manage and monitor for you. Finally, we illustrate how you can leverage the model telemetry to retrain and improve your content.
This white paper will present the opportunities laid down by
data lake and advanced analytics, as well as, the challenges
in integrating, mining and analyzing the data collected from
these sources. It goes over the important characteristics of
the data lake architecture and Data and Analytics as a
Service (DAaaS) model. It also delves into the features of a
successful data lake and its optimal designing. It goes over
data, applications, and analytics that are strung together to
speed-up the insight brewing process for industry’s
improvements with the help of a powerful architecture for
mining and analyzing unstructured data – data lake.
This document provides an overview of a speaker and their upcoming presentation on Microsoft's data platform. The speaker is a 30-year IT veteran who has worked in various roles including BI architect, developer, and consultant. Their presentation will cover collecting and managing data, transforming and analyzing data, and visualizing and making decisions from data. It will also discuss Microsoft's various product offerings for data warehousing and big data solutions.
With new technologies such as Hive LLAP or Spark SQL, do I still need a data warehouse or can I just put everything in a data lake and report off of that? No! In the presentation I’ll discuss why you still need a relational data warehouse and how to use a data lake and a RDBMS data warehouse to get the best of both worlds. I will go into detail on the characteristics of a data lake and its benefits and why you still need data governance tasks in a data lake. I’ll also discuss using Hadoop as the data lake, data virtualization, and the need for OLAP in a big data solution. And I’ll put it all together by showing common big data architectures.
I often hear from clients: “We don’t know much about Big Data – can you tell us what it is and how it can help our business?” Yes! The first step is this vendor-free presentation, where I start with a business level discussion, not a technical one. Big Data is an opportunity to re-imagine our world, to track new signals that were once impossible, to change the way we experience our communities, our places of work and our personal lives. I will help you to identify the business value opportunity from Big Data and how to operationalize it. Yes, we will cover the buzz words: modern data warehouse, Hadoop, cloud, MPP, Internet of Things, and Data Lake, but I will show use cases to better understand them. In the end, I will give you the ammo to go to your manager and say “We need Big Data an here is why!” Because if you are not utilizing Big Data to help you make better business decisions, you can bet your competitors are.
With this support you would be able to have the basic of Azure Data slack and it will help you to pass the DP-200 and DP-201. If you need some basics on Azure, you can download this support : https://www.slideshare.net/AlexandreBERGERE/azure-fundamentals-153339148.
This support is a summary from the paths:
Azure for the Data Engineer
Store data in Azure
Work with relational data in Azure
Large Scale Data Processing with Azure Data Lake Storage Gen2
Implement a Data Streaming Solution with Azure Streaming Analytics
Implement a Data Warehouse with Azure SQL Data Warehouse
in Microsoft Learn.
Think of big data as all data, no matter what the volume, velocity, or variety. The simple truth is a traditional on-prem data warehouse will not handle big data. So what is Microsoft’s strategy for building a big data solution? And why is it best to have this solution in the cloud? That is what this presentation will cover. Be prepared to discover all the various Microsoft technologies and products from collecting data, transforming it, storing it, to visualizing it. My goal is to help you not only understand each product but understand how they all fit together, so you can be the hero who builds your companies big data solution.
This document discusses techniques for optimizing Power BI performance. It recommends tracing queries using DAX Studio to identify slow queries and refresh times. Tracing tools like SQL Profiler and log files can provide insights into issues occurring in the data sources, Power BI layer, and across the network. Focusing on optimization by addressing wait times through a scientific process can help resolve long-term performance problems.
Streaming Real-time Data to Azure Data Lake Storage Gen 2
Check out this presentation to learn the basics of using Attunity Replicate to stream real-time data to Azure Data Lake Storage Gen2 for analytics projects.
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWS
This session will focus on how to get from 'Minimum Viable Product' (MVP) to scale. It will also explain how to deal with unpredictable demand and how to build a scalable business. Attend this session to learn how to:
Scale web servers and app services with Elastic Load Balancing and Auto Scaling on Amazon EC2
Scale your storage on Amazon S3 and S3 Reduced Redundancy Storage
Scale your database with Amazon DynamoDB, Amazon RDS, and Amazon ElastiCache
Scale your customer base by reaching customers globally in minutes with Amazon CloudFront
The new Microsoft Azure SQL Data Warehouse (SQL DW) is an elastic data warehouse-as-a-service and is a Massively Parallel Processing (MPP) solution for "big data" with true enterprise class features. The SQL DW service is built for data warehouse workloads from a few hundred gigabytes to petabytes of data with truly unique features like disaggregated compute and storage allowing for customers to be able to utilize the service to match their needs. In this presentation, we take an in-depth look at implementing a SQL DW, elastic scale (grow, shrink, and pause), and hybrid data clouds with Hadoop integration via Polybase allowing for a true SQL experience across structured and unstructured data.
Data Analytics Meetup: Introduction to Azure Data Lake Storage
Microsoft Azure Data Lake Storage is designed to enable operational and exploratory analytics through a hyper-scale repository. Journey through Azure Data Lake Storage Gen1 with Microsoft Data Platform Specialist, Audrey Hammonds. In this video she explains the fundamentals to Gen 1 and Gen 2, walks us through how to provision a Data Lake, and gives tips to avoid turning your Data Lake into a swamp.
Learn more about Data Lakes with our blog - Data Lakes: Data Agility is Here Now https://bit.ly/2NUX1H6
The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...
Until recently, data was gathered for well-defined objectives such as auditing, forensics, reporting and line-of-business operations; now, exploratory and predictive analysis is becoming ubiquitous, and the default increasingly is to capture and store any and all data, in anticipation of potential future strategic value. These differences in data heterogeneity, scale and usage are leading to a new generation of data management and analytic systems, where the emphasis is on supporting a wide range of very large datasets that are stored uniformly and analyzed seamlessly using whatever techniques are most appropriate, including traditional tools like SQL and BI and newer tools, e.g., for machine learning and stream analytics. These new systems are necessarily based on scale-out architectures for both storage and computation.
Hadoop has become a key building block in the new generation of scale-out systems. On the storage side, HDFS has provided a cost-effective and scalable substrate for storing large heterogeneous datasets. However, as key customer and systems touch points are instrumented to log data, and Internet of Things applications become common, data in the enterprise is growing at a staggering pace, and the need to leverage different storage tiers (ranging from tape to main memory) is posing new challenges, leading to caching technologies, such as Spark. On the analytics side, the emergence of resource managers such as YARN has opened the door for analytics tools to bypass the Map-Reduce layer and directly exploit shared system resources while computing close to data copies. This trend is especially significant for iterative computations such as graph analytics and machine learning, for which Map-Reduce is widely recognized to be a poor fit.
While Hadoop is widely recognized and used externally, Microsoft has long been at the forefront of Big Data analytics, with Cosmos and Scope supporting all internal customers. These internal services are a key part of our strategy going forward, and are enabling new state of the art external-facing services such as Azure Data Lake and more. I will examine these trends, and ground the talk by discussing the Microsoft Big Data stack.
Chug building a data lake in azure with spark and databricks
- The document discusses building a data lake in Azure using Spark and Databricks. It begins with an introduction of the presenter and their experience.
- The rest of the document is organized into sections that discuss decisions around why to use a data lake and Azure/Databricks, how to build the lake by ingesting and organizing data, using Delta Lake for integrated and curated layers, securing the lake, and enabling analytics against the lake.
- The key aspects covered include getting data into the lake from various sources using custom Spark jobs, organizing the lake into layers, cataloging data, using Delta Lake for transactional tables, implementing role-based security, and allowing ad-hoc queries.
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...
SQLBits 2020 presentation on how you can build solutions based on the modern data warehouse pattern with Azure Synapse Spark and SQL including demos of Azure Synapse.
A sharing in a meetup of the AWS Taiwan User Group.
The registration page: https://bityl.co/7yRK
The promotion page: https://www.facebook.com/groups/awsugtw/permalink/4123481584394988/
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...
Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie.
Video of presentation:
https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332
This document provides an overview of a course on implementing a modern data platform architecture using Azure services. The course objectives are to understand cloud and big data concepts, the role of Azure data services in a modern data platform, and how to implement a reference architecture using Azure data services. The course will provide an ARM template for a data platform solution that can address most data challenges.
Modern Business Intelligence and Advanced Analytics
This document summarizes how businesses can transform through business intelligence (BI) and advanced analytics using Microsoft's modern BI platform. It outlines the Power BI and Azure Analysis Services tools for visualization, data modeling, and analytics. It also discusses how Collective Intelligence and Microsoft can help customers accelerate their move to a data-driven culture and realize benefits like increased productivity and cost savings by implementing BI and advanced analytics solutions in the cloud. The presentation includes demonstrations of Power BI and Azure Analysis Services.
Building Cloud-Native Applications with Microsoft Windows Azure
Cloud computing is here to stay, and it is never too soon to begin understanding the impact it will have on application architecture. In this talk we will discuss the two most significant architectural mind-shifts, discussing the key patterns changes generally and seeing how these new cloud patterns map naturally into specific programming practices in Windows Azure. Specifically this relates to (a) Azure Roles and Queues and how to combine them using cloud-friendly design
patterns, and (b) the combination of relational data and non-relational data, how to decide among them, and how to combine them. The goal is for mere mortals to build highly reliable applications that scale economically. The concepts discussed in this talk are relevant for developers and architects building systems for the cloud today, or who want to be prepared to move to the cloud in the future.
This talk was delivered by Bill Wilder at the Vermont Code Camp 2 on 11-Sept-2010.
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Scaling and Modernizing Data Platform with Databricks
This document summarizes Atlassian's adoption of Databricks to manage their growing data pipelines and platforms. It discusses the challenges they faced with their previous architecture around development time, collaboration, and costs. With Databricks, Atlassian was able to build scalable data pipelines using notebooks and connectors, orchestrate workflows with Airflow, and provide self-service analytics and machine learning to teams while reducing infrastructure costs and data engineering dependencies. The key benefits included reduced development time by 30%, decreased infrastructure costs by 60%, and increased adoption of Databricks and self-service across teams.
Organizations need to gain insight and knowledge from a growing number of Internet of Things (IoT), APIs, clickstreams, unstructured and log data sources. However, organizations are also often limited by legacy data warehouses and ETL processes that were designed for transactional data. In this session, we introduce key ETL features of AWS Glue, cover common use cases ranging from scheduled nightly data warehouse loads to near real-time, event-driven ETL flows for your data lake. We discuss how to build scalable, efficient, and serverless ETL pipelines using AWS Glue. Additionally, Merck will share how they built an end-to-end ETL pipeline for their application release management system, and launched it in production in less than a week using AWS Glue.
Today, data lakes are widely used and have become extremely affordable as data volumes have grown. However, they are only meant for storage and by themselves provide no direct value. With up to 80% of data stored in the data lake today, how do you unlock the value of the data lake? The value lies in the compute engine that runs on top of a data lake.
Join us for this webinar where Ahana co-founder and Chief Product Officer Dipti Borkar will discuss how to unlock the value of your data lake with the emerging Open Data Lake analytics architecture.
Dipti will cover:
-Open Data Lake analytics - what it is and what use cases it supports
-Why companies are moving to an open data lake analytics approach
-Why the open source data lake query engine Presto is critical to this approach
This document discusses connecting Oracle Analytics Cloud (OAC) Essbase data to Microsoft Power BI. It provides an overview of Power BI and OAC, describes various methods for connecting the two including using a REST API and exporting data to Excel or CSV files, and demonstrates some visualization capabilities in Power BI including trends over time. Key lessons learned are that data can be accessed across tools through various connections, analytics concepts are often similar between tools, and while partnerships exist between Microsoft and Oracle, integration between specific products like Power BI and OAC is still limited.
For our next ArcReady, we will explore a topic on everyone’s mind: Cloud computing. Several industry companies have announced cloud computing services . In October 2008 at the Professional Developers Conference, Microsoft announced the next phase of our Software + Services vision: the Azure Services Platform. The Azure Services Platforms provides a wide range of internet services that can be consumed from both on premises environments or the internet.
Session 1: Cloud Services
In our first session we will explore the current state of cloud services. We will then look at how applications should be architected for the cloud and explore a reference application deployed on Windows Azure. We will also look at the services that can be built for on premise application, using .NET Services. We will also address some of the concerns that enterprises have about cloud services, such as regulatory and compliance issues.
Session 2: The Azure Platform
In our second session we will take a slightly different look at cloud based services by exploring Live Mesh and Live Services. Live Mesh is a data synchronization client that has a rich API to build applications on. Live services are a collection of APIs that can be used to create rich applications for your customers. Live Services are based on internet standard protocols and data formats.
So you got a handle on what Big Data is and how you can use it to find business value in your data. Now you need an understanding of the Microsoft products that can be used to create a Big Data solution. Microsoft has many pieces of the puzzle and in this presentation I will show how they fit together. How does Microsoft enhance and add value to Big Data? From collecting data, transforming it, storing it, to visualizing it, I will show you Microsoft’s solutions for every step of the way
Join us for a deep dive into Windows Azure. We’ll start with a developer-focused overview of this brave new platform and the cloud computing services that can be used either together or independently to build amazing applications. As the day unfolds, we’ll explore data storage, SQL Azure™, and the basics of deployment with Windows Azure. Register today for these free, live sessions in your local area.
QuerySurge Slide Deck for Big Data Testing Webinar
This is a slide deck from QuerySurge's Big Data Testing webinar.
Learn why Testing is pivotal to the success of your Big Data Strategy .
Learn more at www.querysurge.com
The growing variety of new data sources is pushing organizations to look for streamlined ways to manage complexities and get the most out of their data-related investments. The companies that do this correctly are realizing the power of big data for business expansion and growth.
Learn why testing your enterprise's data is pivotal for success with big data, Hadoop and NoSQL. Learn how to increase your testing speed, boost your testing coverage (up to 100%), and improve the level of quality within your data warehouse - all with one ETL testing tool.
This information is geared towards:
- Big Data & Data Warehouse Architects,
- ETL Developers
- ETL Testers, Big Data Testers
- Data Analysts
- Operations teams
- Business Intelligence (BI) Architects
- Data Management Officers & Directors
You will learn how to:
- Improve your Data Quality
- Accelerate your data testing cycles
- Reduce your costs & risks
- Provide a huge ROI (as high as 1,300%)
Introduction to Microsoft’s Hadoop solution (HDInsight)James Serra
Did you know Microsoft provides a Hadoop Platform-as-a-Service (PaaS)? It’s called Azure HDInsight and it deploys and provisions managed Apache Hadoop clusters in the cloud, providing a software framework designed to process, analyze, and report on big data with high reliability and availability. HDInsight uses the Hortonworks Data Platform (HDP) Hadoop distribution that includes many Hadoop components such as HBase, Spark, Storm, Pig, Hive, and Mahout. Join me in this presentation as I talk about what Hadoop is, why deploy to the cloud, and Microsoft’s solution.
Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.
Big Data Analytics in the Cloud with Microsoft AzureMark Kromer
Big Data Analytics in the Cloud using Microsoft Azure services was discussed. Key points included:
1) Azure provides tools for collecting, processing, analyzing and visualizing big data including Azure Data Lake, HDInsight, Data Factory, Machine Learning, and Power BI. These services can be used to build solutions for common big data use cases and architectures.
2) U-SQL is a language for preparing, transforming and analyzing data that allows users to focus on the what rather than the how of problems. It uses SQL and C# and can operate on structured and unstructured data.
3) Visual Studio provides an integrated environment for authoring, debugging, and monitoring U-SQL scripts and jobs. This allows
AI for an intelligent cloud and intelligent edge: Discover, deploy, and manag...James Serra
Discover, manage, deploy, monitor – rinse and repeat. In this session we show how Azure Machine Learning can be used to create the right AI model for your challenge and then easily customize it using your development tools while relying on Azure ML to optimize them to run in hardware accelerated environments for the cloud and the edge using FPGAs and Neural Network accelerators. We then show you how to deploy the model to highly scalable web services and nimble edge applications that Azure can manage and monitor for you. Finally, we illustrate how you can leverage the model telemetry to retrain and improve your content.
This white paper will present the opportunities laid down by
data lake and advanced analytics, as well as, the challenges
in integrating, mining and analyzing the data collected from
these sources. It goes over the important characteristics of
the data lake architecture and Data and Analytics as a
Service (DAaaS) model. It also delves into the features of a
successful data lake and its optimal designing. It goes over
data, applications, and analytics that are strung together to
speed-up the insight brewing process for industry’s
improvements with the help of a powerful architecture for
mining and analyzing unstructured data – data lake.
Microsoft Data Platform - What's includedJames Serra
This document provides an overview of a speaker and their upcoming presentation on Microsoft's data platform. The speaker is a 30-year IT veteran who has worked in various roles including BI architect, developer, and consultant. Their presentation will cover collecting and managing data, transforming and analyzing data, and visualizing and making decisions from data. It will also discuss Microsoft's various product offerings for data warehousing and big data solutions.
Is the traditional data warehouse dead?James Serra
With new technologies such as Hive LLAP or Spark SQL, do I still need a data warehouse or can I just put everything in a data lake and report off of that? No! In the presentation I’ll discuss why you still need a relational data warehouse and how to use a data lake and a RDBMS data warehouse to get the best of both worlds. I will go into detail on the characteristics of a data lake and its benefits and why you still need data governance tasks in a data lake. I’ll also discuss using Hadoop as the data lake, data virtualization, and the need for OLAP in a big data solution. And I’ll put it all together by showing common big data architectures.
I often hear from clients: “We don’t know much about Big Data – can you tell us what it is and how it can help our business?” Yes! The first step is this vendor-free presentation, where I start with a business level discussion, not a technical one. Big Data is an opportunity to re-imagine our world, to track new signals that were once impossible, to change the way we experience our communities, our places of work and our personal lives. I will help you to identify the business value opportunity from Big Data and how to operationalize it. Yes, we will cover the buzz words: modern data warehouse, Hadoop, cloud, MPP, Internet of Things, and Data Lake, but I will show use cases to better understand them. In the end, I will give you the ammo to go to your manager and say “We need Big Data an here is why!” Because if you are not utilizing Big Data to help you make better business decisions, you can bet your competitors are.
With this support you would be able to have the basic of Azure Data slack and it will help you to pass the DP-200 and DP-201. If you need some basics on Azure, you can download this support : https://www.slideshare.net/AlexandreBERGERE/azure-fundamentals-153339148.
This support is a summary from the paths:
Azure for the Data Engineer
Store data in Azure
Work with relational data in Azure
Large Scale Data Processing with Azure Data Lake Storage Gen2
Implement a Data Streaming Solution with Azure Streaming Analytics
Implement a Data Warehouse with Azure SQL Data Warehouse
in Microsoft Learn.
Think of big data as all data, no matter what the volume, velocity, or variety. The simple truth is a traditional on-prem data warehouse will not handle big data. So what is Microsoft’s strategy for building a big data solution? And why is it best to have this solution in the cloud? That is what this presentation will cover. Be prepared to discover all the various Microsoft technologies and products from collecting data, transforming it, storing it, to visualizing it. My goal is to help you not only understand each product but understand how they all fit together, so you can be the hero who builds your companies big data solution.
This document discusses techniques for optimizing Power BI performance. It recommends tracing queries using DAX Studio to identify slow queries and refresh times. Tracing tools like SQL Profiler and log files can provide insights into issues occurring in the data sources, Power BI layer, and across the network. Focusing on optimization by addressing wait times through a scientific process can help resolve long-term performance problems.
Streaming Real-time Data to Azure Data Lake Storage Gen 2Carole Gunst
Check out this presentation to learn the basics of using Attunity Replicate to stream real-time data to Azure Data Lake Storage Gen2 for analytics projects.
AWS Cloud Kata 2013 | Singapore - Getting to Scale on AWSAmazon Web Services
This session will focus on how to get from 'Minimum Viable Product' (MVP) to scale. It will also explain how to deal with unpredictable demand and how to build a scalable business. Attend this session to learn how to:
Scale web servers and app services with Elastic Load Balancing and Auto Scaling on Amazon EC2
Scale your storage on Amazon S3 and S3 Reduced Redundancy Storage
Scale your database with Amazon DynamoDB, Amazon RDS, and Amazon ElastiCache
Scale your customer base by reaching customers globally in minutes with Amazon CloudFront
The new Microsoft Azure SQL Data Warehouse (SQL DW) is an elastic data warehouse-as-a-service and is a Massively Parallel Processing (MPP) solution for "big data" with true enterprise class features. The SQL DW service is built for data warehouse workloads from a few hundred gigabytes to petabytes of data with truly unique features like disaggregated compute and storage allowing for customers to be able to utilize the service to match their needs. In this presentation, we take an in-depth look at implementing a SQL DW, elastic scale (grow, shrink, and pause), and hybrid data clouds with Hadoop integration via Polybase allowing for a true SQL experience across structured and unstructured data.
Data Analytics Meetup: Introduction to Azure Data Lake Storage CCG
Microsoft Azure Data Lake Storage is designed to enable operational and exploratory analytics through a hyper-scale repository. Journey through Azure Data Lake Storage Gen1 with Microsoft Data Platform Specialist, Audrey Hammonds. In this video she explains the fundamentals to Gen 1 and Gen 2, walks us through how to provision a Data Lake, and gives tips to avoid turning your Data Lake into a swamp.
Learn more about Data Lakes with our blog - Data Lakes: Data Agility is Here Now https://bit.ly/2NUX1H6
The Hive Think Tank - The Microsoft Big Data Stack by Raghu Ramakrishnan, CTO...The Hive
Until recently, data was gathered for well-defined objectives such as auditing, forensics, reporting and line-of-business operations; now, exploratory and predictive analysis is becoming ubiquitous, and the default increasingly is to capture and store any and all data, in anticipation of potential future strategic value. These differences in data heterogeneity, scale and usage are leading to a new generation of data management and analytic systems, where the emphasis is on supporting a wide range of very large datasets that are stored uniformly and analyzed seamlessly using whatever techniques are most appropriate, including traditional tools like SQL and BI and newer tools, e.g., for machine learning and stream analytics. These new systems are necessarily based on scale-out architectures for both storage and computation.
Hadoop has become a key building block in the new generation of scale-out systems. On the storage side, HDFS has provided a cost-effective and scalable substrate for storing large heterogeneous datasets. However, as key customer and systems touch points are instrumented to log data, and Internet of Things applications become common, data in the enterprise is growing at a staggering pace, and the need to leverage different storage tiers (ranging from tape to main memory) is posing new challenges, leading to caching technologies, such as Spark. On the analytics side, the emergence of resource managers such as YARN has opened the door for analytics tools to bypass the Map-Reduce layer and directly exploit shared system resources while computing close to data copies. This trend is especially significant for iterative computations such as graph analytics and machine learning, for which Map-Reduce is widely recognized to be a poor fit.
While Hadoop is widely recognized and used externally, Microsoft has long been at the forefront of Big Data analytics, with Cosmos and Scope supporting all internal customers. These internal services are a key part of our strategy going forward, and are enabling new state of the art external-facing services such as Azure Data Lake and more. I will examine these trends, and ground the talk by discussing the Microsoft Big Data stack.
Chug building a data lake in azure with spark and databricksBrandon Berlinrut
- The document discusses building a data lake in Azure using Spark and Databricks. It begins with an introduction of the presenter and their experience.
- The rest of the document is organized into sections that discuss decisions around why to use a data lake and Azure/Databricks, how to build the lake by ingesting and organizing data, using Delta Lake for integrated and curated layers, securing the lake, and enabling analytics against the lake.
- The key aspects covered include getting data into the lake from various sources using custom Spark jobs, organizing the lake into layers, cataloging data, using Delta Lake for transactional tables, implementing role-based security, and allowing ad-hoc queries.
Big Data and Data Warehousing Together with Azure Synapse Analytics (SQLBits ...Michael Rys
SQLBits 2020 presentation on how you can build solutions based on the modern data warehouse pattern with Azure Synapse Spark and SQL including demos of Azure Synapse.
A sharing in a meetup of the AWS Taiwan User Group.
The registration page: https://bityl.co/7yRK
The promotion page: https://www.facebook.com/groups/awsugtw/permalink/4123481584394988/
Microsoft Ignite AU 2017 - Orchestrating Big Data Pipelines with Azure Data F...Lace Lofranco
Data orchestration is the lifeblood of any successful data analytics solution. Take a deep dive into Azure Data Factory's data movement and transformation activities, particularly its integration with Azure's Big Data PaaS offerings such as HDInsight, SQL Data warehouse, Data Lake, and AzureML. Participants will learn how to design, build and manage big data orchestration pipelines using Azure Data Factory and how it stacks up against similar Big Data orchestration tools such as Apache Oozie.
Video of presentation:
https://channel9.msdn.com/Events/Ignite/Australia-2017/DA332
This document provides an overview of a course on implementing a modern data platform architecture using Azure services. The course objectives are to understand cloud and big data concepts, the role of Azure data services in a modern data platform, and how to implement a reference architecture using Azure data services. The course will provide an ARM template for a data platform solution that can address most data challenges.
This document summarizes how businesses can transform through business intelligence (BI) and advanced analytics using Microsoft's modern BI platform. It outlines the Power BI and Azure Analysis Services tools for visualization, data modeling, and analytics. It also discusses how Collective Intelligence and Microsoft can help customers accelerate their move to a data-driven culture and realize benefits like increased productivity and cost savings by implementing BI and advanced analytics solutions in the cloud. The presentation includes demonstrations of Power BI and Azure Analysis Services.
Building Cloud-Native Applications with Microsoft Windows AzureBill Wilder
Cloud computing is here to stay, and it is never too soon to begin understanding the impact it will have on application architecture. In this talk we will discuss the two most significant architectural mind-shifts, discussing the key patterns changes generally and seeing how these new cloud patterns map naturally into specific programming practices in Windows Azure. Specifically this relates to (a) Azure Roles and Queues and how to combine them using cloud-friendly design
patterns, and (b) the combination of relational data and non-relational data, how to decide among them, and how to combine them. The goal is for mere mortals to build highly reliable applications that scale economically. The concepts discussed in this talk are relevant for developers and architects building systems for the cloud today, or who want to be prepared to move to the cloud in the future.
This talk was delivered by Bill Wilder at the Vermont Code Camp 2 on 11-Sept-2010.
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Amazon Web Services
The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Amazon Web Services
The world is creating more data in more ways than ever before. The average internet user in 2017 generates 1.5GB of data per day, with the rate doubling every 18 months. A single autonomous vehicle can generate 4TB per day. Each smart manufacturing plant generates 1PB per day. Storing, managing, and analyzing this data requires integrated database and analytic services that provide reliability and security at scale. AWS offers a range of managed data services that let customers focus on making data useful, including Amazon Aurora, RDS, DynamoDB, Redshift, Spectrum, ElastiCache, Kinesis, EMR, Elasticsearch Service, and Glue. In this session, we discuss these services, share our vision for innovation, and show how our customers use these services today. Learn More: https://aws.amazon.com/government-education/
Scaling and Modernizing Data Platform with DatabricksDatabricks
This document summarizes Atlassian's adoption of Databricks to manage their growing data pipelines and platforms. It discusses the challenges they faced with their previous architecture around development time, collaboration, and costs. With Databricks, Atlassian was able to build scalable data pipelines using notebooks and connectors, orchestrate workflows with Airflow, and provide self-service analytics and machine learning to teams while reducing infrastructure costs and data engineering dependencies. The key benefits included reduced development time by 30%, decreased infrastructure costs by 60%, and increased adoption of Databricks and self-service across teams.
Organizations need to gain insight and knowledge from a growing number of Internet of Things (IoT), APIs, clickstreams, unstructured and log data sources. However, organizations are also often limited by legacy data warehouses and ETL processes that were designed for transactional data. In this session, we introduce key ETL features of AWS Glue, cover common use cases ranging from scheduled nightly data warehouse loads to near real-time, event-driven ETL flows for your data lake. We discuss how to build scalable, efficient, and serverless ETL pipelines using AWS Glue. Additionally, Merck will share how they built an end-to-end ETL pipeline for their application release management system, and launched it in production in less than a week using AWS Glue.
Today, data lakes are widely used and have become extremely affordable as data volumes have grown. However, they are only meant for storage and by themselves provide no direct value. With up to 80% of data stored in the data lake today, how do you unlock the value of the data lake? The value lies in the compute engine that runs on top of a data lake.
Join us for this webinar where Ahana co-founder and Chief Product Officer Dipti Borkar will discuss how to unlock the value of your data lake with the emerging Open Data Lake analytics architecture.
Dipti will cover:
-Open Data Lake analytics - what it is and what use cases it supports
-Why companies are moving to an open data lake analytics approach
-Why the open source data lake query engine Presto is critical to this approach
This document discusses connecting Oracle Analytics Cloud (OAC) Essbase data to Microsoft Power BI. It provides an overview of Power BI and OAC, describes various methods for connecting the two including using a REST API and exporting data to Excel or CSV files, and demonstrates some visualization capabilities in Power BI including trends over time. Key lessons learned are that data can be accessed across tools through various connections, analytics concepts are often similar between tools, and while partnerships exist between Microsoft and Oracle, integration between specific products like Power BI and OAC is still limited.
For our next ArcReady, we will explore a topic on everyone’s mind: Cloud computing. Several industry companies have announced cloud computing services . In October 2008 at the Professional Developers Conference, Microsoft announced the next phase of our Software + Services vision: the Azure Services Platform. The Azure Services Platforms provides a wide range of internet services that can be consumed from both on premises environments or the internet.
Session 1: Cloud Services
In our first session we will explore the current state of cloud services. We will then look at how applications should be architected for the cloud and explore a reference application deployed on Windows Azure. We will also look at the services that can be built for on premise application, using .NET Services. We will also address some of the concerns that enterprises have about cloud services, such as regulatory and compliance issues.
Session 2: The Azure Platform
In our second session we will take a slightly different look at cloud based services by exploring Live Mesh and Live Services. Live Mesh is a data synchronization client that has a rich API to build applications on. Live services are a collection of APIs that can be used to create rich applications for your customers. Live Services are based on internet standard protocols and data formats.
So you got a handle on what Big Data is and how you can use it to find business value in your data. Now you need an understanding of the Microsoft products that can be used to create a Big Data solution. Microsoft has many pieces of the puzzle and in this presentation I will show how they fit together. How does Microsoft enhance and add value to Big Data? From collecting data, transforming it, storing it, to visualizing it, I will show you Microsoft’s solutions for every step of the way
Join us for a deep dive into Windows Azure. We’ll start with a developer-focused overview of this brave new platform and the cloud computing services that can be used either together or independently to build amazing applications. As the day unfolds, we’ll explore data storage, SQL Azure™, and the basics of deployment with Windows Azure. Register today for these free, live sessions in your local area.
QuerySurge Slide Deck for Big Data Testing WebinarRTTS
This is a slide deck from QuerySurge's Big Data Testing webinar.
Learn why Testing is pivotal to the success of your Big Data Strategy .
Learn more at www.querysurge.com
The growing variety of new data sources is pushing organizations to look for streamlined ways to manage complexities and get the most out of their data-related investments. The companies that do this correctly are realizing the power of big data for business expansion and growth.
Learn why testing your enterprise's data is pivotal for success with big data, Hadoop and NoSQL. Learn how to increase your testing speed, boost your testing coverage (up to 100%), and improve the level of quality within your data warehouse - all with one ETL testing tool.
This information is geared towards:
- Big Data & Data Warehouse Architects,
- ETL Developers
- ETL Testers, Big Data Testers
- Data Analysts
- Operations teams
- Business Intelligence (BI) Architects
- Data Management Officers & Directors
You will learn how to:
- Improve your Data Quality
- Accelerate your data testing cycles
- Reduce your costs & risks
- Provide a huge ROI (as high as 1,300%)
The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.
AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...Amazon Web Services
Analyze Big Data for Consumer Applications with Looker BI and Amazon Redshift Customizing the customer experience based on user behavior is a constant challenge for today’s consumer apps. Business intelligence helps analyze and model large amounts of data. Looker offers a modern approach to BI leveraging AWS that’s fast, agile, and easy to manage. Join this webinar to learn how MessageMe, which provides emotionally engaging messaging apps to consumers, leverages Looker business intelligence software and the Amazon Redshift data warehouse service to analyze billions of rows of customer data in seconds.
Webinar topics include:
• How MessageMe turns billions of rows of customer data stored in Amazon Redshift into actionable insights
• How Looker connects directly to Amazon Redshift in just a few clicks, enabling MessageMe to build a modern, big data analytics in the cloud. Who should attend
• Information or Solution Architects, Data Analysts, BI Directors, DBAs, Development Leads, Developers, or Technical IT Leaders.
Presenters:
• Justin Rosenthal, CTO, MessageMe
• Keenan Rice, VP, Marketing & Alliances, Looker
• Tina Adams, Senior Product Manager, AWS
Semplificare l'analisi dei dati con architetture "Serverless": architetture e...Amazon Web Services
This document discusses serverless big data architectures using AWS services. It describes how serverless computing eliminates the need to provision and manage servers. AWS services like S3, DynamoDB, Kinesis, Lambda, Glue and Athena allow ingesting, storing, processing and analyzing data without managing infrastructure. These services scale automatically based on usage and provide built-in availability and fault tolerance.
Azure Data Factory is a cloud-based data integration service that orchestrates and automates the movement and transformation of data. In this session we will learn how to create data integration solutions using the Data Factory service and ingest data from various data stores, transform/process the data, and publish the result data to the data stores.
Similar to Big Data Analytics from Azure Cloud to Power BI Mobile (20)
Microsoft Reactor Toronto 5/5/2020 | Azure Kubernetes In Action - Running and...Roy Kim
Azure Kubernetes Service (AKS) is a managed container orchestration service. With Kubernetes continuing to grow in popularity, many developers and IT engineers are curious to get started. Roy will demonstrate hosted microservices applications and the Istio service mesh. Along with how to manage your cluster with the Kubernetes Dashboard, Prometheus, Grafana and Azure Monitor. You will see a practical overview how all these pieces fit together.
www.roykim.ca
Twitter: @RoyKimYYZ
Github: https://github.com/RoyKimYYZ
Azure AD App Proxy Login Scenarios with an On Premises Applications - TSPUGRoy Kim
A presentation at a technology meetup.
Roy Kim will walk through various access scenarios and capabilities using Azure AD services and features to access SharePoint 2013/2016 server. This will include a comparison between AD Connect + Azure Application Proxy to publish an internal SharePoint application and 3rd Party Auth0 to assist in federating Azure AD and SSO integration. And also the recently supported Azure AD SAML 1.1 Token.
Roy will go through a demo, its architecture, and commentary of pros and cons. At the end you will have a good understanding of the technology capabilities to determine supporting access and user management scenarios.
Azure Key Vault with a PaaS Architecture and ARM Template DeploymentRoy Kim
This is a presentation I held at a local Azure user group. The session abstract: Azure Key Vault is a tool for securely storing and accessing secrets. We will go through a popular Azure PaaS Architecture pattern using Key Vault to store a password. I will demo and walk through the general configuration of a dedicated Azure Function app, Azure SQL and Key Vault that was deployed with automation. I will then go through fairly advanced techniques and best practices on how to deploy Azure Key Vault and a password secret with ARM templates. Finally, a very brief look at my Azure DevOps Pipeline to deploy the ARM template. You will come away with an understanding of an applied use case of leveraging Azure Key vault for a PaaS solution in better managing a password secret.
Azure App Gateway and Log Analytics under Penetration TestsRoy Kim
An end to end configuration of Azure App Gateway in front of Azure App Service with Log Analytics monitoring. You will see a demo of a simple penetration test and how you can monitor and alert with Log Analytics to detect common web attacks such as SQL injection and cross site scripting with the Gateway’s Web Application Firewall. You will walk away with an understanding of how Azure App Gateway and Log Analytics is applied as a security solution.
Applying Advanced Techniques to Azure Web AppsRoy Kim
A lap around 4 advanced techniques or services to complement an Azure Web App solution.
Application Gateway with Web Application Firewall
Azure SQL VNet Integration with (ASE v2)
Azure CDN
Auto Scale & Visual Studio Load Testing
Design and Configure Azure App Service Web AppsRoy Kim
Roy Kim presented on designing and configuring Azure Web Apps. He discussed key features like multiple languages/frameworks, global scaling, and integrations. Key design elements include app service plans for scaling, network security settings, and deployment options. He demonstrated the Azure portal and configuration of a sample web app. Roy encouraged the audience to get started with Azure trials and tutorials to learn best practices for designing production-ready apps in Azure App Service.
The document discusses SharePoint hybrid, which allows leveraging both SharePoint Server on-premises and SharePoint Online from Office 365 to achieve business goals. It provides an overview of hybrid capabilities like unified navigation, search, profiles, and personal file storage. Strategies for hybrid include keeping some content on-premises while transitioning other content online over time. Resources for learning more about SharePoint hybrid are also included.
SharePoint Hosted Add-in with AngularJS and BootstrapRoy Kim
The document describes a SharePoint hosted add-in that uses AngularJS and Bootstrap to create a rotating banner app. The app allows adding and managing photos, headlines and links. It uses AngularJS for two-way data binding, controllers and routing. Bootstrap is used for layout and styling. The add-in retrieves photos from a SharePoint library using JSOM and displays them with Angular bindings. The document provides an overview of the app requirements, architecture, AngularJS and Bootstrap usage, and code snippets from the Angular controllers.
The document discusses Microsoft Azure cloud services for solution architects, providing an overview of Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) offerings, how to design for scalability and performance, and the importance of using proof of concepts to identify architectural risks early in the design process. It also emphasizes how the agility of Azure allows for rapidly prototyping solutions through a mix of PaaS and IaaS.
SharePoint 2013 Hosted App Presentation by Roy KimRoy Kim
Sharing my experience and knowledge of developing a SharePoint Hosted App during late 2012. This app is a photo slider with a Picture Library and a custom list to hold default settings. Subjects include, REST API, JavaScript Object Model, Client Web Part, App Web, Cross Domain Call to the host web and other developer insights.
Networking For Application Developers by Roy KimRoy Kim
The document discusses networking fundamentals and troubleshooting for application developers. It covers network components like switches, routers, subnets, and firewalls. It also explains protocols like TCP/IP, DNS, DHCP and how to troubleshoot issues with IP addressing, connectivity, name resolution and firewall rules using tools like ipconfig, ping, tracert and netstat.
SharePoint Saturday 2010 - SharePoint 2010 Content Organizer FeatureRoy Kim
SharePoint Saturday Speaker presentation on the SharePoint 2010 Content Organizer Feature. Explain the business values especially around enterprise sites. Also explain
Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Em...Erasmo Purificato
Slide of the tutorial entitled "Paradigm Shifts in User Modeling: A Journey from Historical Foundations to Emerging Trends" held at UMAP'24: 32nd ACM Conference on User Modeling, Adaptation and Personalization (July 1, 2024 | Cagliari, Italy)
Blockchain technology is transforming industries and reshaping the way we conduct business, manage data, and secure transactions. Whether you're new to blockchain or looking to deepen your knowledge, our guidebook, "Blockchain for Dummies", is your ultimate resource.
The DealBook is our annual overview of the Ukrainian tech investment industry. This edition comprehensively covers the full year 2023 and the first deals of 2024.
An invited talk given by Mark Billinghurst on Research Directions for Cross Reality Interfaces. This was given on July 2nd 2024 as part of the 2024 Summer School on Cross Reality in Hagenberg, Austria (July 1st - 7th)
The Rise of Supernetwork Data Intensive ComputingLarry Smarr
Invited Remote Lecture to SC21
The International Conference for High Performance Computing, Networking, Storage, and Analysis
St. Louis, Missouri
November 18, 2021
Comparison Table of DiskWarrior Alternatives.pdfAndrey Yasko
To help you choose the best DiskWarrior alternative, we've compiled a comparison table summarizing the features, pros, cons, and pricing of six alternatives.
Details of description part II: Describing images in practice - Tech Forum 2024BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and transcript: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Transcript: Details of description part II: Describing images in practice - T...BookNet Canada
This presentation explores the practical application of image description techniques. Familiar guidelines will be demonstrated in practice, and descriptions will be developed “live”! If you have learned a lot about the theory of image description techniques but want to feel more confident putting them into practice, this is the presentation for you. There will be useful, actionable information for everyone, whether you are working with authors, colleagues, alone, or leveraging AI as a collaborator.
Link to presentation recording and slides: https://bnctechforum.ca/sessions/details-of-description-part-ii-describing-images-in-practice/
Presented by BookNet Canada on June 25, 2024, with support from the Department of Canadian Heritage.
Best Practices for Effectively Running dbt in Airflow.pdfTatiana Al-Chueyr
As a popular open-source library for analytics engineering, dbt is often used in combination with Airflow. Orchestrating and executing dbt models as DAGs ensures an additional layer of control over tasks, observability, and provides a reliable, scalable environment to run dbt models.
This webinar will cover a step-by-step guide to Cosmos, an open source package from Astronomer that helps you easily run your dbt Core projects as Airflow DAGs and Task Groups, all with just a few lines of code. We’ll walk through:
- Standard ways of running dbt (and when to utilize other methods)
- How Cosmos can be used to run and visualize your dbt projects in Airflow
- Common challenges and how to address them, including performance, dependency conflicts, and more
- How running dbt projects in Airflow helps with cost optimization
Webinar given on 9 July 2024
Quality Patents: Patents That Stand the Test of TimeAurora Consulting
Is your patent a vanity piece of paper for your office wall? Or is it a reliable, defendable, assertable, property right? The difference is often quality.
Is your patent simply a transactional cost and a large pile of legal bills for your startup? Or is it a leverageable asset worthy of attracting precious investment dollars, worth its cost in multiples of valuation? The difference is often quality.
Is your patent application only good enough to get through the examination process? Or has it been crafted to stand the tests of time and varied audiences if you later need to assert that document against an infringer, find yourself litigating with it in an Article 3 Court at the hands of a judge and jury, God forbid, end up having to defend its validity at the PTAB, or even needing to use it to block pirated imports at the International Trade Commission? The difference is often quality.
Quality will be our focus for a good chunk of the remainder of this season. What goes into a quality patent, and where possible, how do you get it without breaking the bank?
** Episode Overview **
In this first episode of our quality series, Kristen Hansen and the panel discuss:
⦿ What do we mean when we say patent quality?
⦿ Why is patent quality important?
⦿ How to balance quality and budget
⦿ The importance of searching, continuations, and draftsperson domain expertise
⦿ Very practical tips, tricks, examples, and Kristen’s Musts for drafting quality applications
https://www.aurorapatents.com/patently-strategic-podcast.html
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsMydbops
This presentation, delivered at the Postgres Bangalore (PGBLR) Meetup-2 on June 29th, 2024, dives deep into connection pooling for PostgreSQL databases. Aakash M, a PostgreSQL Tech Lead at Mydbops, explores the challenges of managing numerous connections and explains how connection pooling optimizes performance and resource utilization.
Key Takeaways:
* Understand why connection pooling is essential for high-traffic applications
* Explore various connection poolers available for PostgreSQL, including pgbouncer
* Learn the configuration options and functionalities of pgbouncer
* Discover best practices for monitoring and troubleshooting connection pooling setups
* Gain insights into real-world use cases and considerations for production environments
This presentation is ideal for:
* Database administrators (DBAs)
* Developers working with PostgreSQL
* DevOps engineers
* Anyone interested in optimizing PostgreSQL performance
Contact info@mydbops.com for PostgreSQL Managed, Consulting and Remote DBA Services
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Chris Swan
Have you noticed the OpenSSF Scorecard badges on the official Dart and Flutter repos? It's Google's way of showing that they care about security. Practices such as pinning dependencies, branch protection, required reviews, continuous integration tests etc. are measured to provide a score and accompanying badge.
You can do the same for your projects, and this presentation will show you how, with an emphasis on the unique challenges that come up when working with Dart and Flutter.
The session will provide a walkthrough of the steps involved in securing a first repository, and then what it takes to repeat that process across an organization with multiple repos. It will also look at the ongoing maintenance involved once scorecards have been implemented, and how aspects of that maintenance can be better automated to minimize toil.
How Social Media Hackers Help You to See Your Wife's Message.pdfHackersList
In the modern digital era, social media platforms have become integral to our daily lives. These platforms, including Facebook, Instagram, WhatsApp, and Snapchat, offer countless ways to connect, share, and communicate.
Quantum Communications Q&A with Gemini LLM. These are based on Shannon's Noisy channel Theorem and offers how the classical theory applies to the quantum world.
Best Programming Language for Civil EngineersAwais Yaseen
The integration of programming into civil engineering is transforming the industry. We can design complex infrastructure projects and analyse large datasets. Imagine revolutionizing the way we build our cities and infrastructure, all by the power of coding. Programming skills are no longer just a bonus—they’re a game changer in this era.
Technology is revolutionizing civil engineering by integrating advanced tools and techniques. Programming allows for the automation of repetitive tasks, enhancing the accuracy of designs, simulations, and analyses. With the advent of artificial intelligence and machine learning, engineers can now predict structural behaviors under various conditions, optimize material usage, and improve project planning.
Big Data Analytics from Azure Cloud to Power BI Mobile
1. Big Data Analytics from Azure
Data Platform to Power BI
Azure Batch, Azure Data Lake, Azure HDInsight, ML, Power BI
March 22, 2017
Roy Kim
@RoyKimYYZ
roykimtoronto@gmail.com
2. Agenda
Overview of Big Data + Azure + Data Insights
Job Postings demo solution architecture &
implementation
Mobile Demo with Power BI
Q&A
Author: Roy Kim
By: Roy Kim
3. Bio
Roy Kim
14+ Years of Microsoft Technology Solutions
.NET, SharePoint, BI, Office 365, Azure Solutions
IT Consultant
University of Toronto – Computer Science Degree
Author: Roy Kim
By: Roy Kim
4. Data to Insight
Author: Roy Kim
Big Data
Data Platform
Technologies
Solution Data Insights
By: Roy Kim
5. Job Postings Demo Solution
Author: Roy Kim References:
https://softwarestrategiesblog.com/2015/09/05/10-ways-big-data-is-revolutionizing-supply-chain-management
Job Postings Azure Data Platform
Data Lake, HDInsight,
SQL, Power BI
Job trends, analysis
By: Roy Kim
11. Job Postings Data Set
Volume
• Many national
job sites
• New job
postings daily
• Metadata and
full text.
Velocity
• New job
postings
created every
minute
Variety
• Semi-
structured
• Job Title
• Location
• Company
• Unstructured
• Job
Description
Veracity
• Incomplete/Im
precise
• Salary, Per
hour
• FT, PT, Temp,
Contract,
Seasonal
• Main
profession
By: Roy Kim
12. Power BI – Job Postings Demo Reports
By: Roy Kim
13. Power BI – Job Postings Demo Reports
By: Roy Kim
14. Job Postings Big Data Solution Architecture
Azure Data Lake
Analytics
Internet Data
Sets
USQL
Storage Account
Blob Store
Azure Batch
.NET Console
App
Blob Store
(WebHDFS)
Azure HDInsight
Hive
Azure
Active Directory
HDInsight
Azure SQL
database
SQL Data
Warehouse
Storage blob
Storage (Azure)
Visual Studio
Online
Data Lake
Azure SQL / Data
Warehouse
SQL DB
Machine
Learning
ML Studio
StorageTierServicesTier
REST/HTML/..
Visualization
/Reporting
Tools
Presentation
Tier
Mobile
Pig
Scoop
By: Roy Kim
Desktop
Batch
Business
Users
Report Builders
Data Analysts
Azure Data
Factory
Pipeline
Data Factory
Browser
Service
Applicatio
n Insights
Microsoft
Azure
Data
Analysis Services
Tabular
Machine
Learning
Storage Account
Blob Store
(HDFS)
Azure Data Lake
Store
Query
15. Job Postings from Internet Job Boards
Web sites that offer APIs
Use any server-side programming language to retrieve data such as NET, Java,
Node.js, etc.
If no APIs, consider HTML web page scraping
REST API
http end points typically return JSON or XML data formats
Html Web Page Scraping
HTML Agility Pack to assist in parsing the Document Object Model for data points.
https://www.nuget.org/packages/HtmlAgilityPack
HTML parsing supporting XPath to traverse the Document Object Model
(DOM)
E.g. doc.DocumentElement.SelectSingleNode(“//div*@id=‘Total Sales’+”)By: Roy Kim
16. Job Postings Data Collector .NET Console Application
.NET console application to read data from the internet and store into Azure Storage
accounts
Concurrent requests to job postings public API and HTML pages
Multi-threaded to increase speed and throughput
Parse HTML pages and JSON
Store JSON files directly into Azure Data Lake Store with ADLS .NET SDK
Leverages Azure Application Insights for logging trace and exception error
messages.
To store files into Azure Data Lake Store, the .NET application needs to access with
an Azure AD service principal with the appropriate access control.
By: Roy Kim
18. Azure Application Insights
By: Roy Kim
Application Insights Core API. This package provides core functionality for transmission of all
Application Insights Telemetry Types and is a dependent package for all other Application Insights
packages.
19. Azure Batch
A managed Azure service executing
command line applications.
For batch processing or batch
computing--running a large volume of
similar tasks to get some desired result.
Commonly used by organizations that
regularly process, transform, and
analyze large volumes of data.
Simply, a set of Azure Virtual Machines
running a console application to process
data that can be on a recurring schedule
and in parallel
References:
https://github.com/Microsoft/azure-docs/blob/master/articles/batch/batch-technical-overview.md
Author: Roy Kim
20. Azure Batch – Demo Implementation
Azure Batch runs the Console Application on a daily schedule against one node (Virtual
Machine - 2 cores)
To run console application in parallel through compute nodes, used the sample Parallel
Tasks .NET solution which uses the Azure Batch Client SDK.
https://github.com/Azure/azure-batch-
samples/tree/master/CSharp/ArticleProjects/ParallelTasks
Azure batch is an architecture option to support data collection in terms of velocity and
volume.
By: Roy Kim
21. Azure Data Lake
Intended for data storage in its raw format for future analysis, processing or
data modelling.
For developers, data scientists, and analysts to store data of any size, shape,
and speed.
To do all types of processing and analytics across different platforms and
languages.
Extract and load, minimal transformations
To manage data in characteristic of variety, velocity and volume
Two Components
1. Azure Data Lake Store
2. Azure Data Lake Analytics
References:
https://azure.microsoft.com/en-us/solutions/data-lake/
By: Roy Kim
22. Azure Data Lake Store
Azure Data Lake Store is a hyper-scale repository for big data analytic
workloads. Azure Data Lake enables you to capture data of any size, type,
and ingestion speed in one single place for operational and exploratory
analytics.
The Azure Data Lake store is an Apache Hadoop file system compatible with
Hadoop Distributed File System (HDFS)
Can be accessed from Hadoop (available with HDInsight cluster) using the
WebHDFS-compatible REST APIs
References:
https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview
By: Roy Kim
23. Azure Data Lake Store
Use Cases
Store social media
posts, log files, sensor
data
Store corporate data
such as
relational databases
(as flat files)
References:
https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview
By: Roy Kim
24. Azure Data Lake Analytics
Azure Data Lake Analytics is built to make big data analytics easy.
Focus on writing, running, and managing jobs, rather than operating distributed
infrastructure. Instead of deploying, configuring, and tuning hardware.
Write queries to transform your data and extract valuable insights. The analytics
service can handle jobs of any scale instantly by setting the dial for how much
power you need.
U-SQL – a Big Data query language. Likeness of SQL + C#
”schema on reads”
Pay for your job when it is running; making it cost-effective.
Data Collector app stores .json files in respective folders
USQL scripts logic:
reads 1000s of JSON files in a given folder
Outputs to one TSV (tab delimited) file
Create a Tables to schematize the TSV files
Query against tables to analyze or transform to a new output file.
References:
https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-overview
By: Roy Kim
25. Azure Data Lake Analytics – Demo Implementation
By: Roy Kim
USQL script: process json files into a tab delimited file
26. Azure Data Lake Analytics – Demo Implementation
By: Roy Kim
Roy Kim
# of JSON
Files
Single output
file
50 compute
nodes
3.4 mins
duration
27. Azure HDInsight
Hadoop refers to an ecosystem of open-source software that is a framework for
distributed processing, storing, and analysis of big data sets on clusters of
commodity computer hardware.
Azure HDInsight makes the Hadoop components from the Hortonworks Data
Platform (HDP) distribution available in Azure, deploys managed clusters with high
reliability and availability, and provides enterprise-grade security and governance
with Active Directory.
HDInsight offers the cluster types - Hadoop, HBase, Spark, Kafka, Interactive Hive,
Storm, customized, etc.
Supports integration with BI tools such as Power BI, Excel, SQL Server Analysis
Services, and SQL Server Reporting Services.
By: Roy Kim
28. Azure HDInsight – Demo Implementation
Hadoop Cluster Type
Data Source
Windows Azure Storage Account
Data Lake Store access
Data Lake Store Account
Hive Tables
JobPostings (internal) Table
Schema definition
Data loaded from Azure Data Lake Store .TSV file into Hadoop cluster’s WASB
JobPostings External Table
Data referenced in Azure Data Lake Store .TSV file. This is external to
HDInsight storage account.
By: Roy Kim
29. Azure HD Insight – Demo Implementation
Considerations
To manage the compute costs, script the provisioning and de-provisioning of the
cluster.
While a cluster is running, execute scripts and query the data into self service BI
tools and into other data warehouses.
In comparison to Azure Data Lake, ADL Analytics may be more cost effective
since it is pay per use at a more granular level - # of nodes and execution time.
E.g. Running against 100 nodes may cost a few dollars per minute in ADL
Analytics; whereas, in HDInsight, 13 nodes for small VM size may cost a few
dollars an hour.
By: Roy Kim
30. Azure SQL Database
A relational database-as-a-service in the cloud built on the Microsoft SQL Server
engine
No need to manage the infrastructure.
Scale up or down based on Database Transaction Units (DTUs).
1TB storage maximum
Can be used as a simpler data warehouse.
By: Roy Kim
31. Azure SQL Database – Demo Implementation
Developed a simple data warehouse
modelling
Job Postings data loaded from ADLS
Star schema
Added a date dimension table
Table of # of jobs for each province by
a date hierarchy
By: Roy Kim
32. Azure Data Factory
Cloud-based data integration service that orchestrates and automates
the movement and transformation of data.
Create data pipelines that move and transform data, and then run the pipelines on a specified
schedule (hourly, daily, weekly, etc.)
By: Roy Kim
33. Azure Data Factory
Category Data store Supported as a source Supported as a sink
Azure Azure Blob storage
Azure Data Lake Store
Azure SQL Database
Azure SQL Data Warehouse
Azure Table storage
Azure DocumentDB
Azure Search Index
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
Databases SQL Server*
Oracle*
MySQL*
DB2*
Teradata*
PostgreSQL*
Sybase*
Cassandra*
MongoDB*
Amazon Redshift
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
File File System*
HDFS*
Amazon S3
FTP
✓
✓
✓
✓
✓
Others Salesforce
Generic ODBC*
Generic OData
Web Table (table from HTML)
GE Historian*
✓
✓
✓
✓
✓
By: Roy Kim
34. Azure Machine Learning – Demo Implementation
By: Roy Kim
Predicting Salary for a
given set of parameters
such as job title and
location
37. The main features of
your Power BI service UI:
1. navigation bar
2. dashboard with tiles
3. Q&A question box
4. help and feedback
buttons
5. dashboard title
6. Office 365 app
launcher
7. Power BI home
buttons
8. Additional dashboard
actions
Power BI App Service
By: Roy Kim
38. • Frequently updated and accessed reports
• Minutes, hours, daily, weekly
• Fast and easy access of reports and dashboards
• IoT and sensor data
• Retail and customer analytics
• Team and organizational performance and productivity e.g. ticket
management
• Collaborative analysis and decision making
• Not always in front of a large screen device
Key Mobile Scenarios
By: Roy Kim
39. • Navigation
• Dashboards and Reports
• Responsive design
• Visualization interaction
• Sharing
• Annotations
• Q&A
• Alerts
• Favourites
Annotations
Mobile App IOS Key Features & Demo
By: Roy Kim
41. Security Architecture
Azure Data Lake
Analytics
Internet Data
Sources
USQL
Storage Account
Blob Store
Azure Batch
.NET Console
App
Azure Data Lake Store
Blob Store
(HDFS)
Azure HDInsight
Hive
Storage Account
Blob Store
(HDFS)
Azure
Active Directory
HDInsight
Azure SQL
database
SQL Data
Warehouse
Storage blob
Storage (Azure)
Visual Studio
Online
Data Lake
Azure SQL / Data
Warehouse
SQL DB
Analysis Services
(preview)
Tabular
StorageTierServicesTier
REST/HTML/..
Visualization
/Reporting
Tools
Pig
Scoop
Roy Kim
Desktop
Batch
BI
Developers
IT Ops
Azure Data
Factory
Pipeline
Data Factory
Microsoft
AzureAPI
Key
API
Key
AAD App
Service Principal
AAD App
Service Principal
AAD User
SQL account
AAD User
SQL account
Account
Key
AAD User
End Users
By: Roy Kim
Applicatio
n Insights
42. Data Processing & Formats
Azure Data Lake
Analytics
Internet Data
Sources
USQL
Azure Batch
.NET Console
App
Azure HDInsight
Hive
HDInsight
Azure SQL
database
SQL Data
Warehouse
Data Lake
Azure SQL
SQL DB
Analysis Services
(preview)
Tabular
DataFormatServicesTier
REST/HTML/.. Pig
Scoop
Roy Kim
Batch
Azure Data
Factory
Pipeline
Data Factory
JSON
HTML
JSON
TSV Hive Ext.
Table
Relational
DB
Hive Int.
Table
Table
By: Roy Kim
43. Closing Remarks
Cloud services such as Azure Data Platform provide new capabilities in
Data Analytics. That is in terms of scale, cost and agility.
Azure Data Lake is a productive option for organizations new to
Hadoop. Yet continue to plan for other Hadoop offerings best fit for
other scenarios.
Many azure services fit together to make the appropriate solution.
That is SaaS, PaaS, IaaS, Data, App, Operational, etc.
As part of planning and design, be aware of MS roadmap and industry
trends.
By: Roy Kim