AI for an intelligent cloud and intelligent edge: Discover, deploy, and manage with Azure ML services

A talk presented by Max Schultze from Zalando and Arif Wider from ThoughtWorks at NDC Oslo 2020. Abstract: The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability. At Zalando - europe’s biggest online fashion retailer - we realised that accessibility and availability at scale can only be guaranteed when moving more responsibilities to those who pick up the data and have the respective domain knowledge - the data owners - while keeping only data governance and metadata information central. Such a decentralized and domain focused approach has recently been coined a Data Mesh. The Data Mesh paradigm promotes the concept of Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgement of data ownership. This talk will take you on a journey of how we went from a centralized Data Lake to embrace a distributed Data Mesh architecture and will outline the ongoing efforts to make creation of data products as simple as applying a template.

Past, present and future of data mesh at Intuit. This deck describes a vision and strategy for improving data worker productivity through a Data Mesh approach to organizing data and holding data producers accountable. Delivered at the inaugural Data Mesh Leaning meetup on 5/13/2021.

Machine learning development brings many new complexities beyond the traditional software development lifecycle. Unlike traditional software development, ML developers want to try multiple algorithms, tools and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. In this talk, learn how to operationalize ML across the full lifecycle with Databricks Machine Learning.

The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.

Dustin Vannoy is a field data engineer at Databricks and co-founder of Data Engineering San Diego. He specializes in Azure, AWS, Spark, Kafka, Python, data lakes, cloud analytics, and streaming. The document provides an overview of various Azure data and analytics services including Azure SQL DB, Cosmos DB, Blob Storage, Data Lake Storage Gen 2, Databricks, Synapse Analytics, Data Factory, Event Hubs, Stream Analytics, and Machine Learning. It also includes a reference architecture and recommends Microsoft Learn paths and community resources for learning.

Databricks is a Software-as-a-Service-like experience (or Spark-as-a-service) that is a tool for curating and processing massive amounts of data and developing, training and deploying models on that data, and managing the whole workflow process throughout the project. It is for those who are comfortable with Apache Spark as it is 100% based on Spark and is extensible with support for Scala, Java, R, and Python alongside Spark SQL, GraphX, Streaming and Machine Learning Library (Mllib). It has built-in integration with many data sources, has a workflow scheduler, allows for real-time workspace collaboration, and has performance improvements over traditional Apache Spark.

Delta Lake brings reliability, performance, and security to data lakes. It provides ACID transactions, schema enforcement, and unified handling of batch and streaming data to make data lakes more reliable. Delta Lake also features lightning fast query performance through its optimized Delta Engine. It enables security and compliance at scale through access controls and versioning of data. Delta Lake further offers an open approach and avoids vendor lock-in by using open formats like Parquet that can integrate with various ecosystems.

Tech talk on what Azure Databricks is, why you should learn it and how to get started. We'll use PySpark and talk about some real live examples from the trenches, including the pitfalls of leaving your clusters running accidentally and receiving a huge bill ;) After this you will hopefully switch to Spark-as-a-service and get rid of your HDInsight/Hadoop clusters. This is part 1 of an 8 part Data Science for Dummies series: Databricks for dummies Titanic survival prediction with Databricks + Python + Spark ML Titanic with Azure Machine Learning Studio Titanic with Databricks + Azure Machine Learning Service Titanic with Databricks + MLS + AutoML Titanic with Databricks + MLFlow Titanic with DataRobot Deployment, DevOps/MLops and Operationalization

By using a Data Lake, you no longer need to worry about structuring or transforming data before storing it. A Data Lake on AWS enables your organization to more rapidly analyze data, helping you quickly discover new business insights. Join us for our webinar to learn about the benefits of building a Data Lake on AWS and how your organization can begin reaping their rewards. In this webinar, select APN Partners will share their specific methodology for implementing a Data Lake on AWS and best practices for getting the most from your Data Lake.

Organizations today need a broad set of enterprise data cloud services with key data functionality to modernize applications and utilize machine learning. They need a platform designed to address multi-faceted needs by offering multi-function Data Management and analytics to solve the enterprise’s most pressing data and analytic challenges in a streamlined fashion. They need a worry-free experience with the architecture and its components. A complete machine learning infrastructure cost for the first modern use case at a midsize to large enterprise will be anywhere from $2M to $14M. Get this data point as you take the next steps on your journey.

Organizations are grappling to manually classify and create an inventory for distributed and heterogeneous data assets to deliver value. However, the new Azure service for enterprises – Azure Synapse Analytics is poised to help organizations and fill the gap between data warehouses and data lakes.

The document provides an overview of big data architectures and the data lake concept. It discusses why organizations are adopting data lakes to handle increasing data volumes and varieties. The key aspects covered include: - Defining top-down and bottom-up approaches to data management - Explaining what a data lake is and how Hadoop can function as the data lake - Describing how a modern data warehouse combines features of a traditional data warehouse and data lake - Discussing how federated querying allows data to be accessed across multiple sources - Highlighting benefits of implementing big data solutions in the cloud - Comparing shared-nothing, massively parallel processing (MPP) architectures to symmetric multi-processing (

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

Azure AI meetup in Houston. Automated Machine Learning and Azure Machine Learning. End to end workshop with training and deployment to Azure.

This is a brief introduction to Microsoft Azure cloud. I used these slides in an intro session for developers. I did few demos during the session that not included in the slide. Brand name and logos are properties of their respective owners.

The document discusses several announcements related to Amazon Web Services (AWS) data and analytics services. Some of the key announcements include: - Zero-ETL integration between Amazon Aurora and Amazon Redshift to eliminate the need for extract, transform, and load processes between the two services. - Updates to AWS Glue including new engines, data formats, and support for the Cloud Shuffle Service Plugin for Apache Spark. - Enhancements to Amazon SageMaker such as automated data preparation using machine learning, geospatial modeling capabilities, and shadow testing for machine learning models. - New services including Amazon DataZone for data discovery and access across organizations, Amazon Omics for genomic data storage and analysis, and AWS

The document provides information about an experienced machine learning solutions architect. It includes details about their experience and qualifications, including 12 AWS certifications and over 6 years of AWS experience. It also discusses their vision for MLOps and experience producing machine learning models at scale. Their role at Inawisdom as a principal solutions architect and head of practice is mentioned.

This is a hack material to help IoT partners/customers to know how to build a ML Edge module with Azure ML Service and deploy it to the Edge device.

The breath and depth of Azure products that fall under the AI and ML umbrella can be difficult to follow. In this presentation I’ll first define exactly what AI, ML, and deep learning is, and then go over the various Microsoft AI and ML products and their use cases.

第35回 Machine Learning 15minutes! (2019/04/27) https://machine-learning15minutes.connpass.com/event/124780/ [第35回 Machine Learning 15minutes!] Microsoft AI Updates https://satonaoki.wordpress.com/2019/04/27/ml15min-microsoft-ai-2/

This talk summarizes key points for big data advanced analytics on Microsoft Azure. First, there is a review of the major technologies. Second, there is a series of technology demos (focusing on VMs, Databricks and Azure ML Service). Third, there is some advice on using the Team Data Science Process to help plan projects. The deck has web resources recommended. This presentation was delivered at the Global Azure Bootcamp 2019, Atlanta GA location (Alpharetta Avalon).

Cloud is for all. There's so much passion and convictions among people involved in Cloud Computing that we might forget how inclusive is the Cloud and Microsoft Azure in particular. We all have heard about the DevOps story, where developers and IT operators are working hand and hand to create more value for the customer, but it's much more than this.

This document discusses the partnership between Microsoft Azure and GE's Predix platform for industrial IoT. For Microsoft, the partnership will help existing industrial customers build and operate IIoT solutions using Azure's capabilities in artificial intelligence, data analytics, and security. For GE, Predix will benefit from Azure's large global footprint and hybrid cloud capabilities. The combination of Predix and Azure aims to bridge the gap between operational technology and information technology for industrial customers worldwide.

Automated machine learning (automated ML) automates feature engineering, algorithm and hyperparameter selection to find the best model for your data. The mission: Enable automated building of machine learning with the goal of accelerating, democratizing and scaling AI. This presentation covers some recent announcements of technologies related to Automated ML, and especially for Azure. The demonstrations focus on Python with Azure ML Service and Azure Databricks.

by Mahendra Bairagi, AI Specialist Solutions Architect, AWS As the CTO of a new startup, you have taken up a challenge of improving the EDM music festival experience. At venues with multiple stages, festival-goers are always looking to identify DJ stage areas with the liveliest atmosphere. This causes them to constantly move around between different stages and miss out on having fun. You are looking to use Machine Learning and IoT technologies to solve this unique problem. Do you accept the Challenge? The objective of this task is to help the festival-goers quickly identify the DJ stage where crowd is the happiest. You've seen a lot of buzz around computer vision, machine learning, and IoT and want to use this technology to detect crowd emotions. From your initial research there are existing ML models that you can leverage to do face and emotion detection, but there are two ways that the predictions (inference) can be done; on the cloud and on the camera itself, but which one will work the best for your needs at the festival? You are going to test both approaches and find out! In this workshop you will use AWS and Intel technologies to learn how to build, deploy, and run ML inference on the cloud as well as on the IoT Edge. You will learn to use Amazon SageMaker with Intel C5 Instances, AWS DeepLens, AWS Greengrass, Amazon Rekognition, and AWS Lambda to build an end-to-end IoT solution that performs machine learning.

This presentation provides a survey of the advanced analytics strengths of Microsoft Azure from an enterprise perspective (with these organizations being the bulk of big data users) based on the Team Data Science Process. The talk also covers the range of analytics and advanced analytics solutions available for developers using data science and artificial intelligence from Microsoft Azure.

DataPalooza at the San Francisco Loft: In this workshop you will use AWS and Intel technologies to learn how to build, deploy, and run ML inference on the cloud as well as on the IoT Edge. You will learn to use Amazon SageMaker with Intel C5 Instances, AWS DeepLens, AWS Greengrass, Amazon Rekognition, and AWS Lambda to build an end-to-end IoT solution that performs machine learning.

David J. Rosenthal gave a presentation about Microsoft's Azure cloud platform. He discussed how Azure can help companies with digital transformation by engaging customers, empowering employees, and optimizing operations. He provided examples of how companies are using Azure services like AI, IoT, analytics and more to modernize applications, gain insights from data, and improve productivity. Rosenthal emphasized that Azure offers a secure, flexible cloud platform that businesses can use to innovate, grow and transform both today and in the future.

As the CTO of a new startup, you have taken up a challenge of improving the EDM music festival experience. At venues with multiple stages, festival-goers are always looking to identify DJ stage areas with the liveliest atmosphere. This causes them to constantly move around between different stages and miss out on having fun. In this workshop you will use AWS and Intel technologies to learn how to build, deploy, and run ML inference on the cloud as well as on the IoT Edge. You will learn to use Amazon SageMaker with Intel C5 Instances, AWS DeepLens, AWS Greengrass, Amazon Rekognition, and AWS Lambda to build an end-to-end IoT solution that performs machine learning.

In this presentation you'll find Machine Learning / Deep Learning tools and services from Microsoft. Including Azure Machine Learning Workbench, Azure Notebooks, Azure Data Science Virtual Machines and more. Here are the demos & resources https://github.com/ikivanc/Azure-ML-Workbench-Iris-Dataset-Classification https://github.com/ikivanc/Azure-ML-Resources

Manufacturing companies in all sectors—including automotive, aerospace, semiconductor, and industrial manufacturing—rely on design and engineering software in their product development processes. These computationally-intensive applications—such as computer-aided design and engineering (CAD and CAE), electronic design automation (EDA), other performance-critical applications—require immense scale and orchestration to meet the demands of today’s manufacturing requirements. In this session, you learn how to achieve the maximum possible performance and throughput from design and engineering workloads running on Amazon EC2, elastic GPUs, and managed services such as AWS Batch and Amazon AppStream 2.0. We demonstrate specific optimization techniques and share samples on how to accelerate batch and interactive workloads on AWS. We also demonstrate how to extend and migrate on-premises, high performance compute workloads with AWS, and use a combination of On-Demand Instances, Reserved Instances, and Spot Instances to minimize costs.

This document provides a summary of a presentation on innovating with AI at scale. The presentation discusses: 1. Implementing AI use cases at scale across industries like retail, life sciences, and transportation. 2. Deploying AI models to the edge using tools like TensorFlow and TensorRT for high-performance inference on devices. 3. Best practices and frameworks for distributed deep learning training on large clusters to train models faster.

This document summarizes a presentation given at AWS re:Invent 2017 about optimizing design and engineering performance in the cloud. The presentation covered deploying CAD/CAE/EDA applications in the cloud, optimizing storage and compute, managing technical software, enabling remote graphics and collaboration. It also included a case study from Hiroshi Kobayashi of Western Digital describing their use of AWS for CPU and GPU clusters to optimize product design.

This document discusses how VIA Technologies used AWS to address challenges from the COVID-19 pandemic for their 6nm IC design project. The pandemic impacted their project schedule unexpectedly and required work from home. AWS helped by quickly building a secure EDA infrastructure that improved productivity and may have allowed their project timeline to be accelerated. It provided proven EDA execution, smooth data transfer, and ongoing cost monitoring benefits. This case demonstrated how the cloud can provide new approaches for IC design projects during difficult situations.

Un tour d'horizon des solutions disponibles chez Microsoft pour bâtir une solution IoT. Il est question de Microsoft Azure bien-sûr, mais pas seulement. Windows, Machine Learning, Bots, OCF/AllJoyn, Hololens

This document discusses machine learning inference at the edge using Apache MXNet and AWS services. It begins with an overview of challenges with deep learning at the edge due to resource constraints and network connectivity. It then discusses Apache MXNet and how it can be used for flexible experimentation in the cloud, scalable training in the cloud, and good prediction performance at the edge by supporting different hardware. The document outlines options for predicting using cloud-based services like AWS Lambda and SageMaker endpoints or device-based prediction using tools like AWS Greengrass. It concludes by introducing AWS DeepLens, a deep learning enabled video camera, and providing resources for further information.