MADHAVA PATIBANDLA

My Takeaways from Gartner Data and Analytics presentation "Informatica: How the Cabinet Office is Fostering a Data Culture for Sustained Maturity and Growth?" Thanks to Julia Plant of the Cabinet Office and Steve Holyer of Informatica for this presentation #datastrategy #datamaturity #dataculture #valuechain #valuegeneration #GartnerDA

5

1 Comment

Completed the Databricks Accredited Databricks Fundamentals course! Thrilled to deepen my expertise in scalable data processing and analytics. Looking forward to driving innovation and efficiency through data-driven solutions. #Databricks #BigData #DataProcessing

5

What happens when there is a business use case where you have to work on creating models and the data required is sitting at different places like in onpremise systems (Hadoop , DB2 or EDW) and some of the data is in cloud (Azure, AWS or GCP ) The data required to create models is present in chunks at different places. How will you work on these data since the data is not present in single source ? One solution to this is : You have to bring the data into one place so that you can work on the data. How to bring the chunks of data from different sources to a single place since all of the data are present in different place ? So what we can do is migrate the required data from the onpremise systems to cloud and then perform data manipulations on the whole chunk of data using platforms like databricks or snowflake. But there comes a fallout to this solution or approach. Since we are migrating the data it comes with a cost to the business. There will be ingress and egress cost attached to data that is getting moved from one place to other. Also if the data size is huge it will add to huge cost to business. There will also be data duplicacy since the data will be present at both the places. So can we look for any other approach to this ? Is there any specific approach from where we can reduce data duplicacy? or even moving the data from onpremise to cloud can we work on this use case ? Yes here comes denodo in the picture. Denodo is a data virtualization tool where we can create connectivity between various data sources and perform our manipulations in the denodo itself. There is no need to move your data and no data duplicacy will be happening. Denodo serves as a platform where you can do manipulations in the data, create models and join data present in different sources ( be it cloud or be it onpremise ). There is no requirement to move the chunk of data to denodo. You just have to create a connectivity and you are good to go. #data #usecase #analytics #Denodo #costoptimization #datavirtualization

18

1 Comment

Snowflake Data Governance Framework by Abhijit Debnath 5 Minutes read

10

It has been a great tech talk session about Databricks by Rajesh Veeraraghavalu

31

Good Overview!!! I highly recommend Vidhi Chugh's datacamp course—it's definitely worth checking out!

12

Discover how Aditi Consulting facilitated seamless synergy between a construction equipment giant's data science and marine digital teams. Dive into our case study highlighting the deployment of integrated reporting solutions and multi-faceted dashboards, powered by Tableau: https://lnkd.in/gy9iDKAG

24

Databricks LakeFlow Data engineering is crucial for democratizing data and AI within businesses, but it remains a challenging and complex field. Data teams face various challenges while building end-to-end solutions. 1.      Ingesting data from siloed proprietary systems, maintaining intricate logic for data preparation, and dealing with failures and latency spikes that can lead to operational disruptions 2.      Existing solutions are often fragmented and incomplete, resulting in low data quality, reliability issues, high costs, and an increasing backlog of work 3. Deploying pipelines and monitoring data quality typically requires additional, disparate tools, further complicating the process LakeFlow addresses these challenges by simplifying all aspects of data engineering via a single, unified experience built on the Databricks Data Intelligence Platform, with deep integrations with Unity Catalog for end-to-end governance and serverless compute enabling highly efficient and scalable execution Key features of LakeFlow 1.      LakeFlow Connect – Provides a breadth of native, scalable connectors for databases such as MySQL, Postgres, SQL Server and Oracle as well as enterprise applications like Salesforce, Dynamics, Sharepoint, Workday and NetSuite. These connectors are fully integrated with Unity Catalog, providing for robust data governance 2.      LakeFlow Pipelines - Built on Databricks highly scalable Delta Live Tables technology, LakeFlow Pipelines allows data teams to implement data transformation and ETL in SQL or Python. Customers can now enable Real Time Mode for low-latency streaming without any code changes. LakeFlow eliminates the need for manual orchestration and unifies batch and stream processing 3.      LakeFLow Jobs - LakeFlow Jobs automates deploying, orchestrating and monitoring data pipelines in a single place, making it easier for data teams to meet their data delivery promises. It provides enhanced control flow capabilities and full observability to help detect, diagnose and mitigate data issues for increased pipeline reliability Summary This unified solution for data ingestion, transformation, and orchestration, built on Databricks’ Delta Live Tables (DLT) framework will be a game-changer. As a Databricks partner, we are excited about LakeFlow, enabling us to help clients enhance data management. One of the most exciting features of LakeFlow is its intuitive pipeline user interface. Even non-coders will be able to define an end-to-end pipeline fast. It will be interesting to see how LakeFlow fares compared to the existing Workflow jobs Share your thoughts in the comments below. Let's spark a conversation! #LTIMindtreeXDatabricks #Techbytes #DataPrivacy #DatabricksInnovation #LTIMDatabricksCoE LTIMindtreeDatabricksJitendra ChakravarthyRakesh SanchetiUsha VenkatManas Ranjan NayakSamuel GallaSachin PandeAbhishek PatelJanga Phani ReddyJennifer Rivet

10

1 Comment

🎉 Day 2 at Informatica World with CTI DATA ! 🎉 Today's focus: data science, data mesh, and some awesome new features! 🧠💡 🚀 InfaCore: Seamless data science and ML integration. Drag-and-drop simplicity! ✨ ModelServe: Effortless ML model deployment at scale. Fast, reliable, and hassle-free! 🛠️ Human Task in Cloud Data Application: Add human decisions in workflows. Perfect for approvals and exception management. Just drag, drop, and configure! Highlights: 🔍 Data Science in IDMC: Real-time and enterprise analytics made easy. 🔗 Data Mesh and Governance: Discoverable, valuable, trustworthy, and secure data assets. My head is buzzing with ideas! 🤯 Can't wait to implement these learnings. Stay tuned for more updates! 🚀💬 With these insights, my dog is eyeing a promotion to Chief Data Paw-fficer! 🐶📈 #InformaticaWorld #CTIData #DataScience #DataMesh #InfaCore #ModelServe #HumanTask #DataGovernance #AI #MachineLearning #BigData #DataIntegration #CloudComputing

23

1 Comment

Happy to share that I have recently earned Databricks Data Engineer Associate certification 🎉 Big thanks to Databricks Academy ILT and E-learning courses helped me to jumpstart my Databricks journey and upskill myself in Databricks. Special thanks to Derar Alhussein for his concise but comprehensive Udemy Preparation course, which helped me to revise all the exam topics. #databricks #databrickscertified #dataengineeerassociate #databricksacademy #dataengineering #lakehouse #deltalake #spark #bigdata

52

8 Comments

Adaptive and flexible governance enables Data & Analytics leaders to select different governance styles for differing business scenarios. Use Gartner for IT's roadmap to understand the key stages, resources and people required to plan and execute an effective D&A governance initiative. #GartnerDA #Data #Analytics #DataManagement

4

Completed the comprehensive LookML Developer training and certification, mastering data modeling, analytics, and visualization techniques. #LookML #DataAnalytics #Certification #ProfessionalDevelopment

21

Finally got around to posting this! We have been using Snowflake as the data platform of choice for all treasury applications at Snowflake. Reach out to me if you have any crazy ideas about data apps that you always wanted to build. We can make them happen. With Ajita Sharma at Snowflake Summit 2024 #snowflake #snowflake_summit

137

4 Comments

Introducing Databricks Assistant Autocomplete. Assistant Autocomplete real time code suggestions as you type in SQL and Python. It uses context from code cells, Unity Catalog metadata, DataFrame data, and more to relevant suggestions as you type.

7

1 Comment

Join with us at the Informatica World Conference for an insightful session on practical AI! Listen to our expert Sanjeev Naidu to know how to leverage clean, trusted, real-time #data to make transparent and unbiased decisions. Discover the power of explainable #AI, minimize biases and errors, and unlock real-world value with practical applications. #InformaticaWorld #AI #DataInnovation #InformaticaWorld2024 #MDM #IDMC #DataInnovation #TrustedAI #InformaticaWorld #MiracleSoftwareSystems #miraclesoftconferences

4

Just finished the course “MLOps Essentials: Monitoring Model Drift and Bias” by Kumaran Ponnambalam! Check it out: https://lnkd.in/gjfSsCVA #mlops.

19

3 Comments