Benoit Dageville

This is a huge step towards aligning the open data lakehouse formats. Recently Microsoft announced compatibilty with Snowflake for their fabric OneLake which was already compatible with Databricks. The future of data lakehouses is open and interoperable… your storage format should not constraint you from using your favorite compute engine..

14

There's a big difference between Database Modeling and Semantic Layers- How would you explain it? Even though they look alike, the underlying meaning is vastly different. ▶ The type of data modeling you would do with database modelers like SqlDBM and semantic layers are interrelated in that we help design and organize database structures in a way that supports the creation of a semantic layer. However: ▶ A database modeling solution is focused on the 𝐩𝐡𝐲𝐬𝐢𝐜𝐚𝐥 and 𝐥𝐨𝐠𝐢𝐜𝐚𝐥 aspects of database modeling, ensuring the data structures align with technical requirements and business concepts. ▶ A semantic layer operates at a higher level of abstraction often within 𝐁𝐈 𝐭𝐨𝐨𝐥𝐬 such as PowerBI, Looker, ThoughtSpot, and Tableau. It maps the technical details of data that were designed by tools like SqlDBM or not properly designed at all, to business-friendly terms and concepts. ▶A tool like SqlDBM will essentially lay the 𝐟𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 that a semantic layer translates into an easily navigable and understandable format for end-users. ---- "Enabling non-technical business users to perform data analysis without needing to understand complex database queries or structures. However, leveraging a data modeling tool and having a good data model will make the complexity of building and running a semantic layer more manageable and efficient." - Keith Belanger, 𝘚𝘲𝘭𝘋𝘉𝘔 𝘗𝘳𝘰𝘥𝘶𝘤𝘵 𝘌𝘷𝘢𝘯𝘨𝘦𝘭𝘪𝘴𝘵 & 𝘚𝘯𝘰𝘸𝘧𝘭𝘢𝘬𝘦 𝘋𝘢𝘵𝘢 𝘚𝘶𝘱𝘦𝘳𝘩𝘦𝘳𝘰 ---- This is why you hear someone say "We do our data modeling in Power BI" - they are modeling their 𝐬𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐥𝐚𝐲𝐞𝐫 in Power BI. This is an image from Power BI, if it had Dark Mode it would look even more similar! #DataModeling #DataArchitecture #SqlDBM

55

1 Comment

Snowflake 🤝 Houseware 🤝 LangChain While building AI agents on Snowflake, we realized how the Langchain ecosystem didn't have established connectors to Cortex LLM models. Nipun & Shubham from the team recently built out a Chat Model integration for Snowflake Cortex inside Langchain's community libraries, and raised a pull request just earlier today(link in comments). This abstraction would allows developers to easily build AI applications on top of Cortex LLMs using the Langchain ecosystem, making Cortex a first-class citizen here onwards. OSS FTW 🚀

59

4 Comments

As the dust starts to settle around the Tabular and Databricks news, a lot of us are left wondering what this means for the future of Apache Iceberg. As someone who has been a part of the data ecosystem for two decades, I think it is important to remember three things: 1️⃣ Iceberg is a community-driven project. Top committers span from a range of companies including Apple, AWS, Alibaba and Netflix. This is in stark contrast to Delta Lake, which is effectively an open sourced Databricks project. Iceberg is not Tabular, and Tabular is not Iceberg.  2️⃣ The next race will be won at the engine-layer. We see Trino—and Icehouse—as the winner. Iceberg was built at Netflix to be queried by Trino. The largest organizations in the world use Trino and Iceberg together as the bedrock for analytics. There’s a reason some of the biggest Iceberg users like Netflix, Pinterest, and Apple all talked about the Trino Icehouse at the Apache Iceberg Summit just three weeks ago. 3️⃣ It will take time. Moving from legacy formats to Apache Iceberg is not an overnight switch and has the potential to become as much of a pipe dream as “data centralization” has been. The community needs platforms that will support their entire data architecture, not just a singular format. We originally founded Starburst 7 years ago to serve this mission when the battle was between Hadoop and Teradata, and the challenges are just as real today. So what’s next… For the first time in the 40+ years of data warehousing history, the entire industry has woken up to the reality that data should be stored in open formats in an object storage lake, specifically Apache Iceberg. Mergers and acquisitions are natural and happen all the time in our space, but these types of architectural re-alignments happen once every 10-20 years in enterprise software. Last, but not least, I want to congratulate my friends Ryan Blue, Daniel Weeks, and Jason Reid on a blockbuster acquisition! Listen to Ryan Blue, Michel Tricot, and I discuss all this and more ➡  https://lnkd.in/eYzNw8v6

711

22 Comments

Super exciting recent SqlDBM update! ✨ The release of Tx - transformational modeling! SqlDBM is the first ever ultimate data modeling platform, combining Relational and Transformational modeling in one, built for Snowflake! ❄️ Message me if you’d like a demo! 🚀 #datamodeling #transformationalmodeling #dataengineering

16

1 Comment

Really interesting read on the evolution of Data Warehouses, Data Lakes and Lake Houses and how Databricks and Snowflake are battling for the top spot. https://lnkd.in/egg4P5VG This is one of the more complete analysis I've seen on the evolution of the analytics space and it's big gorilla's. In the end, it gets into the AI discussion of big LLM's vs. small models and how SLM's will have a place in the analytics world, which I completely agree with.

9

1 Comment

Modern GTM Data should look like modern data for LLMs Harry Stebbings episode with Alexandr Wang a few weeks ago smacked us in the face with this. Alex is the leader in the #AI data space. He said frontier data is what is going to power these models going forward. New data, complex data and data that is created by leveraging our intellectual capabilities in reverse engineering how we as humans would interact with models.  Chain-of-thought prompting is a prompt engineering technique that aims to improve language models' performance on tasks requiring logic, calculation and decision-making by structuring the input prompt in a way that mimics human reasoning. This is required after you apply the reasoning that enables AI models to break down complicated problems into easy-to-handle intermediate steps, leading them to solve complex multi-step reasoning problems. Wait? Didn't I just do this in Clay in a demo two weeks ago? Isn't this all the rage in GTM data? YES - it is. GTM Architecture isn't changing - it has changed. The architect is the most important player right now. We need to reverse engineer how to be successful with GTM data which means we get meetings, we provide value and we get a mutual exchange with our target audience in a consistent and repeatable manner.  You need the basic building blocks to chain these thoughts together: 𝐃𝐚𝐭𝐚: Can't just use the same old data providers and win. Dozens of data sources need to be combined, enriched, verified and 𝐚𝐧𝐚𝐥𝐲𝐳𝐞𝐝 𝐭𝐨𝐠𝐞𝐭𝐡𝐞𝐫. 𝐄𝐱𝐞𝐜𝐮𝐭𝐢𝐨𝐧: This can all be automated. No need to pay someone for outcomes you expect from them when they have to design this manually and at best will take them 40% of their time but they have no experience in this. It isn't their fault! 𝐈𝐧𝐩𝐮𝐭: What do you do to the data that makes it more meaningful? Hiring, spending, technology, patterns, competitive information.  So much more. 𝐀𝐈: Yes. We need #AI and it should be used. However, prompting is a skill and needs to be done right and yes - humans need to be in the loop 𝐄𝐦𝐚𝐢𝐥 𝐒𝐞𝐪𝐮𝐞𝐧𝐜𝐞: You can't rely on the old way of delivering. Need multiple domains, need them rotating, need verified emails and you need a system that runs lights out that you don't support 𝐓𝐫𝐢𝐠𝐠𝐞𝐫: When is the best time to fire my cannon? 𝐑𝐞𝐥𝐞𝐯𝐚𝐧𝐜𝐲: Why does this even matter, at what point, what is the value? 𝐕𝐨𝐥𝐮𝐦𝐞: Less will be more. Shotgun approach and spray and pray looking at open rates and applying the wrong math won't work. A lot has changed. The challenge is fun. Data used in AI systems can teach us GTM folks a lot. When patterns tell you what is going to happen - listen to them.

22

1 Comment

❄ In case you missed it, Snowflake recently released Arctic, a family of performant and compute-efficient Apache 2.0 licensed Large Language Models (LLMs). Curious about how Arctic might fit your needs? Come explore its capabilities at Clarifai alongside the 70+ other LLMs currently available on our platform. Let's build the next groundbreaking GenAI app together! https://lnkd.in/gNz5Babr

20

This week, Snowflake CEO Sridhar Ramaswamy gave the opening keynote at the Snowflake Data Summit 2024 in San Francisco featuring Jensen Huang. They shared a vision for “The era of enterprise AI”. 3 Takeaways 1) Snowflake announced a partnership where NVIDIA software will be able to be run in Snowflake environments. As Jensen said "bringing the best computing power to your data, instead of transferring your data to our compute engines". This will work in tandem with Snowflake's Arctic LLMs. 2) As data grows increasingly complex, keeping AI tools simple becomes more and more paramount to ensure rapid enterprise adoption. Launches like Snowflake Cortex AI Analyst, enable business users to interact with structured data using natural language to find answers faster, self-serve insights and save time. 3) The number of AI use cases is large and seem to be highly industry specific in the enterprise across both internal applications and customer facing experiences. Anu Jain, Global Head of Enterprise Data at J.P. Morgan is using AI to inform to give their frontline teams the tools to anticipate what products their customers are likely to need next. Caitlin Halferty, Global CDO at Ericsson is using AI to model sites their team would work on and determine hazards, costs, and how many workers to send. Thomas Davey, CDO at Booking.com is using AI to reinvent the booking journey with intelligent AI suggestions and itineraries for customers based on hyper-personalization.

60

1 Comment

News: #snowflakeSummit. Open Data Revolution Continues My Analysis: Snowflake’s strategy around Iceberg is a nuanced topic with significant implications. While the simplicity of Snowflake’s compute services is enhanced by Iceberg, the real value lies in how Snowflake chooses to handle interoperability and external engine support. The Polaris news might crucial in defining its position in the ever-evolving data platform market. Polaris is another #opensource gift from Netflix. Implications: Many including us were asking..will Snowflake choose to embrace Iceberg and its open data potential, even if it means disrupting Snowflake’s current model? Looks like the team is going to disrupt themselves to bring in a new era in open data. The technology industry has a saying, especially true during major market shifts: “Better to eat your own before someone else does.” This means that companies should cannibalize their own advantages before competitors do. For Snowflake, this is happening as the fully embracing Iceberg and its open data capabilities before competitors leverage these advantages against them.

4

Imagine seamlessly ingesting incremental data from *any* API right into #DeltaLake via Structured #streaming. This article by Irfan Elahi goes deep into the new Custom Data Source API in #apachespark, showing how you can create your own data source for streaming - with checkpointing capabilities to achieve exactly-once guarantees! https://lnkd.in/gEvr5kQJ

17

Should be a very interesting webinar if you have the time to attend. A look at how to leverage Kumo.AI GNNs to easily enable AI on your data stored in Snowflake.

11

#Kafka is so popular... 13.5k forks, thousands of users! ⁉ But how many of these users have exactly once event processing ⁉ 👉 Within DBOS transact, get exactly once event processing with a single annotation! 👉 You don't need 2 years in CS school to get this distributed system problem right anymore :) #Confluent #nodejs #typescript

13

Attending #DataAISummit next week? Swing by the O'Reilly booth (#127) at the Data+AI Summit on June 12 from 3:15PM - 3:45PM PT for a book signing with me! We're giving away my earlier book "Modern Data Engineering with Apache Spark" but we can talk Delta, Spark, or anything really or all things The Delta Lake Definitive Guide. #opensource #oss #deltalake #freebooks

86

6 Comments

Kyle Hale with a step by step walkthrough of how you teach AI/BI Genie about the semantics of your data.

27

"Every business wants to be an AI business" - I had an opportunity to attend the Data+AI Summit last week and was blown away by the pace of innovation and the business outcomes the companies can drive for their customers. I put down some of my thoughts in this blog post. I couldn't be more excited about our mission and the platform we are building to help our customers in this journey along with our partner Databricks.

21

A super clear, straight forward example of how to do cost effective RAG using LangChain for your document pipeline and Groq for compute resources. The code demonstrates how to speed up RAG and reduce costs using quantized embeddings (reducing 32 bit values to 8) Nice job Sahibpreet Singh! https://lnkd.in/gf5QMCKK

44

2 Comments

Thats the exact explanation I gave to a group of investors last week. (… it’s just Precision/Recall, attributes vectorization as “embedding”, K-Nearest Neighbor, User Search Intent, DAG, ML algorithms, etc all over again …). LOL and they asked what’s new this time around. And I was like not much, ML is still the backbone, and AI is the inference and actions based on the model. It works better when there is a feedback loop in a closed system. But the current state of the art like ChatGPT is based on scraping the Internet (garbage in, garbage out). Kinda like early search engines. What is a better analogy is that NVIDIA/Snowflake etc are selling tools, aka pickaxes to prospectors. The gold is up to you to find. And the opportunity right now is to monetize industry specific (healthcare, travel, financial services) clean data to build models with (aka a treasure map for the prospectors). From an enterprise perspective, what’s out there today aka generative AI isn’t very useful. Too much hype. To build useful engines will require an effort similar to search revolution 2 decades ago. From

14

3 Comments

We recently upgraded to use the BigQuery Storage Write API to stream data into BigQuery. This results in a performance gain of at least 2-2.5x and also lowers customers’ BigQuery costs. The cost reduction comes from Storage Write API being cheaper than the older streaming API and a higher 2TB limit of free streaming per month. Love seeing improvements that make things better, cheaper, faster 🤩 #BigQuery #streaming #dataengineering #datareplication

16

3 Comments