Dusty Chadwick

Watch this Snowflake space folks. 👀 As a once “old skool” IT architect, I spend A LOT of time thinking about security for my company - despite having one of the most narrow threat surfaces I’ve ever been responsible for. But admittedly, I’ve felt jealous in the past when SaaS data platform vendors have touted their “secure” data solutions as a panacea to the needs of IT security, when in my own simple setup I always I could always find another layer that could be added to improve my employees and our clients security positioning. I’ve often thought to myself: “I must be missing something because the controls for that service can’t be that easy. I MUST be overthinking my own company’s security”. Then I read about the recent Ticketmaster/LiveNation and Santander issues and think, “No it really wasn’t as easy and secure as they said it was”. So again, watch this Snowflake space folks. 👀

8

Last year at the EQT Ventures - Stockholm Data Summit, I had the opportunity to present how EQTs AI team, Motherbrain Labs, utilizes DBT for more than just data transformation. I showcased how we integrate AI, external tools, and APIs more closely with our data warehouse directly in DBT. This workflow streamlines our processes and accelerates the merging of external and internal data. The key advantage? It positions our data/AI team closer to leadership, enabling us to tackle more strategic questions that often require a sophisticated mix of both internal and external information in a more iterative and efficient manner. Previously, our workflow depended on a sizable "always running" compute cluster. Today, thanks to the efforts of Sebastian Stan from Motherbrain Labs and Ernesto Ongaro from DBT, with significant support from the team at Snowflake, we have open-sourced this workflow into the DBT core setup for Snowflake - transitioning us away from a bespoke cluster to a model that is open source and much more cost-effective. If you're interested in learning more or starting to implement it in your data teams, we've co-authored a blog post with DBT that details the functionality. I'm eager to see how other teams using DBT and Snowflake will take advantage of these new "external EL capabilities." https://lnkd.in/e9s4CwUg

51

9 Comments

Interesting discussion with Dave Farley and Emily Bache. There are a lot of things that I agree with in it, however the one main sticking point that I have is that a true "engineer" in the more common incarnations, Mechanical, Civil, etc, require registration and licensing from a local state/district and/or governing body. I'd LOVE for programming to have such a system, but we currently don't. That's one reason why I settle with code craft and call myself a crafter. I agree it's not a perfect term, but to me its better than giving the false impression that there's a standard of skillset or understanding like other engineering disciplines. What do you think? https://lnkd.in/gXqmkden

15

31 Comments

There's a big difference between Database Modeling and Semantic Layers- How would you explain it? Even though they look alike, the underlying meaning is vastly different. ▶ The type of data modeling you would do with database modelers like SqlDBM and semantic layers are interrelated in that we help design and organize database structures in a way that supports the creation of a semantic layer. However: ▶ A database modeling solution is focused on the 𝐩𝐡𝐲𝐬𝐢𝐜𝐚𝐥 and 𝐥𝐨𝐠𝐢𝐜𝐚𝐥 aspects of database modeling, ensuring the data structures align with technical requirements and business concepts. ▶ A semantic layer operates at a higher level of abstraction often within 𝐁𝐈 𝐭𝐨𝐨𝐥𝐬 such as PowerBI, Looker, ThoughtSpot, and Tableau. It maps the technical details of data that were designed by tools like SqlDBM or not properly designed at all, to business-friendly terms and concepts. ▶A tool like SqlDBM will essentially lay the 𝐟𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐝𝐚𝐭𝐚 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 that a semantic layer translates into an easily navigable and understandable format for end-users. ---- "Enabling non-technical business users to perform data analysis without needing to understand complex database queries or structures. However, leveraging a data modeling tool and having a good data model will make the complexity of building and running a semantic layer more manageable and efficient." - Keith Belanger, 𝘚𝘲𝘭𝘋𝘉𝘔 𝘗𝘳𝘰𝘥𝘶𝘤𝘵 𝘌𝘷𝘢𝘯𝘨𝘦𝘭𝘪𝘴𝘵 & 𝘚𝘯𝘰𝘸𝘧𝘭𝘢𝘬𝘦 𝘋𝘢𝘵𝘢 𝘚𝘶𝘱𝘦𝘳𝘩𝘦𝘳𝘰 ---- This is why you hear someone say "We do our data modeling in Power BI" - they are modeling their 𝐬𝐞𝐦𝐚𝐧𝐭𝐢𝐜 𝐥𝐚𝐲𝐞𝐫 in Power BI. This is an image from Power BI, if it had Dark Mode it would look even more similar! #DataModeling #DataArchitecture #SqlDBM

55

1 Comment

Imagine seamlessly ingesting incremental data from *any* API right into #DeltaLake via Structured #streaming. This article by Irfan Elahi goes deep into the new Custom Data Source API in #apachespark, showing how you can create your own data source for streaming - with checkpointing capabilities to achieve exactly-once guarantees! https://lnkd.in/gEvr5kQJ

17

Ever join a Data/Analytics team without proper documentation? How about one where the data was 95% unstructured JSON blobs? 🙋 Yup me toooooo. I was very surprised when the conventional wisdom was: ⏳ SELECT * from events where... LIMIT 10 😰 Copy and paste individual json_blob to a JSON prettifier 🧠 Try and remember for next time. 🙅🙅🙅🙅🙅🙅🙅 Not anymore. With AutoDocs, you get a fully indexed, searchable and up to date reference for your event schemas. No special SDKs or heavy pipelines required. Want to learn more? Let me know! https://lnkd.in/eSeMCYUv

17

OpenAI Spring Update includes a new Omnimodel release for free: GPT-4o - The voice interaction is pretty fluid - The desktop and coding help was great - The live video interaction is a great new addition - The live translation is one of my favorites Looking forward to building some cool applications inside and outside of work with this new model 😃 https://lnkd.in/eknP8v75 #openai #llm #omnimodel #multimodal #text #speech #image #video #coding #translation #datascience #deeplearning #machinelearning #ai

4

Whoah. Semantic Metadata is cool again! I guess it's time for me to dust off my "Semantic Integration with Stanbol and Jena" presentation and start delivering it again. It's too bad that the Stanbol project was retired at the ASF though. As were Marmotta and Any23 if memory serves correctly. I might seriously consider starting forks of one or all of those. And in any case, Jena / Fuseki are still going strong. Which is mondo righteous! #ApacheJena #SemanticWeb #Metadata #RDF #Ontology #KnowledgeGraph #OWL #SPARQL

2

There's a growing narrative that RAG will become obsolete due to long context windows, but I strongly disagree. Here’s why: - Metadata filtering (like dates and view counts) is essential in many practical applications. - The "R" in RAG is increasingly agentic, making critical decisions about source selection and data aggregation. - Longer context windows don't negate the benefits of filtering out irrelevant documents for better results. RAG is here to stay because even if we can stuff the entire context with 10 million tokens, the LLM will always find the answer more efficiently if you can narrow the input.

27

1 Comment

I presented about Apache XTable (Incubating) & how it brings interoperability to #lakehouse table formats at Dremio's Subsurface. Check out the recording of the talk in comments! #dataengineering #softwareengineering

49

3 Comments

Super exciting recent SqlDBM update! ✨ The release of Tx - transformational modeling! SqlDBM is the first ever ultimate data modeling platform, combining Relational and Transformational modeling in one, built for Snowflake! ❄️ Message me if you’d like a demo! 🚀 #datamodeling #transformationalmodeling #dataengineering

16

1 Comment

The term “Developer Ecosystem” has been rolling around in my brain lately, so let’s discuss. Here’s the problem I’m noticing. The term #developerexperience is experiencing a similar problem to the term #technicaldebt. It correctly explains the problem for folks who have experience working in software and organizational systems that are inefficient and ineffective. The struggle of just getting good work out the door is real, and we know how we could make things better. We get it, so using these terms feels good because we finally have a phrase that describes a more abstract concept that we feel is important. But the value we feel in using these terms in our familiar domain doesn’t quite translate to people who don’t have experience working as or with software developers. The meaning gets lost in translation and the people who have the most influence into what organizational initiatives get funded can’t conceptualize the benefit. As I’m observing the trends in the #devex movement, I’m seeing that metrics like DORA, SPACE and the DevEx Model are highly focused on an *integrated* experience, not an *individual* experience. It’s about ensuring developers are working in concert with the organizational system instead of fighting it. The struggle is often strong and significant, but it’s easy to think it’s trivial if you haven’t been there. This nuance is pretty hidden, so it’s understandable when it gets glossed over or misinterpreted. I think this is why folks in other domains think about DevEx as emphasizing the productivity of individuals or assuming that devs want to be pampered and treated better than their colleagues. I’m curious if framing the overall problem as ecosystem health would help influence change. Would CEOs and CFOs grasp and fund initiatives more? My hunch is yes after so many years of selling the value of modernizing legacy systems, but I’m curious to hear your thoughts.

10

10 Comments

What's your tolerance between "just do something manually" vs "automate this" for a repeating process? Always feels like that fine line between "I should write a script to do this" (which could take minutes to hours to days) and "well it's easy to get the data and the ROI isn't worth it". I think as engineers we default to thinking that the coding part is more "fun" even if it takes longer. I think it also down to the nuances of the data you're pulling. * Does it require (additional) logic? * Is there _future_ benefit from having this (able to categorize it/additional insights) * Making it available for others to use * Learning experience I don't think there's a wrong answer (well... maybe if you spend 2+ weeks on a 5 minute manual task) #automation #manual #tasks #roi #software

12

16 Comments

Just going to leave this here. If you're in search and not exploring semantic/multimodal search, you're going to get lapped by your competitor. Want to be blown away, shoot me a DM and we'll put together a custom demo/index with your data and show you instant improvements to relevance - because it's just that easy to build custom demos with Marqo. #vectorsearch #embeddings #AI

42

Buried lately so a bit tardy with my monthly EAI newsletter here on LI, but it's coming along. Thought I would share a clip that may be if interest--> As the Big Techs grew rapidly over the past 25 years, so did their R&D budgets, so it should be no surprise that most of the world’s R&D spend in advanced IT revolves around the strategic interests of a small group of Big Tech companies. In 2022, for example, Amazon, Alphabet, Meta, Apple and Microsoft collectively spent $202 billion in R&D, with Amazon leading at over $73B followed by Alphabet at nearly $40B. The next five biggest R&D spenders listed on NASDAQ only spent only $20B, which includes Nvidia at $7.3B. For perspective, the entire R&D budget for the U.S. Federal Government in 2022 was about $180B across all agencies. So, when people describe Big Techs as sovereigns (as in nations), the tech oligopoly is actually larger than the largest economy in terms of R&D, and it's of course invested in their strategic interests to protect moats, avoid disruption, and continue to grow. LLM chatbots are the latest and perhaps greatest example of extreme behavior in strategic investments by Big Tech companies.

1

🎉 Mixpanel just dropped over a dozen features without adding complexity to our interface. We did this by building what we think of as "legos for analytics". In this post, I'll share a bit about our team's journey and what it enables for customers. Like all products, Mixpanel started out simple. Complexity crept in over the years as a natural result of shipping fast. 2 years ago, we hit a breaking point. Do we refuse feature requests to keep the product simple? Or squeeze them in, earn the short-term revenue, and make our product more complex? Turns out, there's another way. We could rethink our core concepts to make the product both more powerful AND simple. This is harder in the short-term, but pays off in the long-term. The first step was boiling Mixpanel down to its core concepts: - Events: Something a user does, like a Signup or Purchase. - Behaviors: A sequence of events, like a funnel (A -> B -> C) or retention (A -> A a week later) - Metrics: A way to count behaviors. eg: unique users, unique sessions, conversion rate, property sum, frequency, … - Visualizations: A way of plotting metrics. We realized we could address many of our feature requests by making these concepts into composable blocks in our product: simple to understand, but infinitely flexible when composed together. Like legos, but for analytics! Thus began a year-long journey to unlock the full power of these legos in Mixpanel. While we had a grand vision, we shipped incrementally to derisk it and unlock value for customers along the way. Here's just a small sample of what we shipped: 1. More powerful funnels: https://lnkd.in/gGUq2bZK 2. Revenue-weighted funnels: https://lnkd.in/g7JBer3q 3. Mix and match metrics in a chart: https://lnkd.in/gmXxgSj4 4. Standardized metric and behavior definitions: https://lnkd.in/g9Z7yucD 5. Behavioral segmentation to find "aha" moments: https://lnkd.in/gr-eFh7M These changes are now live for 100% of customers. Crucially -- we didn’t have to add any new concepts to make all this power possible; these features just “emerged” from making our existing concepts work together. This not only unlocks power for customers, it gives us a strong foundation on which to ship even more, faster. Massive kudos to the team who built this -- this is the kind of hard engineering and design work that creates leverage! Maverick Lee, Rajiv Kadaba, Ryan Evan Davis, Zihan (Hans) Li, Celia Choy, Justin Lottes, Akshay Suggula If you haven't used Mixpanel in the last few years, give it a spin and let us know what you think. It takes only 5 minutes and 1 line of code to get started.

310

25 Comments

🙋 Quick question: I used to run a directory of data tools but it became to cumbersome to maintain due to acquisitions and companies shutting down. Should I revive it? It still exists here: https://lnkd.in/g56dgnCF

4

14 Comments