Dagster

We recently shipped a new Asset Health dashboard in Dagster+. It’s a single place for data platform teams to keep tabs on the state of every asset across their organization. It's a powerful new tool to monitor execution failures, data quality problems, and freshness issues. As data orchestration becomes more asset-oriented and event-driven, the classic pipeline timeline view becomes less useful for monitoring. It helps you understand when things ran, but doesn’t paint a clear enough picture of the current state of the data assets you’ve built. For example, you probably don’t care much about failures that automatically recovered; but you DO care about datasets that are overdue or failing critical quality checks. You might set up alerts for this, but they often end up spamming a Slack channel that half the team has already muted. Asset Heath is designed to be the first place a data team goes on Monday morning. Its purpose is to strip away the noise, and help data practitioners quickly discover and diagnose issues with the datasets they’re responsible for. It's available now for Dagster+ customers.

Introducing our new blog series, Dagster Power User! 🚀 Every Friday for the next few weeks, on our technical blog Indicium Engineering on Medium, our data engineer Edson Nogueira will be hosting a series of posts about his experience using Dagster. In this first text, dive into how Dagster with dbt can revolutionize the Modern Data Stack, enabling technical and business perspectives to align on the tangible value that modern data platforms provide for stakeholders. Click the link to read the article:  https://lnkd.in/ePkYTt7i And stay tuned for the complete series to get valuable insights. You can also check Edson’s recent contributions to the Dagster open-source project on Git-Hub. Check here for the latest release: https://lnkd.in/eQvaZMCp —---- Apresentando a nova série da Indicium, Dagster Power User! 🚀 Nas próximas semanas, às sextas-feiras, em nosso blog técnico Indicium Engineering no Medium, Edson Nogueira, nosso data engineer, vai compartilhar uma série de artigos sobre sua experiência com o uso do Dagster. Neste primeiro texto, mergulhe em como o Dagster com dbt pode revolucionar a Modern Data Stack, permitindo que perspectivas técnicas e de negócios se alinhem sobre o valor tangível que as plataformas de dados modernas fornecem para as partes interessadas. Clique no link para ler o artigo:  https://lnkd.in/ePkYTt7i E acompanhe a série completa para obter insights valiosos. Você também pode conferir as contribuições recentes do Edson para o Dagster no GitHub. Veja aqui o último release: https://lnkd.in/eQvaZMCp #Dagster #dbt #indicium #ModernDataStack #data #dados #plataformadedados

Elevate your Snowflake Summit experience with an unforgettable after-hours gathering hosted by us and our partners at Monte Carlo, Atlan, Satori, Dagster Labs and dbt Labs! Join us for immersive experiences, networking, and good vibes. RSVP now! #SnowflakeSummit #DataInnovation #AfterHoursParty 

"How to Fine-Tune LLMs in 2024 with Hugging Face", but with Dagster, Modal and Llama3! 👉https://lnkd.in/g5cFggsD A couple of months ago Philipp Schmid released a blog post "How to Fine-Tune LLMs in 2024 with Hugging Face" [https://lnkd.in/gYtu-QJ4]. I found it very nice and useful, but after using it repeatedly, I decided to make it reusable, reproducible, and easier to start with! It's an expansion of the original blog post, augmented with Docker, Inc, Dagster, and Modal Labs, and of course, based on Llama3. The full code, Docker image, and model checkpoint are available. #LLM #ML #LLMOps #MLOps #infrastructure #GPU #Llama3 #LLMfinetuning

10 days to go to the April 2024 cohort of Data Engineer Camp! It's going to be the fifth cohort we run with some significant changes to the curriculum: - Swapped Airflow in favor of Dagster to teach the concept of Data Orchestration. We find the asset-oriented approach advocated by dagster to be highly compelling, especially for companies that operate at scale. More on the asset orchestration thesis here: https://lnkd.in/gA_ZbmdE - Deeper dive into OLAP performance architecture of big data processing systems like Snowflake and Spark, where we take a closer look at concepts like Columnar File Storage, Pruning and Clustering. - Improved our Fundamental ETL with Python and SQL teaching codebase to make heavier use of Object-Oriented Programming, Unit Testing and Containerization to resemble codebases you'll find at big tech companies and open-source projects. See our detailed curriculum here: https://lnkd.in/gMXa29kZ There's only 9 seats remaining in the next cohort as we like to keep class size to a maximum of 30 students to keep our quality of teaching as high as possible. Find out more about our students' capstone projects by signing up to the demo day here: https://lnkd.in/gJs66e3x

I know I'm super late to this, but Dagster + dbt Labs is a match made in heaven. I spent some time this week migrating all of my dbt jobs to Dagster Cloud. Dagster's dbt integration made this a billion times easier than it should have been. I now have a level of visibility into every job's model dependencies that I've never had before. Lots more to say on this but I'll keep it short and sweet for now!

Happening Now! We are live with Dagster talking about how to develop locally and ship to production with confidence! Don't miss it.

Så blev det officielt nu er vi Dagster Partnere i Neurospace. Det bliver vi efter at have arbejdet med Dagster gennem de sidste 1.5 - 2 år sammen med vores partnere og vi benytter det også til vores interne data platform kaldet Mission Control. Det er ret spændende alt det Dagster Labs har gang i og Dagster+ gør det kun endnu mere spændende. Jeg tror der er rigtig mange, særligt her i Danmark, som kunne få meget ud af at gentænke jeres ETL setups, hvis ikke for at få omkostningerne ned så for at træde ind i en ny era med så mange nye kapabiliteter såsom data kontrakter, data tests, auto materialization, partitioner, og data friskhed! Så kan du selv vælge om du vil have Open Source versionen eller betale lidt for at få ekstra muligheder og slippe for lidt af besværet. Særligt er der nogle rigtig spændende tanker i at lade orkestrateren stå for både Data katalog og Cost management. Det får man dog kun med Dagster+. Hvis du synes det lyder spændende, så ved du hvor du kan finde os og vi tager altid gerne en dialog eller PoC!

Thank you to everyone who joined us on Friday for our event celebrating the next generation of Dagster Cloud. We had a blast. There's still time to register for the big launch, so be sure to save your spot now. Dagster+ is only days away! Join us on the 17th here: https://bit.ly/3xMPGVu

Mark your calendars for the Dagster+ launch event!