Leveling Up: SciPlay's Journey with Managed Airflow in Gaming Data Orchestration

SciPlay, a leading developer and publisher of digital games on mobile and web platforms, faced significant challenges as it sought to expand its global portfolio of games. As part of its growth strategy, SciPlay required a robust solution to manage and expand its data operations efficiently, reduce risks associated with data handling, and empower its data teams to innovate and scale their operations. The solution came in the form of Astro, Astronomer’s fully-managed platform to take Apache Airflow® to the next level.

Initial Challenges and Strategic Implementation

One of the primary challenges SciPlay faced in adopting Airflow was gaining traction outside of the initial data team. Airflow was initially deployed as an orchestration tool for ETL and data platform operations, with the vision of expanding its use across various departments to handle the business’s data needs across multiple gaming products. Yohai Vidergor, Vice President of Global Data at SciPlay highlighted the broader vision behind adopting Airflow, noting, “We started using it as an orchestration tool for our ETL and data platform. The goal was to establish Airflow as an enterprise orchestration tool for everything we do, given the diverse needs across about 10 different games, each with its own department. We also have a marketing department with its own R&D, and all of them need robust data integrations.”

Despite this initial hurdle in achieving broader departmental adoption, the strategic implementation gradually demonstrated Airflow’s benefits across multiple domains. This initiative resulted in four departments successfully adopting Airflow, marking a significant milestone in SciPlay’s data orchestration journey.

The transition was initially challenging for the engineering teams, who were not traditionally oriented towards data or familiar with using tools like Airflow. “Our Engineering teams are not data-oriented and not using Airflow,” says Yohai. “But once they saw how flexible it was and how easy it was to onboard, they quickly got the hang of it and just ran with it. Now, they are actually using it more than the data teams in terms of usage.”

With Astro, we built a specific piece of code that triggers extracts in Tableau only once the ETL processes are complete. This means no more guessing on timings and no more unnecessary delays.

Yohai Vidergor, VP of Global Data at SciPlay

Goals and Solutions Provided by Astro

With Airflow proven as the right and valuable solution, SciPlay looked to a managed service that would allow them to achieve greater scale of Airflow’s benefits, across their 10 different games and many engineering and data teams. SciPlay immediately understood the value of streamlining operations and empowering its DevOps team; this eliminated the need for in-house deployment and maintenance, freeing up valuable resources. “One of the goals was to not manage our own Airflow deployment and instead use a managed service.” explains Yohai.

Employing Astro’s comprehensive feature set, SciPlay strategically fostered broader adoption across teams. This proactive approach empowered SciPlay to achieve significant scalability, efficiently growing from managing tens of thousands to over 100,000 tasks per month. Astro’s reliability, coupled with its managed service aspect, and ease of use, instilled widespread confidence across SciPlay’s teams, making it the go-to solution for data orchestration.

Additionally, a key priority was migrating gaming data to a robust and scalable data platform. Airflow’s ability to handle complex dependencies and workflows, a limitation with previous tools, made this transition seamless.

Yohai explained, “We basically built a new data platform, migrating all of our games to this platform that we built. We wanted a tool we could rely on with the flexibility to write code, manage dependencies, and orchestrate very complex processes which were more complicated to achieve with the tools we previously used.”

Expanding and Scaling with Astro

Astro’s flexibility and scalability empowered SciPlay to accelerate onboarding for new teams. The Central Technologies and Ad Tech groups quickly leveraged Astro to establish efficient pipelines, seamlessly integrating with ad providers and third-party platforms. This resulted in a significant expansion of data pipelines and processing volume, where Astronomer’s robust support and Airflow’s ability to manage complex dependencies flawlessly facilitated this remarkable growth.

The inherent flexibility of Astro has proven to be a strategic advantage in workforce management as well. “Astro gives you more flexibility and more efficiency, and makes you less dependent on hiring specific people with specific skill sets,” explains Yohai. This aspect of Airflow has enabled teams to tackle sophisticated tasks without the need to expand their teams, further enhancing operational agility.

Furthermore, the adoption of Astro fueled the development of innovative data science workflows and real-time data pipelines. These advancements directly impacted game user experience and engagement, showcasing SciPlay’s strategic use of data to drive continuous innovation across its gaming portfolio.

It’s incredibly easy to set up new pipelines; I can create 10 new ones a day if needed. This agility has allowed us to scale up to the point where we needed to create specific deployments for larger teams to better manage and measure their usage.

Yohai Vidergor, VP of Global Data at SciPlay

Outcomes and Business Impact

By proactively implementing Astro and expanding its use across teams, SciPlay has achieved a transformative leap in their data strategy: with one central data platform that accelerates the innovation across teams, while achieving operational efficiency at scale. This strategic move empowers the teams to seamlessly onboard new games, explore new data-driven opportunities, and optimize game development processes, all propelling sustained business growth and success.

Sciplay chose Astro for its robust capability to manage critical data workflows crucial for powering and updating real-time, interactive dashboards in Tableau, which are essential for game performance analysis and marketing optimization. These dashboards are critical as they enable real-time decision-making and significantly affect the company’s operational and strategic outcomes.

“We used to have a major bottleneck with Tableau, especially when dealing with hundreds of extracts running simultaneously,” mentions Yohai. “With Astro, we built a specific piece of code that triggers extracts in Tableau only once the ETL processes are complete. This means no more guessing on timings and no more unnecessary delays.”

This customization not only optimized the timing of data availability but also spread out the exact usage, significantly reducing the load on their systems and alleviating the previously common bottlenecks. This adaptive approach showcases how Astro’s flexibility has enabled SciPlay to develop tailored solutions for complex challenges, enhancing operational efficiency and data handling capabilities.

Moreover, the ease of setting up new pipelines with Astro has dramatically increased productivity within SciPlay’s teams. “It’s incredibly easy to set up new pipelines; I can create 10 new ones a day if needed. This agility has allowed us to scale up to the point where we needed to create specific deployments for larger teams to better manage and measure their usage,” Yohai added.

Conclusion

SciPlay’s forward-thinking adoption of Astro has transformed its data orchestration strategy. This proactive move enables the company to effectively manage and expand its global gaming portfolio while simultaneously reducing operational risks and empowering its data teams. Astro’s flexibility, scalability, and reliability have become cornerstones to SciPlay’s success, showcasing the transformative power of well-implemented data orchestration tools to drive business growth and innovation in the competitive digital gaming landscape.