Five Early Challenges Of Building Streaming Fast Data Applications

This document discusses streaming data between Confluent Cloud and MongoDB Atlas. It provides an overview of MongoDB Atlas and its fully managed database capabilities in the cloud. It then demonstrates how to stream data from a Python generator application to MongoDB Atlas using Confluent Cloud and its connectors. The presentation concludes by providing a reference architecture for connecting Confluent Platform to MongoDB.

This document discusses using the actor model with domain-driven design for reactive systems. It provides an overview of the actor model and how it relates to DDD. Key points include: - The actor model uses message passing between independent units called actors to build distributed, concurrent systems. - The actor model is well-suited for building reactive systems to handle high volumes of asynchronous data and messages. - DDD focuses on modeling the core domain and business logic, while actors and messaging enable rapid implementation in a clear and understandable way. - Together, the actor model and DDD support building systems that can respond quickly to changing business needs through loose coupling and isolation of concerns.

Reactive Streams 1.0.0 is now live, and so are our implementations in Akka Streams 1.0 and Slick 3.0. Reactive Streams is an engineering collaboration between heavy hitters in the area of streaming data on the JVM. With the Reactive Streams Special Interest Group, we set out to standardize a common ground for achieving statically-typed, high-performance, low latency, asynchronous streams of data with built-in non-blocking back pressure—with the goal of creating a vibrant ecosystem of interoperating implementations, and with a vision of one day making it into a future version of Java. Akka (recent winner of “Most Innovative Open Source Tech in 2015”) is a toolkit for building message-driven applications. With Akka Streams 1.0, Akka has incorporated a graphical DSL for composing data streams, an execution model that decouples the stream’s staged computation—it’s “blueprint”—from its execution (allowing for actor-based, single-threaded and fully distributed and clustered execution), type safe stream composition, an implementation of the Reactive Streaming specification that enables back-pressure, and more than 20 predefined stream “processing stages” that provide common streaming transformations that developers can tap into (for splitting streams, transforming streams, merging streams, and more). Slick​ is a relational database query and access library for Scala that enables loose-coupling, minimal configuration requirements and abstraction of the complexities of connecting with relational databases. With Slick 3.0, Slick now supports the Reactive Streams API for providing asynchronous stream processing with non-blocking back-pressure. Slick 3.0 also allows elegant mapping across multiple data types, static verification and type inference for embedded SQL statements, compile-time error discovery, and JDBC support for interoperability with all existing drivers.

When the University of California, San Diego launched its largest investment in tech in 2018, they planned to future proof their business processes and systems. Unexpectedly, it also prepared them to handle a global pandemic that changed every norm for the campus. With shelter-in-place orders taking immediate effect, they needed to quickly set up a robust online learning platform - one with powerful analytics to track student success. And, for the times students and staff are on campus, a contact tracing application was essential for their safety. We’d like to offer a conversation with Scott Lee to tell you more about UC San Diego’s rapid transformation from a traditional, on-campus institution to one of the leading examples of remote learning, and the critical role data connectivity played in making this possible.

In this talk by David Ogren, Enterprise Architect at Lightbend, we draw from experiences helping our clients successfully create, migrate to, and manage cloud-native system architectures. We look at some of the common pitfalls and anti-patterns of modernization efforts, and some of the best practices for taking an incremental approach to transforming legacy systems. See the full post with video on the Lightbend blog: https://www.lightbend.com/blog/microservices-kubernetes-application-modernization

Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and provides a single scalable platform for messaging, storage, and processing. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, allowing certain use cases to still leverage existing tools. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" - replacing some functions but integrating with other existing technologies to provide a bridge to new architectures.

The evolution of micro services architecture. Mainframe, Midrange, Client Server, SOA. Best practices of microservices. Load balancing, BigData, design patterns. When and why to use microservices.

Microservices became the new black in enterprise architectures. APIs provide functions to other applications or end users. Even if your architecture uses another pattern than microservices, like SOA (Service-Oriented Architecture) or Client-Server communication, APIs are used between the different applications and end users. Apache Kafka plays a key role in modern microservice architectures to build open, scalable, flexible and decoupled real time applications. API Management complements Kafka by providing a way to implement and govern the full life cycle of the APIs. This session explores how event streaming with Apache Kafka and API Management (including API Gateway and Service Mesh technologies) complement and compete with each other depending on the use case and point of view of the project team. The session concludes exploring the vision of event streaming APIs instead of RPC calls.

This document discusses Scala as a language for fast data and architectures for fast data systems. It provides an overview of Scala's advantages for fast data including its JVM compatibility, type safety, concise syntax, and support for functional programming. It also discusses why Scala is preferable to other languages like Java, Python, Go, and C++ for fast data workloads. The document outlines some of the tradeoffs involved in architecting systems for fast data and emphasizes approaches like isolation, event-driven data management, and "ACID v2" to build scalable fast data systems.

Microservice architecture (MSA) helps address modern software development needs by breaking applications into independently deployable microservices that communicate asynchronously. MSA evolves past SOA approaches by focusing on small, single-purpose services rather than large monolithic applications. Key aspects of MSA include ubiquitous language, well-defined boundaries, independent deployability, asynchronous communication via messaging, and each service owning its own data. MSA also relies on modern cloud-ready infrastructure approaches like service discovery, elastic scaling, and failure resilience at the service level rather than relying on specific hardware.

This document discusses iRobot's adoption of serverless architecture and the reasons for choosing it. Some key benefits identified are lower latency and cost compared to monolithic or microservices architectures. Specific challenges addressed by serverless include deployment, service discovery, and security. While serverless addresses many issues, the document notes there is still room for improvement from cloud providers in areas like deployment models and integration testing.

Apache Kafka is changing the way we build scalable and highly available software systems. Providing a simplified path to eventual consistency and event sourcing Kafka gives us the platform to make these patterns a reality for a much broader segment of applications and customers than was possible in the past. Cloud Events is an interoperable specification for eventing that is part of the CNCF. This session will combine open source and open standards to show you how you can build highly reliable application that scale linearly, provide interoperability and are easily extensible leveraging both push and pull semantics. Concrete real world examples will be shown of how Kafka makes event sourcing more approachable and how streams and events complement each other including the difference between business events and technical events.

This webinar by Orkhan Gasimov (Senior Solution Architect, Consultant, GlobalLogic) was delivered at Java Community Webinar #3 on October 16, 2020. During webinar we had simplified overview of classical and modern architecture patterns and concepts that are used for development of distributed applications during the last decade. More details and presentation: https://www.globallogic.com/ua/about/events/java-community-webinar-3/

Technical thought leadership presentation to discuss how leading organizations move to real-time architecture to support business growth and enhance customer experience. This is a forum to discuss use cases with your peers to understand how other digital-native companies are utilizing data in motion to drive competitive advantage. Agenda: - Data in Motion with Event Streaming and Apache Kafka - Streaming ETL Pipelines - IT Modernisation and Hybrid Multi-Cloud - Customer Experience and Customer 360 - IoT and Big Data Processing - Machine Learning and Analytics

This document discusses how Confluent Platform provides elastic scaling for Apache Kafka. It offers fully managed cloud services through Confluent Cloud or self-managed software. Confluent Cloud allows users to easily scale Kafka workloads from 0 MBps to GBps without complex provisioning. It also offers pay-for-use pricing where customers only pay for the data streamed, with the ability to scale to zero. For self-managed deployments, Confluent Platform enables dynamic scaling of Kafka clusters on Kubernetes through features like tiered storage and self-balancing clusters that can rebalance partitions in seconds versus hours for other Kafka services.

https://www.updateconference.net/en/2019/session/a-guide-through-the-azure-messaging-services A guide through the Azure Messaging services - Update Conference

Apache Kafka is critical to PayPal's analytics platform. It handles a stream of over 20 billion events per day across 300 partitions. To democratize access to analytics data, PayPal built a Connect platform leveraging Kafka to process and send data in real-time to tools of customers' choice. The platform scales to process over 40 billion events daily using reactive architectures with Akka and Alpakka Kafka connectors to consume and publish events within Akka streams. Some challenges include throughput limited by partitions and issues requiring tuning for optimal performance.

This document discusses cloud-native data and patterns for managing data in microservices architectures. It describes using data services and APIs to interface with existing data sources. Patterns like caching data at the edge with various caching strategies are discussed. The document also covers using multiple small databases with each microservice rather than a shared database. Event sourcing and CQRS patterns are presented as ways to integrate data across services. Finally, the impact on roles like database administrators is considered in cloud-native data environments.

<November 2017 Updated from earlier presentations on Cloud-native Data> Cloud-native applications form the foundation for modern, cloud-scale digital solutions, and the patterns and practices for cloud-native at the app tier are becoming widely understood – statelessness, service discovery, circuit breakers and more. But little has changed in the data tier. Our modern apps are often connected to monolithic shared databases that have monolithic practices wrapped around them. As a result, the autonomy promised by moving to a microservices application architecture is compromised. What we need are patterns and practices for cloud-native data. The anti-patterns of shared databases and simple proxy-style web services to front them give way to approaches that include use of caches (Netflix calls caching their hidden microservice), database per service and polyglot persistence, modern versions of ETL and data integration and more. In this session, aimed at the application developer/architect, Cornelia will look at those patterns and see how they serve the needs of the cloud-native application.

This document discusses whether and how to build microservices. It includes: 1) Presentations by Sanjay Goil, VP of Product Management at Oracle, and Paul Parkinson, Cloud Platform Dev Lead at Oracle on microservices and building a sample microservices application. 2) Recommendations from Oracle ACEs Guido and Rolando on microservices approaches and modernizing existing SOA architectures for microservices. 3) A discussion of how a converged database can simplify building microservices by supporting messaging, multiple data types, and cloud services. 4) A demo of building a microservices application for a food delivery app using technologies like Helidon and a converged database.