The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Streaming Event
•
0 likes•126 views
The document discusses building a bridge from on-premise Kafka installations to Confluent Cloud. It describes Confluent Replicator, which can replicate data between on-premise and cloud Kafka clusters while protecting business-critical data. Various deployment considerations for Replicator are discussed, including placement, connectivity options, and handling bi-directional replication. Confluent Cloud is introduced as a fully managed Kafka service that reduces operational burden and allows customers to focus on high-value code.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Streaming Event
Similar to The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Streaming Event (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
The Bridge to Cloud (Peter Gustafsson, Confluent) London 2019 Confluent Streaming Event
- 1. 221 Building a Bridge to Cloud Kafka re-engineered for the Cloud Confluent Streaming Event Series – Amsterdam 6th Oct 2019 Peter Gustafsson, Systems Engineer - Confluent Cloud peter.gustafsson@confluent.io
- 2. 222This image cannot currently be displayed. Contents ● The Cloud Shift & Drivers for a Bridge to Cloud ● The Bridge ● Confluent Cloud 101 – Kafka as a Fully Managed Service ○ Architecture, Connectivity & Security ● Bridge Deployment Considerations ● Q&A
- 3. 223This image cannot currently be displayed. The Great “Cloud Shift” A cloud strategy is not complete without a cloud migration story.
- 4. 224This image cannot currently be displayed. Cloud Migration: A one time thing?
- 5. 225This image cannot currently be displayed. In reality, we keep running bi-directionaly over time Our apps are complex, intricately tied together. Some, built to run on legacy, immovable systems. Liberating the data is at the core of the migration. Prepare to have a hybrid environment for a long time.
- 6. 226This image cannot currently be displayed. Customer Feedback – Bridge to Cloud Drivers - Deploy New Use Cases - Developer Velocity - Access to Cloud Ecosystems - Cost Optimization - Mobility
- 7. 227This image cannot currently be displayed. The Bridge - Confluent Replicator Protect business-critical data and projects by replicating down to topic-level configurations Bridge and extend your DataCenter to Cloud or replicate from Cloud to Cloud Confluent Replicator
- 8. 228This image cannot currently be displayed. Architecture Complexity
- 9. 229This image cannot currently be displayed. Reduce Complexity Confluent Cloud
- 10. 230230 Borderless Complete Streaming Platform
- 11. 231 Confluent Cloud 101 - The Fully Managed Kafka Service from the Creators of Kafka Base Facts: Available on AWS, GCP & Azure. Runs in Confluent Account/Subscription/Org-Project. Consumed like a Fully Managed SaaS Service Available on GCP and Azure (soon) Marketplaces. Options: Pay-as-you-go or Commited Spend Options. Standard (shared) & Dedicated Tenant Options. Private & Public Connectivity Options Multi Availability Zone & Support Options. Replicator & Confluent IP Option.
- 12. 232 Why Kafka as a fully- managed service? “Focus Writing High Value Code Supporting the Business, not operating infrastructure.” Deploy in Minutes Ensure service levels Integrate with cloud services Eliminate operational burden
- 13. 233 DATA COMPATIBILITY | Schema registry DEVELOPMENT & CONNECTIVITY | Clients | Connectors | KSQL APACHE KAFKA | Connect | Pub-Sub | Streams ® Database changes Log events IoT events Web events Transformations Custom apps Analytics Monitoring Hadoop Database Data warehouse CRM Confluent Cloud DATA INTEGRATION REAL-TIME APPS Confluent Cloud Kafka re-engineered for cloud MANAGEMENT & MONITORING | Multi-tenancy | Quotas | Usage tracking and billing
- 14. 234 Confluent Cloud Battle tested for massive scale, secure mission-critical pipelines ● Designed by creators of Kafka ● Operated by Kafka committers ● 100+ years of combined Kafka experience ● 80% of Kafka code commits ● Trusted by a third of Fortune 100 companies ● Sub-25ms^ latencies at multi- GBps throughput ● Highly secure with SASL Authentication, Encryption, Private Networking, ACLs, and RBAC* ● 99.95% uptime SLA guarantee ● < 1 hr response time SLA for P1 issues ● Full dev cycle support Delivered by Experts Trusted by Enterprises Backed by Guarantees ^P95 latency *RBAC coming soon to Confluent Cloud
- 15. 235 Confluent Cloud | Singular Kafka focus and innovation Apache Kafka re-engineered for the cloud Confluent Vision for Kafka Global ● Automated disaster recovery ● Global applications with geo-awareness Infinite ● Efficient and infinite data with tiered storage ● Unlimited horizontal scalability for single clusters Elastic ● Faster elastic scaling when adding brokers and partitions
- 16. 236 Confluent Cloud | Connectivity Options Kafka in AWS VPC AWS Transit Gateway Kafka Clients OnPrem Direct ConnectVPC Peering Public Endpoints Kafka Clients in VPC/onPrem Internet VPC/Vnet Peering Kafka Clients in GCP/AWS or Azure VPC VPC Peering
- 17. 237 Where to Deploy Replicator? Replicator is currently a Source Connector, Needs Kafka Connect Workers to run on. Connect Workers are grouped in a Connect Cluster and connected to a Kafka Cluster per config. Recommendation : Dedicated Connect Cluster for Replicator
- 18. 238 Deployment Considerations ● Where do we deploy Replicator? At Source or Destination? …. It depends… ○ Overall recommendation is close to the destination to ensure reliable producer operation. ○ Connectivity & Security – Can we deploy close to destination cluster or will that create firewall / connectivity problems? ● Deploy Replicator in the most Secure Zone – This avoids inbound connections from a less secure zone as this is a hard sell to the security team. ● Bi-Directional replication ○ Use provenance headers to avoid message duplication loops ○ Again..deploy Replicator in the most Secure Zone
- 19. 239 Public Endpoints Internet OnPrem to Cloud Replication over Public Endpoints ● Simple and straightforward no Firewall/Connectivty Complexities ● Deploy dedicated Replicator workers for optimal performance ● For bi-directional replication deploy both replicators inside the firewall. Confluent Platform onPrem R R
- 20. 240 Azure Customer VNet Peering R Cloud to Cloud Replication with VPC peering R AWS / GCP Private Networking Domain ● Deploying Replicator in the most Secure Zone ● For bi-directional replication deploy both replicators inside the most Secure Zone. Internet
- 21. 241 Azure Customer Vnet/VPC R Cloud to Cloud Replication R AWS / GCP ● Deploying Replicator in the most Secure Zone ● For bi-directional replication deploy both replica inside the most Secure Zone. Internet Customer Vnet/VPC
- 22. 242This image cannot currently be displayed. Q&A https://www.confluent.io/resources/bridge-to-cloud ● More reading at docs.confluent.io Try Confluent Cloud – Free $50 credit for the first 3 months
- 23. 243This image cannot currently be displayed.
- 24. 244 eu-north1 Native Geo Replication and Confluent Cloud (Vision) eu-west1 Kafka Native Geo Replication ● Latency Considerations - sub 30-50ms makes pan-eu clusters viable. ● Fast and automatic client fail-over will help RP use cases. ● Does not really solve the bridge-to-cloud use case. ”Stretched” Confluent Cloud Cluster