From Zero to Hero with Kafka Connect
•
5 likes•9,359 views
Watch this talk here: https://www.confluent.io/online-talks/from-zero-to-hero-with-kafka-connect-on-demand Integrating Apache Kafka® with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren't working. This talk will discuss the key design concepts within Apache Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We'll do a live demo of building pipelines with Apache Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we'll go hands-on in methodically diagnosing and resolving common issues encountered with Apache Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Apache Kafka Connect in containers.
Report
Share
From Zero to Hero with Kafka Connect
- 1. From Zero to Hero with Kafka Connect @rmoff A practical guide to becoming l33t with Kafka Connect a.k.a.
- 2. @rmoff From Zero to Hero with Kafka Connect What is Kafka Connect?
- 3. From Zero to Hero with Kafka Connect @rmoff Sources Streaming Integration with Kafka Connect Kafka Brokers Kafka Connect syslog
- 4. From Zero to Hero with Kafka Connect @rmoff Streaming Integration with Kafka Connect Kafka Brokers Kafka Connect Amazon S3 Google BigQuery Sinks
- 5. From Zero to Hero with Kafka Connect @rmoff Streaming Integration with Kafka Connect Kafka Brokers Kafka Connect syslog Amazon S3 Google BigQuery
- 6. From Zero to Hero with Kafka Connect @rmoff { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url": "jdbc:mysql://asgard:3306/demo", "table.whitelist": "sales,orders,customers" } https://docs.confluent.io/current/connect/ Look Ma, No Code!
- 7. From Zero to Hero with Kafka Connect @rmoff Streaming Pipelines RDBMS Kafka Connect Kafka Connect Amazon S3 HDFS
- 8. From Zero to Hero with Kafka Connect @rmoff KafkaConnect Writing to data stores from Kafka App Data Store
- 9. From Zero to Hero with Kafka Connect @rmoff Evolve processing from old systems to new RDBMS Existing App New App <x> Kafka Connect
- 10. @rmoff From Zero to Hero with Kafka Connect Demo http:!//rmoff.dev/kafka-connect-code
- 11. @rmoff From Zero to Hero with Kafka Connect Configuring Kafka Connect Inside the API - connectors, transforms, converters
- 12. From Zero to Hero with Kafka Connect @rmoff Kafka Connect basics KafkaKafka ConnectSource
- 13. From Zero to Hero with Kafka Connect @rmoff Connectors KafkaKafka ConnectSource Connector
- 14. From Zero to Hero with Kafka Connect @rmoff Connectors "config": { [...] "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector", "connection.url": "jdbc:postgresql://postgres:5432/", "topics": "asgard.demo.orders", }
- 15. From Zero to Hero with Kafka Connect @rmoff Connectors Connect Record Native data Connector KafkaKafka ConnectSource
- 16. From Zero to Hero with Kafka Connect @rmoff Converters Connect Record Native data bytes[] KafkaKafka ConnectSource Connector Converter
- 17. From Zero to Hero with Kafka Connect @rmoff Serialisation & Schemas -> Confluent Schema Registry Avro Protobuf JSON CSV https://qconnewyork.com/system/files/presentation-slides/qcon_17_-_schemas_and_apis.pdf
- 18. From Zero to Hero with Kafka Connect @rmoff The Confluent Schema Registry Source Avro Message Target Schema RegistryAvro Schema Kafka Connect Kafka ConnectAvro Message
- 19. From Zero to Hero with Kafka Connect @rmoff Converters key.converter=io.confluent.connect.avro.AvroConverter key.converter.schema.registry.url=http://localhost:8081 value.converter=io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url=http://localhost:8081 Set as a global default per-worker; optionally can be overriden per-connector
- 20. From Zero to Hero with Kafka Connect @rmoff What about internal converters? value.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter=org.apache.kafka.connect.json.JsonConverter value.internal.value.converter=org.apache.kafka.connect.json.JsonConverter key.internal.value.converter.bork.bork.bork=org.apache.kafka.connect.json.JsonConverter key.internal.value.please.just.work.converter=org.apache.kafka.connect.json.JsonConverter
- 21. From Zero to Hero with Kafka Connect @rmoff Single Message Transforms KafkaKafka ConnectSource Connector Transform(s) Converter
- 22. From Zero to Hero with Kafka Connect @rmoff Single Message Transforms "config": { [...] "transforms": "addDateToTopic,labelFooBar", "transforms.addDateToTopic.type": "org.apache.kafka.connect.transforms.TimestampRouter", "transforms.addDateToTopic.topic.format": "${topic}-${timestamp}", "transforms.addDateToTopic.timestamp.format": "YYYYMM", "transforms.labelFooBar.type": "org.apache.kafka.connect.transforms.ReplaceField$Value", "transforms.labelFooBar.renames": "delivery_address:shipping_address", } Do these transforms Transforms config Config per transform
- 23. From Zero to Hero with Kafka Connect @rmoff Extensible Connector Transform(s) Converter
- 24. From Zero to Hero with Kafka Connect @rmoff Confluent Hub hub.confluent.io
- 25. @rmoff From Zero to Hero with Kafka Connect Deploying Kafka Connect Connectors, Tasks, and Workers
- 26. From Zero to Hero with Kafka Connect @rmoff Connectors and Tasks JDBC Source S3 Sink JDBC Task #2JDBC Task #1 S3 Task #1
- 27. From Zero to Hero with Kafka Connect @rmoff Connectors and Tasks JDBC Source S3 Sink JDBC Task #2JDBC Task #1 S3 Task #1
- 28. From Zero to Hero with Kafka Connect @rmoff Connectors and Tasks JDBC Source S3 Sink JDBC Task #2JDBC Task #1 S3 Task #1
- 29. From Zero to Hero with Kafka Connect @rmoff Tasks and Workers JDBC Source S3 Sink JDBC Task #2JDBC Task #1 S3 Task #1 Worker
- 30. From Zero to Hero with Kafka Connect @rmoff Kafka Connect Standalone Worker JDBC Task #2JDBC Task #1 S3 Task #1 Worker Offsets
- 31. From Zero to Hero with Kafka Connect @rmoff "Scaling" the Standalone Worker JDBC Task #2 JDBC Task #1 S3 Task #1 Worker OffsetsOffsets Worker Fault-tolerant? Nope.
- 32. From Zero to Hero with Kafka Connect @rmoff JDBC Task #2 Kafka Connect Distributed Worker JDBC Task #1 JDBC Task #2 S3 Task #1 Offsets Config Status Fault-tolerant? Yeah! Worker Kafka Connect cluster
- 33. From Zero to Hero with Kafka Connect @rmoff Scaling the Distributed Worker JDBC Task #1 JDBC Task #2 S3 Task #1 Offsets Config Status Fault-tolerant? Yeah! Worker Worker Kafka Connect cluster
- 34. From Zero to Hero with Kafka Connect @rmoff Distributed Worker - fault tolerance JDBC Task #1 S3 Task #1 Offsets Config Status Worker Worker Kafka Connect cluster
- 35. From Zero to Hero with Kafka Connect @rmoff Distributed Worker - fault tolerance JDBC Task #1 S3 Task #1 Offsets Config Status Worker Kafka Connect cluster JDBC Task #2
- 36. From Zero to Hero with Kafka Connect @rmoff Multiple Distributed Clusters JDBC Task #1 S3 Task #1 Offsets Config Status Kafka Connect cluster #1 JDBC Task #2 Kafka Connect cluster #2 Offsets Config Status
- 37. @rmoff From Zero to Hero with Kafka Connect Containers
- 38. From Zero to Hero with Kafka Connect @rmoff Kafka Connect images on Docker Hub confluentinc/cp-kafka-connect-base kafka-connect-elasticsearch kafka-connect-jdbc kafka-connect-hdfs […] confluentinc/cp-kafka-connect
- 39. From Zero to Hero with Kafka Connect @rmoff Adding connectors to a container confluentinc/cp-kafka-connect-base JAR Confluent Hub
- 40. From Zero to Hero with Kafka Connect @rmoff At runtime JAR confluentinc/cp-kafka-connect-base kafka-connect: image: confluentinc/cp-kafka-connect:5.2.1 environment: CONNECT_PLUGIN_PATH: '/usr/share/java,/usr/share/confluent-hub-components' command: - bash - -c - | confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.0 /etc/confluent/docker/run http://rmoff.dev/ksln19-connect-docker
- 41. From Zero to Hero with Kafka Connect @rmoff Build a new image FROM confluentinc/cp-kafka-connect:5.2.1 ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components" RUN confluent-hub install --no-prompt neo4j/kafka-connect-neo4j:1.0.0 JAR confluentinc/cp-kafka-connect-base
- 42. From Zero to Hero with Kafka Connect @rmoff Automating connector creation # # Download JDBC drivers cd /usr/share/java/kafka-connect-jdbc/ curl https:"//cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-8.0.13.tar.gz | tar xz # # Now launch Kafka Connect /etc/confluent/docker/run & # # Wait for Kafka Connect listener while [ $$(curl -s -o /dev/null -w %{http_code} http:"//$$CONNECT_REST_ADVERTISED_HOST_NAME:$… echo -e $$(date) " Kafka Connect listener HTTP state: " $$(curl -s -o /dev/null -w %{http_… sleep 5 done # # Create JDBC Source connector curl -X POST http:"//localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name": "jdbc_source_mysql_00", "config": { "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "connection.url": "jdbc:mysql:"//mysql:3306/demo", "connection.user": "connect_user", "connection.password": "asgard", "topic.prefix": "mysql-00-", "table.whitelist" : "demo.customers", } }' # Don't let the container die sleep infinity http://rmoff.dev/ksln19-connect-docker
- 43. @rmoff From Zero to Hero with Kafka Connect Troubleshooting Kafka Connect
- 44. From Zero to Hero with Kafka Connect @rmoff Troubleshooting Kafka Connect $ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" | jq '.connector.state' "RUNNING" $ curl -s "http://localhost:8083/connectors/source-debezium-orders/status" | jq '.tasks[0].state' "FAILED" http://go.rmoff.net/connector-status RUNNING FAILED Connector Task
- 45. From Zero to Hero with Kafka Connect @rmoff Troubleshooting Kafka Connect curl -s "http:!//localhost:8083/connectors/source-debezium-orders-00/status" | jq '.tasks[0].trace' "org.apache.kafka.connect.errors.ConnectExceptionntat io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:230)ntat io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:197)ntat io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onCommunicationFailure(BinlogReader.java: 1018)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:950)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)ntat com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)ntat java.lang.Thread.run(Thread.java: 748)nCaused by: java.io.EOFExceptionntat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:190)ntat com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java :35)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java :27)ntat com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java: 212)ntat io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:224)ntat com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:922)nt!!... 3 moren"
- 46. From Zero to Hero with Kafka Connect @rmoff The log is the source of truth $ confluent log connect $ docker-compose logs kafka-connect $ cat /var/log/kafka/connect.log
- 47. From Zero to Hero with Kafka Connect @rmoff Kafka Connect Symptom not Cause Task is being killed and will not recover until manually restarted " "
- 48. @rmoff From Zero to Hero with Kafka Connect Error Handling and Dead Letter Queues
- 49. From Zero to Hero with Kafka Connect @rmoff org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
- 50. From Zero to Hero with Kafka Connect @rmoff Mismatched converters "value.converter": "AvroConverter" org.apache.kafka.common.errors.SerializationException: Unknown magic byte! Use the correct Converter for the source dataⓘ Messages are not Avro
- 51. From Zero to Hero with Kafka Connect @rmoff Mixed serialisation methods "value.converter": "AvroConverter" org.apache.kafka.common.errors.SerializationException: Unknown magic byte! Some messages are not Avro Use error handling to deal with bad messagesⓘ
- 52. From Zero to Hero with Kafka Connect @rmoff Error Handling and DLQ Handled Convert -> read/write from Kafka -> [de]-serialisation Transform Not Handled Start -> Connections to a data store Poll / Put -> Read/Write from/to data store* * can be retried by Connect https://cnfl.io/connect-dlq
- 53. From Zero to Hero with Kafka Connect @rmoff Fail Fast Kafka Connect Source topic messages Sink messages https://cnfl.io/connect-dlq
- 54. From Zero to Hero with Kafka Connect @rmoff YOLO ¯_(ツ)_/¯ Kafka Connect Source topic messages Sink messages errors.tolerance=all https://cnfl.io/connect-dlq
- 55. From Zero to Hero with Kafka Connect @rmoff Dead Letter Queue Kafka Connect Source topic messages Sink messages Dead letter queue errors.tolerance=all errors.deadletterqueue.topic.name=my_dlq https://cnfl.io/connect-dlq
- 56. From Zero to Hero with Kafka Connect @rmoff Re-processing the Dead Letter Queue Source topic messages Sink messages Dead letter queue Kafka Connect (Avro sink) Kafka Connect (JSON sink) https://cnfl.io/connect-dlq
- 57. @rmoff From Zero to Hero with Kafka Connect Metrics and Monitoring
- 58. From Zero to Hero with Kafka Connect @rmoff REST API http://go.rmoff.net/connector-status
- 59. From Zero to Hero with Kafka Connect @rmoff JMX
- 60. From Zero to Hero with Kafka Connect @rmoff http://cnfl.io/book-bundle
- 61. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
- 62. 💬 Confluent Community Slack group http://cnfl.io/slack