SlideShare a Scribd company logo

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

•
•
Guido Schmutz
Guido Schmutz

A Kafka cluster stores streams of records (messages) in categories called topics. It is the architectural backbone for integrating streaming data with a Data Lake, Microservices and Stream Processing. Today’s enterprises have their core systems often implemented on top of relational databases, such as the Oracle RDBMS. Implementing a new solution supporting the digital strategy using Kafka and the ecosystem can not always be done completely separate from the traditional legacy solutions. Often streaming data has to be enriched with state data which is held in an RDBMS of a legacy application. It’s important to cache this data in the stream processing solution, so that It can be efficiently joined to the data stream. But how do we make sure that the cache is kept up-to-date, if the source data changes? We can either poll for changes from Kafka using Kafka Connect or let the RDBMS push the data changes to Kafka. But what about writing data back to the legacy application, i.e. an anomaly is detected inside the stream processing solution which should trigger an action inside the legacy application. Using Kafka Connect we can write to a database table or view, which could trigger the action. But this not always the best option. If you have an Oracle RDBMS, there are many other ways to integrate the database with Kafka, such as Advanced Queueing (message broker in the database), CDC through Golden Gate or Debezium, Oracle REST Database Service (ORDS) and more. In this session, we present various blueprints for integrating an Oracle RDBMS with Apache Kafka in both directions and discuss how these blueprints can be implemented using the products mentioned before.

1 of 50
Download to read offline
BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Guido Schmutz (guido.schmutz@trivadis.com)
Agenda 1. Introduction 2. Blueprints Oracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Summary
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 170th edition
Introduction
Microservices / Modern Applications • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! M3M2 ACL Event Hub M1
Microservices / Modern Applications Integrate with Traditional System M3M2 ACL Event Hub M1 ACL • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! Traditional App
Use Case Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense “Modern Apps”Traditional Apps (Legacy)
MessageMessageMessageMessage MessageMessage Properties - Message Message Message A1 A2 A3 Message B1 B2 B3 B4 A1 A2 A3 B B1 B2 B3 B4 Table A A1 A2 A3 Table B B1 B2 B3 B4 FlatDB Model Aggregate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Properties - Latency Traditional System Event Hub Data Flow RDBMS latency latency Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Properties – Anti-Corruption Layer (ACL) Traditional System Event Hub Data Flow RDBMS Traditional System Event Hub Data Flow RDBMS ACL ACL Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Examples: • View Layer • Storage Procedure • JSON Support in DB • … Examples: • StreamSets • Kafka Connect • Kafka Streams / KSQL • … Database Dataflow
Blueprints Oracle RDBMS => Apache Kafka
Blueprints Oracle RDBMS => Apache Kafka (DB-K) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Que with bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5
DB-K_1: Polling of RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_1: Polling of RDBMS table/view Kafka Connect with JDBC Source Connector
Kafka Connect & JDBC Connector • Many connectors available • Single Message Transforms (SMT) • declarative style, simple data flows • framework is part of Apache Kafka
DB-K_1 – Configure JDBC Connector #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
DB-K_2: Change Data Capture (CDC) on RDBMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy DB-K_2: Change Data Capture (CDC) on RDBMS Using Oracle GoldenGate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Alternatives: StreamSets Data Collector Attunity Debezium …
DB-K_3: Polling of RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow DB-K_3: Polling of RDBMS API StreamSets invokes Oracle Rest Data Service Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Oracle REST Data Services (ORDS) • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
DB-K_3 – Setup ORDS (I) ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
DB-K_3 – Setup ORDS (II) ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name ''emailAddress'' VALUE cu.email), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''id'' VALUE li.product_id, ''name'' VALUE li.product_name, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
StreamSets Data Collector • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing • custom sources, sinks, processors • Monitoring and Error Detection
DB-K_4: Produce to Event Hub from RDBMS Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Native Kafka Producer using Java in DB Does not feel right!
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Invoke REST Proxy from PL/SQL Invoking a REST Service from DB not well-supported
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon … DB-K_4: Produce to Event Hub from RDBMS Oracle Big Data SQL integrates with Kafka Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
DB-K_5: RDBMS Queue with bridge to Event Hub Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
DB-K_5: RDBMS Queue with bridge to Event Hub Oracle Advanced Queuing & Kafka Connect JMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
BEGIN dbms_aqadm.create_queue_table ( queue_table => 'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.princ ipal=order_processing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonS impleConverter`" } }' DB-K_5 – Setup AQ and Kafka Connect (I)
DB-K_5: RDBMS Queue with bridge to Event Hub Oracle AQ with Kafka API & MirrorMaker Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense AQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing
Blueprints Apache Kafka => Oracle RDBMS
Blueprints Apache Kafka => Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4
K-DB_1: Write to RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_1: Write to RDBMS table/view Kafka Connect and JDBC Sink Connector
K-DB_2: Write over RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_2: Write over RDBMS API Kafka Connect invokes Oracle Rest Data Service
DB-K_2 – Setup ORDS (I) ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
K-DB_3: Consume from Event Hub Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon… K-DB_3: Consume from Event Hub Oracle Big Data SQL exposes topic as table Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
K-DB_4: Event Hub with bridge to RDBMS queue Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ K-DB_4: Event Hub with bridge to RDBMS queue Oracle Advanced Queuing & Kafka Connect JMS Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing K-DB_4: Event Hub with bridge to RDBMS queue Oracle AQ with Kafka API & MirrorMaker Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
Summary
Summary Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 https://github.com/gschmutz/various-demos/tree/master/bidirectional-integration-oracle-kafka DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Queue with bridge to Event Hub
Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service sEvent Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

More Related Content

Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka

  • 1. BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka Guido Schmutz (guido.schmutz@trivadis.com)
  • 2. Agenda 1. Introduction 2. Blueprints Oracle RDBMS => Apache Kafka 3. Blueprints Apache Kafka => Oracle RDBMS 4. Summary
  • 3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 170th edition
  • 5. Microservices / Modern Applications • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! M3M2 ACL Event Hub M1
  • 6. Microservices / Modern Applications Integrate with Traditional System M3M2 ACL Event Hub M1 ACL • Highly decoupled • Independently deployable • Bounded Context/Aggregate (DDD) • Responsible for their data • Favour asynchronous, event-driven interaction over synchronous • Smart Endpoints and Dump Pipes • Use Anti-Corruption Layer (ACL) if no fit! Traditional App
  • 7. Use Case Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order Customer (compacted) Notification Microservice Notification Logic Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense “Modern Apps”Traditional Apps (Legacy)
  • 8. MessageMessageMessageMessage MessageMessage Properties - Message Message Message A1 A2 A3 Message B1 B2 B3 B4 A1 A2 A3 B B1 B2 B3 B4 Table A A1 A2 A3 Table B B1 B2 B3 B4 FlatDB Model Aggregate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 9. Properties - Latency Traditional System Event Hub Data Flow RDBMS latency latency Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 10. Properties – Anti-Corruption Layer (ACL) Traditional System Event Hub Data Flow RDBMS Traditional System Event Hub Data Flow RDBMS ACL ACL Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Examples: • View Layer • Storage Procedure • JSON Support in DB • … Examples: • StreamSets • Kafka Connect • Kafka Streams / KSQL • … Database Dataflow
  • 11. Blueprints Oracle RDBMS => Apache Kafka
  • 12. Blueprints Oracle RDBMS => Apache Kafka (DB-K) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Que with bridge to Event Hub DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5
  • 13. DB-K_1: Polling of RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 14. Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_1: Polling of RDBMS table/view Kafka Connect with JDBC Source Connector
  • 15. Kafka Connect & JDBC Connector • Many connectors available • Single Message Transforms (SMT) • declarative style, simple data flows • framework is part of Apache Kafka
  • 16. DB-K_1 – Configure JDBC Connector #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "jdbc-driver-source", "config": { "connector.class": "JdbcSourceConnector", "connection.url":"jdbc:oracle:thin//oracle-db:1521/XEPDB1", "mode": "timestamp", "timestamp.column.name":”modified_at", "table.whitelist":”order", "validate.non.null":"false", "topic.prefix":”orderprocessing_", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable": "false", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "name": "jdbc-driver-source", "transforms":"createKey,extractInt", "transforms.createKey.type":"org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields":"id", "transforms.extractInt.type":"org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractInt.field":"id" } }'
  • 17. DB-K_2: Change Data Capture (CDC) on RDBMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 18. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Redo Log REST to Event Hub Rest Proxy DB-K_2: Change Data Capture (CDC) on RDBMS Using Oracle GoldenGate Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense Alternatives: StreamSets Data Collector Attunity Debezium …
  • 19. DB-K_3: Polling of RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 20. Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow DB-K_3: Polling of RDBMS API StreamSets invokes Oracle Rest Data Service Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 21. Oracle REST Data Services (ORDS) • makes it easy to develop modern REST interfaces for relational data in the Oracle Database and the Oracle Database 18c JSON Document Store • ORDS maps HTTP(S) verbs (GET, POST, PUT, DELETE, etc.) to database transactions and returns any results formatted using JSON • Java middle tier application on WebLogic, Tomcat, Docker, Standalone (for development)
  • 22. DB-K_3 – Setup ORDS (I) ORDS.ENABLE_SCHEMA( p_enabled => TRUE, p_schema => 'ORDER_PROCESSING', p_url_mapping_type => 'BASE_PATH', p_url_mapping_pattern => 'order_processing', p_auto_rest_auth => FALSE); ORDS.DEFINE_MODULE( p_module_name => 'order_processing', p_base_path => '/orders/', p_items_per_page => 25, p_status => 'PUBLISHED', p_comments => NULL); ORDS.DEFINE_TEMPLATE( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_priority => 0, p_etag_type => 'HASH', p_etag_query => NULL, p_comments => NULL);
  • 23. DB-K_3 – Setup ORDS (II) ORDS.DEFINE_HANDLER( p_module_name => 'order_processing', p_pattern => 'changes/:offset', p_method => 'GET', p_source_type => 'resource/lob', p_items_per_page => 25, p_source => 'SELECT ''application/json'', json_object(''orderId'' VALUE po.id, ''orderDate'' VALUE po.order_date, ''orderMode'' VALUE po.order_mode, ''customer'' VALUE json_object(''firstName'' VALUE cu.first_name, ''lastName'' VALUE cu.last_name ''emailAddress'' VALUE cu.email), ''lineItems'' VALUE (SELECT json_arrayagg( json_object(''ItemNumber'' VALUE li.id, ''Product'' VALUE json_object(''id'' VALUE li.product_id, ''name'' VALUE li.product_name, ''unitPrice'' VALUE li.unit_price), ''quantity'' VALUE li.quantity)) FROM order_item_t li WHERE po.id = li.order_id), ''offset'' VALUE TO_CHAR(po.modified_at, ''YYYYMMDDHH24MISS'')) FROM order_t po LEFT JOIN customer_t cu ON (po.customer_id = cu.id) WHERE po.modified_at > TO_DATE(:offset, ''YYYYMMDDHH24MISS'')'
  • 24. StreamSets Data Collector • GUI-based, drag-and drop Data Flow Pipelines • Both stream and batch processing • custom sources, sinks, processors • Monitoring and Error Detection
  • 25. DB-K_4: Produce to Event Hub from RDBMS Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 26. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Native Kafka Producer using Java in DB Does not feel right!
  • 27. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Rest Proxy ? Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense DB-K_4: Produce to Event Hub from RDBMS Invoke REST Proxy from PL/SQL Invoking a REST Service from DB not well-supported
  • 28. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon … DB-K_4: Produce to Event Hub from RDBMS Oracle Big Data SQL integrates with Kafka Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 29. DB-K_5: RDBMS Queue with bridge to Event Hub Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 30. DB-K_5: RDBMS Queue with bridge to Event Hub Oracle Advanced Queuing & Kafka Connect JMS Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 31. BEGIN dbms_aqadm.create_queue_table ( queue_table => 'ORDER_QT', queue_payload_type => 'SYS.AQ$_JMS_TEXT_MESSAGE', sort_list => 'PRIORITY,ENQ_TIME', multiple_consumers => FALSE, message_grouping => dbms_aqadm.none ); END; / DB-K_5 – Setup AQ and Kafka Connect (I) BEGIN dbms_aqadm.create_queue ( queue_name => 'ORDER_AQ', queue_table => 'ORDER_QT', max_retries => 1, retry_delay => 2, -- seconds retention_time => 60*60*24*7 -- 1w ); END; / BEGIN dbms_aqadm.start_queue( queue_name => 'ORDER_AQ', enqueue => TRUE, dequeue => TRUE ); END; /
  • 32. curl -X "POST" "$DOCKER_HOST_IP:8083/connectors" -H "Content-Type: application/json" --data '{ "name": "jms-source", "config": { "name": "jms-source", "connector.class": "com.datamountaineer...JMSSourceConnector", "connect.jms.initial.context.factory": "oracle.jms.AQjmsInitialContextFactory", "connect.jms.initial.context.extra.params": "db_url=jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1,java.naming.security.princ ipal=order_processing,java.naming.security.credentials=order_processing", "tasks.max": "1", "connect.jms.connection.factory": "ConnectionFactory", "connect.jms.url": "jdbc:oracle:thin:@//192.168.73.86:1521/XEPDB1", "connect.jms.kcql": "INSERT INTO order SELECT * FROM order_aq WITHTYPE QUEUE WITHCONVERTER=`com.datamountaineer.streamreactor.connect.converters.source.JsonS impleConverter`" } }' DB-K_5 – Setup AQ and Kafka Connect (I)
  • 33. DB-K_5: RDBMS Queue with bridge to Event Hub Oracle AQ with Kafka API & MirrorMaker Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense AQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing
  • 34. Blueprints Apache Kafka => Oracle RDBMS
  • 35. Blueprints Apache Kafka => Oracle RDBMS (K-DB) Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4
  • 36. K-DB_1: Write to RDBMS table/view Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 37. Event Hub Stream Data Integration API Applications / Data Sources Data FlowRDBMS Application Logic Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_1: Write to RDBMS table/view Kafka Connect and JDBC Sink Connector
  • 38. K-DB_2: Write over RDBMS API Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 39. Event Hub Stream Data Integration API Applications / Data Sources Data Flow RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense K-DB_2: Write over RDBMS API Kafka Connect invokes Oracle Rest Data Service
  • 40. DB-K_2 – Setup ORDS (I) ORDS.DEFINE_HANDLER( p_module_name => 'customer', p_pattern => 'customer', p_method => 'POST', p_source_type => 'plsql/block', p_items_per_page => 0, p_source => 'DECLARE L_CU CLOB := :body_text; BEGIN INSERT INTO customer_t (id, first_name, last_name, title, notification_on, email, slack_handle, twitter_handle) SELECT * FROM json_table(L_CU, ''$'' COLUMNS ( id NUMBER PATH ''$.id'', first_name VARCHAR2 PATH ''$.firstName'', last_name VARCHAR2 PATH ''$.lastName'', title VARCHAR2 PATH ''$.title'', notification_on VARCHAR2 PATH ''$.notificationOn'', email VARCHAR2 PATH ''$.email'', slack_handle VARCHAR2 PATH ''$.slackHandle'', twitter_handle VARCHAR2 PATH ''$.twitterHandle'' )); INSERT INTO address_t (customer_id, id, street, nr, city, postcode, country) SELECT * FROM json_table( ... )
  • 41. K-DB_3: Consume from Event Hub Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 42. Event Hub Stream Data Integration API Applications / Data Sources RDBMS Application Logic API Stream Data Integration & Analytics Stream Analytics Data Flow REST to Event Hub Oracle Big Data SQL Coming soon… K-DB_3: Consume from Event Hub Oracle Big Data SQL exposes topic as table Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 43. K-DB_4: Event Hub with bridge to RDBMS queue Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS Queue Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 44. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ K-DB_4: Event Hub with bridge to RDBMS queue Oracle Advanced Queuing & Kafka Connect JMS Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 45. Stream Data Integration & Analytics Stream Analytics Event Hub Stream Data Integration API Data Flow Application / Data Sources Data Flow Application Logic RDBMS QueueAQ (Kafka API) Oracle works on a Kafka API for Advanced Queuing K-DB_4: Event Hub with bridge to RDBMS queue Oracle AQ with Kafka API & MirrorMaker Flat Aggregate Low Latency High Latency DB Dataflow Message Latency ACL Open Source CommercialLicense
  • 47. Summary Customer Microservice { } Customer API CustomerCustomer Logic Order Processing System { } Order API OrderOrder Logic REST REST Event Hub Customer Mat View Order (compacted) Customer (compacted) Notification Microservice Notification Logic Schema Registry K-DB_1: Write to RDBMS table/view K-DB_2: Write over RDBMS API K-DB_3: Consume from Event Hub K-DB_4: Event Hub with bridge to RDBMS Queue K_DB-1 K_DB-2 K_DB-3 K_DB-4 DB_K-1 DB_K-2 DB_K-3 DB_K-4 DB_K-5 https://github.com/gschmutz/various-demos/tree/master/bidirectional-integration-oracle-kafka DB-K_1: Polling of RDBMS table/view DB-K_2: Change Data Capture (CDC) on RDBMS DB-K_3: Polling of RDBMS API DB-K_4: Produce to Event Hub from RDBMS DB-K_5: RDBMS Queue with bridge to Event Hub
  • 48. Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service Event Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream
  • 49. Bulk Source Ref Architecture Data Platform Service Event Stream Bulk Data Flow Event Source Location DB Extract File Weather DB IoT Data Mobile Apps Social File Import / SQL Import Consumer BI Apps Data Science Workbench Enterprise App Enterprise Data Warehouse SQL / Search SQL “Native” Raw RDBMS “SQL” / Search Service sEvent Hub Hadoop ClusterdHadoop ClusterBig Data Platform SQL Export Storage Storage Raw Refined/ UsageOpt Microservice Cluster Stream Processing Cluster Stream Processor Model / State Edge Node Rules Event Hub Storage Governance Data Catalog Rules Engine Parallel Processing Query Engine Microservice Data { } API Event Stream Event Stream Modern Data Platform Event Stream