Scylla Summit 2017: Streaming ETL in Kafka for Everyone with KSQL

PRESENTATION TITLE ON ONE LINE
AND ON TWO LINES
First and last name
Position, company
Streaming ETL in Kafka
for Everyone with KSQL
Software Engineer, Confluent Inc.
Hojjat Jafarpour

AND ON TWO LINES
First and last name
Position, company
Hojjat Jafarpour
2
Software Engineer at Confluent
○Starter KSQL project at Confluent
Previously at Tidemark, Quantcast, Informatica and
NEC Labs
PhD in Computer Science from UC Irvine
○Data management, pub/sub and streaming
hojjat@confluent.io
@hojjat

AND ON TWO LINES
First and last name
Position, company
Streaming ETL, with Apache Kafka and Confluent
Platform
3

AND ON TWO LINES
First and last name
Position, company
4

AND ON TWO LINES
First and last name
Position, company
5

AND ON TWO LINES
First and last name
Position, company
6

AND ON TWO LINES
First and last name
Position, company
7

AND ON TWO LINES
First and last name
Position, company
Kafka Connect : Stream data in and out of Kafka
8
Amazon
S3

AND ON TWO LINES
First and last name
Position, company
Single Message Transform (SMT)
9
▪ Modify events before storing in
Kafka:
o Mask/drop sensitive information
o Set partitioning key
o Store lineage
▪ Modify events going out of
Kafka:
o Route high priority events to faster
data stores
o Direct events to different
Elasticsearch indexes
o Cast data types to match destination

AND ON TWO LINES
First and last name
Position, company
10
But I need to
join…aggregate…filter
…

AND ON TWO LINES
First and last name
Position, company
KSQL from Confluent
11
A Developer Preview of
KSQL
An Open Source Streaming SQL
Engine for Apache KafkaTM

AND ON TWO LINES
First and last name
Position, company
KSQL: a Streaming SQL Engine for Apache Kafka™ from
Confluent
▪ Enables stream processing with zero coding required
▪ The simplest way to process streams of data in real-time
▪ Powered by Kafka: scalable, distributed, battle-tested
▪ All you need is Kafka–No complex deployments of bespoke
systems for stream processing
12
Ksql>

AND ON TWO LINES
First and last name
Position, company
KSQL: the Simplest Way to Do Stream Processing
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 SECONDS)
GROUP BY card_number
HAVING count(*) > 3;
13

AND ON TWO LINES
First and last name
Position, company
KSQL Concepts
▪ STREAM and TABLE as first-class citizens
o Interpretations of topic content
▪ STREAM - data in motion
▪ TABLE - collected state of a stream
o One record per key (per window)
o Current values (compacted topic) ← Not yet in KSQL
▪ STREAM – TABLE Joins
14

AND ON TWO LINES
First and last name
Position, company
Window Aggregations
Three types supported (same as KStreams):
● TUMBLING: Fixed-size, non-overlapping, gap-less windows
• SELECT ip, count(*) AS hits FROM clickstream
WINDOW TUMBLING (size 1 minute) GROUP BY ip;
● HOPPING: Fixed-size, overlapping windows
• SELECT ip, SUM(bytes) AS bytes_per_ip_and_bucket FROM clickstream
WINDOW HOPPING ( size 20 second, advance by 5 second) GROUP BY ip;
● SESSION: Dynamically-sized, non-overlapping, data-driven window
• SELECT ip, SUM(bytes) AS bytes_per_ip FROM clickstream
WINDOW SESSION (20 second) GROUP BY ip;
15

AND ON TWO LINES
First and last name
Position, company
Streaming ETL, powered by Apache Kafka and Confluent
Platform
16
KSQL

AND ON TWO LINES
First and last name
Position, company
Simple Web Analytics Pipeline
● Pageview stream
● User table
● Materialized views
o Region visitor count
o Region visitor demography
17
CREATE STREAM pageviews (viewtime BIGINT, userid VARCHAR, pageid VARCHAR) WITH
(kafka_topic='pageviews', value_format=JSON);
CREATE TABLE users (registertime BIGINT, gender VARCHAR, regionid VARCHAR, userid
VARCHAR) WITH (kafka_topic='users', value_format='JSON');

AND ON TWO LINES
First and last name
Position, company
18
Region visitor count
CREATE STREAM joined_pageviews AS
SELECT users.userid AS userid, pageid, regionid, gender
FROM pageviews LEFT JOIN users ON pageviews.userid = users.userid;
CREATE TABLE region_visitor_count AS
SELECT regionid , COUNT(*) AS visit_count
FROM joined_pageviews
WINDOW TUMBLING (size 30 second)
GROUP BY regionid;

AND ON TWO LINES
First and last name
Position, company
19
Region visitor demography
CREATE TABLE region_visitor_demo_count AS
SELECT regionid, gender, COUNT(*) AS visit_count
FROM joined_pageviews
WINDOW TUMBLING (size 30 second)
GROUP BY gender, regionid;

AND ON TWO LINES
First and last name
Position, company
Streaming ETL, powered by Apache Kafka and Confluent
Platform
20
KSQL

AND ON TWO LINES
First and last name
Position, company
Confluent Platform: Enterprise Streaming based on Apache
Kafka™
21
Database
Changes
Log Events loT Data
Web
Events
…
CRM
Data Warehouse
Database
Hadoop
Data
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time
Applications
…
Apache Open Source Confluent Open Source Confluent Enterprise
Confluent Platform
Confluent Platform
Apache Kafka™
Core | Connect API | Streams API
Data Compatibility
Schema Registry
Monitoring & Administration
Confluent Control Center | Security
Operations
Replicator | Auto Data Balancing
Development and Connectivity
Clients | Connectors | REST Proxy | KSQL | CLI

AND ON TWO LINES
First and last name
Position, company
Date to remember
22
• Kafka Summit 2018
• April 23-24 in London!
• More details:
https://kafka-summit.org/

AND ON TWO LINES
First and last name
Position, company
THANK YOU
hojjat@confluent.io
@hojjat
Please stay in touch
Any questions?
https://github.com/confluentinc/ksql/
https://www.confluent.io/download/

Scylla Summit 2017: Streaming ETL in Kafka for Everyone with KSQL

More Related Content

What's hot

What's hot (20)

Similar to Scylla Summit 2017: Streaming ETL in Kafka for Everyone with KSQL

Similar to Scylla Summit 2017: Streaming ETL in Kafka for Everyone with KSQL (20)

More from ScyllaDB

More from ScyllaDB (20)

Recently uploaded

Recently uploaded (20)

Scylla Summit 2017: Streaming ETL in Kafka for Everyone with KSQL