Oracle Stream Analytics - Simplifying Stream Processing

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Oracle Stream Analytics
Simplifying Stream Processing
29.9.2016 – DOAG 2016 Big Data Days
Guido Schmutz

Guido Schmutz
Working for Trivadis for more than 19 years
Oracle ACE Director for Fusion Middleware and SOA
Co-Author of different books
Consultant, Trainer, Software Architect for Java, SOA & Big Data / Fast Data
Member of Trivadis Architecture Board
Technology Manager @ Trivadis
More than 25 years of software development experience
Contact: guido.schmutz@trivadis.com
Blog: http://guidoschmutz.wordpress.com
Slideshare: http://www.slideshare.net/gschmutz
Twitter: gschmutz
Oracle Stream Analytics - Simplifying Stream Processing2

Unser Unternehmen.
Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution
Engineering und der Erbringung von IT-Services mit Fokussierung auf -
und -Technologien in der Schweiz, Deutschland, Österreich und
Dänemark. Trivadis erbringt ihre Leistungen aus den strategischen Geschäftsfeldern:
Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme.
B E T R I E B

KOPENHAGEN
MÜNCHEN
LAUSANNE
BERN
ZÜRICH
BRUGG
GENF
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
WIEN
Mit über 600 IT- und Fachexperten bei Ihnen vor Ort.
14 Trivadis Niederlassungen mit
über 600 Mitarbeitenden.
Über 200 Service Level Agreements.
Mehr als 4'000 Trainingsteilnehmer.
Forschungs- und Entwicklungsbudget:
CHF 5.0 Mio.
Finanziell unabhängig und
nachhaltig profitabel.
Erfahrung aus mehr als 1'900 Projekten
pro Jahr bei über 800 Kunden.

Agenda
1. Introduction to Streaming Analytics
2. Oracle Stream Analytics
3. Demo

Introduction to Streaming Analytics

Traditional Data Processing - Challenges
• Introduces too much “decision latency”
• Responses are delivered “after the fact”
• Maximum value of the identified situation is lost
• Decision are made on old and stale data
• “Data a Rest”

The New Era: Streaming Data Analytics / Fast Data
• Events are analyzed and processed in
real-time as the arrive
• Decisions are timely, contextual and
based on fresh data
• Decision latency is eliminated
• “Data in motion”

Event / Stream Processing Architecture
Data
Ingestion
Batch
compute
Data
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Content
Logfiles
Social
RDBMS
ERP
Sensor
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Result Store
Messaging
Result Store
Oracle Stream Analytics - Simplifying Stream Processing
= Data in Motion = Data at Rest
9

“Lambda Architecture” for Big Data
Data
Ingestion
(Analytical) Batch Data Processing
Batch
compute
Result StoreData
Sources
Channel
Data
Consumer
Reports
Service
Analytic
Tools
Alerting
Tools
Content
RDBMS
Social
ERP
Logfiles
Sensor
Machine
(Analytical) Real-Time Data Processing
Stream/Event Processing
Batch
compute
Messaging
Result Store
Query
Engine
Result Store
Computed
Information
Raw Data
(Reservoir)
Oracle Stream Analytics - Simplifying Stream Processing
= Data in Motion = Data at Rest
Pulling
Ingestion
10

When to Stream / When not?
Constant low
Milliseconds & under
Low milliseconds to seconds,
delay in case of failures
10s of seconds of more,
Re-run in case of failures
Real-Time Near-Real-Time Batch

“No free lunch”
Constant low
Milliseconds & under
Low milliseconds to seconds,
delay in case of failures
10s of seconds of more,
Re-run in case of failures
Real-Time Near-Real-Time Batch
“Difficult” architectures, lower latency “Easier architectures”, higher latency

Why Event / Stream Processing?
Visualize Business in real-time
• Dashboards can help people to visualize, monitor and make sense of massive amount of
incoming data in real-time
Detect Urgent Situations
• Based on simple or complex analytical patterns of urgent business events
• Urgent because they happen in real-time
Automate immediate actions
• Run in the background quietly until detecting an urgent situation (risk or opportunity)
• Alerts can go to humans through email, text or push notifications or to other applications trough
message queues or service call

Oracle Stream Analytics

History of Oracle Stream Analytics
Oracle Complex Event
Processing (OCEP)
Oracle Event Processing (OEP)
Oracle Stream Explorer (SX)
Oracle Event Processing
for Java Embedded
Oracle Stream Analytics (OSA)
Oracle Edge Analytics (OAE)
BEA Weblogic Event Server
Oracle CQL
Oracle IoT Cloud Service
2016
2015
2007
2008
2012
2013

OEA
• Filtering
• Correlation
• Aggregation
• Pattern
matching
Devices /
Gateways
Services
Computing Edge Enterprise
“Sea of data”
Macro-event
High-value
Actionable
In-context
EDGE
Analytics
Stream
Analytics
FOG
• High Volume
• Continuous Streaming
• Extreme Low Latency
• Disparate Sources
• Temporal Processing
• Pattern Matching
• Machine Learning
Oracle Stream Analytics: From Noise to Value
• High Volume
• Continuous Streaming
• Sub-Millisecond Latency
• Disparate Sources
• Time-Window Processing
• Pattern Matching
• High Availability / Scalability
• Coherence Integration
• Geospatial, Geofencing
• Big Data Integration
• Business Event Visualization
• Action!

Oracle Stream Analytics Platform
What it does
• Compelling, friendly and visually stunning real time
streaming analytics user experience for Business users to
dynamically create and implement Instant Insight solutions
Key Features
• Analyze simulated or live data feeds to determine event
patterns, correlation, aggregation & filtering
• Pattern library for industry specific solutions
• Streams, References, Maps & Explorations
Benefits
• Accelerated delivery time
• Hides all challenges & complexities of underlying real-time
event-driven infrastructure

Oracle Stream Analytics – Self-Service Stream
Processing!
Understanding of CQL Filtering, Correlation, Pattern: NOT NEEDED
Understanding of IT Deployment and Management: NOT NEEDED
Understanding of Development, Java, Best Practices: NOT NEEDED
Understanding of the Event Driven Platform: NOT NEEDED

Oracle Stream Analytics – Terminology
Explorer: The Application User Interface Catalog: The repository for browsing resources

Stream: incoming flow of events that you
want to analyze (CSV, Kafka, JMS, Rest,
MQTT, …)
Exploration: application that correlates events
from streams and data sources, using filters,
groupings, summaries, ranges, and more

Shape: A blueprint of an event in a stream or
data in a data source. How the business data
is represented in the selected stream
Map: collection of geo-fences
Reference: A connection to static data that is
joined to a stream to enrich it and/or to be used in
business logic and output

Pattern: A pre-built Exploration that
addresses a particular business scenario in a
focused and simplified User Interface
Connection: collection of metadata required to
connect to an external system
Targets: defines an interface with a downstream
system

Business accessibility to Geo-Streaming Analytics
Real Time Streaming Solutions face an increasing need to track "assets of interest" and
initiate actions based on encroachment of boundary proximity to fixed and moving
objects and other geographic, temporal, or event conditions.
Geo-Fence, Fence, Polygon
Geo-Streaming

“ Add value to your real time streaming data discovery and analytics by applying and including
mathematical, statistical analysis to the live output stream”
“These streaming “Excel spreadsheets” really do come to life”
Expression Builder enabling calculations

Concept of Connections and their reuse in Streams

Decision Table for Nested IF-THEN-ELSE Rules

Topology View and Navigation

Relationship between Streams (Sources), References
and Explorations

Demo

Oracle Stream Analytics Demo Use Case: Truck
Movements
Truck
Data
Ingestion
Geo-Fencing
2016-06-02 14:39:56.605|98|27|Mark
Lochbihler|803014426|Wichita to
Little Rock Route 2|Normal|38.65|-
90.21|5187297736652502631
{"timestamp": "2016-06-02
14:39:56.991", "truckId": 99,
"driverId": 31, "driverName":
"Rommel Garcia", "routeId":
1565885487, "routeName":
"Springfield to KC Via Hanibal",
"eventType": "Normal", "latitude":
37.16, "longitude": "-94.46",
"correlationId":
5187297736652502631}
Reckless Driving
Detector
NEAR
ENTER
Truck
Driver
DashboardMovement Movement
JSON
Reckless
Driver

Continuous Ingestion in Stream Processing
DB Source
Big Data
Log
Stream
Processing
IoT Sensor
Event Hub
Topic
Topic
REST
Topic
IoT GW
CDC GW
Connect
CDC
DB Source
Log CDC
Native
IoT Sensor
IoT Sensor
33
Dataflow GW
Topic
Topic
Queue
MQTT GW
Topic
Dataflow GW
Dataflow
TopicREST
33
File Source
Log
Log
Log
Social
Native
Topic
Topic

Apache Kafka – High-volume messaging system
Distributed publish-subscribe messaging system
Designed for processing of high-volume, real
time activity stream data (logs, metrics
collections, social media streams, …)
Topic Semantic
does not implement JMS standard!
Initially developed at LinkedIn, now part of
Apache
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer

Demo: Oracle Stream Analytics

Summary

Native Stream Processing => OEP server
Ingestion
Event
Source
Event
Source
Event
Source
Individual Event
PPPPPPPPPPPP

Micro-Batch Stream Processing => Spark Streaming
Ingestion
Event
Source
Event
Source
Event
Source
PPPPPP

Summary
Oracle Stream Analytics leverages the capabilities found in Oracle Event Processing
(OEP)
Empowering Business users to gain insight into real-time information and take
appropriate actions when needed => makes stream processing accessible
Makes Stream/Event Processing less technical => “Excel spread sheet” on Streams
Part of Oracle IoT Cloud Service
Support Spark Streaming as a deployment platform for Streaming ML
Interesting road map: Rule Engine, Machine Learning, Extensible Patterns

Oracle Stream Analytics on Docker
Oracle Stream Analytics 12.2.1 Documentation
Oracle Stream Analytics 12.2.1 Download

Guido Schmutz
Technology Manager
guido.schmutz@trivadis.com

Oracle Stream Analytics - Simplifying Stream Processing

Related slideshows

More Related Content

Oracle Stream Analytics - Simplifying Stream Processing