This slide deck introduces the WSO2 Enterprise Integrator analytics profile and explore its observability features.
Watch the webinar here: https://wso2.com/library/webinars/2018/09/how-to-create-observable-integration-solutions-using-wso2-enterprise-integrator
Report
Share
Report
Share
1 of 36
Download to read offline
More Related Content
How to Create Observable Integration Solutions Using WSO2 Enterprise Integrator
1. How to Create Observable Integration
Solutions Using WSO2 Enterprise
Integrator
Chanika Geeganage and Lahiru Madushanka
Integration Team
1
2. Agenda
• Introduction to observability in WSO2 Enterprise Integrator
• Observability for integrations
• Observability for business processes
• Observability for messaging
• Extending EI Analytics
2
4. Importance of Observability
• Observability is a measure of how well internal states of a
system can be inferred from knowledge of its external outputs.
• Observability helps in
– Decision making and building incident preparedness
– Identifying constrained resources and forecasting growth
– Helping reduce the cycle of fire fighting within teams
4
5. Introduction to Observability in WSO2 EI
• Events
• Logs
• Tracing
• Java Management Extensions (JMX)
5
Business / Operational Insight
5
Applications
Services and systems
Improvements / Modifications
Existing Services
Identify erroneous
incidents
Monitoring requests count
Trace messages
WSO2
Enterprise
Integrator
Monitor Infrastructure
utilization
7. ESB Event Monitoring with EI Analytics
7
Thrift Data
Agent
Thrift Data
Receiver
Real
Time
Events Dashboard
Storage
Analyze
Summarize
Worker Portal
8. Aggregations over long time periods
• Incremental aggregation for
– every second, minute, hour, … , year
• Support for out of order event arrival
• Fast data retrieval from memory and disk
8
Current Min
Current Hour
Sec
Min
Hour
0 - 1 - 5 ...
- 1
- 2 - 3 - 4 - 64 - 65 ...
- 2
- 124
9. Prerequisites for EI Analytics
• To enable mediation statistics and message tracing,
<EI_HOME>/conf/synapse.properties
9
Property Name Description
mediation.flow.statistics.enable Enables statistics for the selected ESB artifact(s).
mediation.flow.statistics.tracer.collect.payloads Enables tracing for the selected ESB artifact(s) and
records message payloads.
mediation.flow.statistics.tracer.collect.properties Enables tracing for the selected ESB artifact(s) and
records message context (transport) properties.
mediation.flow.statistics.collect.all Enables mediation statistics for all the artifacts in the
ESB profile by default.
10. Prerequisites for EI Analytics cont.
• To publish data from the ESB profile to EI analytics profile,
<EI_HOME>/repository/deployment/server/eventpublishers/MessageFlowConfigurationPublisher.xml
<EI_HOME>/repository/deployment/server/eventpublishers/MessageFlowStatisticsPublisher.xml
10
Property Name Description
username The username to be used when accessing the
WSO2 EI Analytics server to publish
configurations.
password The password to be used when accessing the
WSO2 EI Analytics server to publish
configurations.
receiver URL The URL of the thrift port to which the EI
configurations should be published. The format of
the URL is as follows.
tcp://<localhost>:<THRIFT_PORT>
12. Real-time Analytics
• Process events as and when received
• Default Siddhi App consists of Siddhi queries, input, and output streams
12
-- if DecompressedEventStream's componentType is ProxyService or API or InboundEndPoint, add
it into PreProcessedESBStatStream
from DecompressedEventStream[componentType == "proxy service" or componentType == "api" or
componentType == "inbound endpoint"]
select componentId, componentName, componentType, duration, ifThenElse(faultCount > 0, 1, 0)
as faultCount, startTime, entryPoint, metaTenantId
insert current events into PreProcessedESBStatStream;
13. Monitoring ESB Profile with Analytics
Dashboard
• ESB profile statistics overview
• Statistics for proxy services
• Statistics for sequences
• Statistics for REST APIs
• Statistics for endpoints and inbound endpoints
• Statistics for mediators and messages
13
15. Data Purging
• By default enabled in Siddhi App
• Defined using @purge
@store(type = 'rdbms', datasource = 'EI_ANALYTICS')
@info(name = 'MediatorStat')
@purge(enable='true', interval='60 min', @retentionPeriod(sec='1 day', min='72 hours', hours='90 days', days='1
year', months='2 years', years='3 years'))
define aggregation MediatorStatAgg
from PreProcessedMediatorStatStream
select metaTenantId, entryPoint, entryPointHashcode, componentId, hashCode, componentName, componentType,
sum(duration) as totalDuration ,avg(duration) as avgDuration, min(duration) as minDuration, max(duration) as
maxDuration, count() as noOfInvocation, sum(faultCount) as faultCount, startTime
group by metaTenantId, componentId, componentName, componentType, entryPoint, entryPointHashcode, hashCode
aggregate by startTime every sec...years;
19. Generate Reports via BPMN-Explorer
• Process Instance Count
• Average Time Duration Of
Completed Processes
• Task Durations Of Completed
Process
• User Vs Number Of Tasks
Completed
• User Vs Average Time Taken to
Complete Tasks
• Task Instance Count Variation
Over Time
• Process Instance Count Variation
Over Time
• User Performance Of Tasks
Instances Over Time
22. Broker-Specific MBeans
• JMX monitoring on following MBeans using JConsole or other third party monitoring tool.
(AppDynamics)
22
Property Name Description
ClusterManagementInformation MBean Cluster specific attributes.
MessageStatusInformation MBean Retrieve the lifecycle status of the
messages in the store to a specific Excel
sheet.
QueueManageInformation MBean Queue specific attributes and operations.
SubscriptionManageInformation MBean Subscription specific operations.
26. JMX Monitoring
• WSO2 Enterprise Integrator exposes a number of management resources as JMX MBeans
which helps to monitor the running server.
• To monitor statistics relating to artifacts that are deployed in the EI server Mediation Flow
Statistics View Mbean can be used.
26
28. Customize Statistics Publishing
• A custom statistics observer can be created by implementing
org.wso2.carbon.das.messageflow.data.publisher.observer.MessageFlowObserver interface.
28
public class MyStatisticsObserver implements MessageFlowObserver {
//Invoked when this observer is notified
public void updateStatistics(PublishingFlow publishingFlow) {
//Process data
}
}
<MediationFlowStatisticConfig>
<Observers>
org.wso2.custom.MyStatisticsObserver
</Observers>
</MediationFlowStatisticConfig>
● Add all custom observers as a comma separated list in the <EI_HOME>/conf/carbon.xml.
29. EI logs Monitoring with Elastic Stack
• Setting up Filebeat to read log files from EI.
• Update filebeat-yaml with the path to EI logs folder.
• Setting up Logstash to take the log lines from Filebeat, convert them to JSON strings, and ship
them to Elasticsearch.
• Setting up Elasticsearch to store and search logs.
• Setting up Kibana to monitor EI.
29
30. ESB Statistic Monitoring with ElasticStack
• Custom message flow observer implemented to publish EI statistics to ElasticStatisticPublisher.
30
33. EI Performance Monitoring with
AppDynamics
• Configuring App Agent and WSO2 Enterprise Integrator
Edit the integrator.sh in the EI-Installation/bin by inserting the line-
-javaagent:"/path-to-app-agent-directory/javaagent.jar"
33