SlideShare a Scribd company logo

DataPower-MQ Integration Deep Dive

•
•
Morag Hughson
Morag Hughson

Presentation from Robin Wiley. DataPower can connect to MQ as a client, sending and receiving messages to and from queues. Unlike traditional MQ client programming, the DataPower client interface has certain predefined functions that limit the scope of activities that can be performed. This session will explain what you can and can't do when integrating DataPower into your MQ infrastructure. Topics covered will include syncpoint processing, message grouping, message properties, JMS considerations, connec-tion options, and many more.

Related slideshows

Data power use cases
Data power use casesData power use cases
Data power use cases
 
IBM Datapower Security Scenario with JWS & JWE
IBM Datapower Security Scenario with JWS & JWEIBM Datapower Security Scenario with JWS & JWE
IBM Datapower Security Scenario with JWS & JWE
 
APIC/DataPower security
APIC/DataPower securityAPIC/DataPower security
APIC/DataPower security
 
1 of 47
WebSphere Technical University, Dusseldorf, 2014 225 E. City Avenue Suite 106 Bala Cynwyd, PA 19004 610-206-0101 www.LearnQuest.com Presents DataPower-MQ Integration Deep Dive
2 Your Presenter: Robin Wiley ► Lead Instructor, IBM WebSphere Training, LearnQuest MQ Administration Application Development DataPower Administration Service Development ► IBM Certified: MQ V7 Administrator MQ V7 Solution Designer DataPower Solution Implementer ► Over 40 years IT Industry Experience Network Integration Managing Software Development ► Experience: 18 years with MQ; 8 with DataPower Technical Architecture, Project Management, Installation, Training
3 Agenda ► Queue Manager Object ► Queue Manager Group Object ► MQ Front Side Handler ► MQ URL ► Message Headers ► Error Handling ► Transaction Management
4 Queue Manager Object
5 Queue Manager Object ► Host Name (IPv4) address:port 192.168.57.1:1414 address(port) 192.168.57.1(1414) address 192.168.57.1 default port 1414 ► Host Name (IPv6) [address]:port [2202::148:248]:1414 address(port) 2202::148:248(1414) address 2202::148:248 default port 1414 ► Host Name (DNS) hostname:port myserver:1414 hostname(port) myserver(1414) hostname myserver default port 1414
6 Queue Manager Object
7 Queue Manager Object
8 Queue Manager Object ► Channel Name SVRCONN name as defined on the Queue Manager ► Channel Heartbeat (seconds) Approximate time between heartbeat flows on the channel 0 = no heartbeat flow exchanged Does not set the heartbeat on the channel Negotiates heartbeat value with channel -- greater is used ► Cache Timeout (seconds) How long the appliance keeps alive a dynamic connection in the connection cache Must be greater than the negotiated heartbeat interval but less than the Queue Manager keep alive interval (defined on the host)
9 Queue Manager Object ► User Name Supplied to Queue Manager at connection Maximum 12 characters ► Alternate User Off (default) = use Message Descriptor User Identifier for queue authorization On = use Object Descriptor Alternate User Identifier for queue authorization (need to create Object Descriptor) ► XML Manager Recommend using a dedicated XML Manager per QM object
10 Queue Manager Object ► Maximum Message Size Limit the size of the MQ message payload ► Units of Work Controls syncpoint processing (transaction management) Affects MQ Front Side Handlers only Two values: 0 or 1 ► Units of Work = 0 No syncpoint control Front Side Handler MQGET is immediate and irrevocable If error, message integrity is responsibility of DataPower
11 Queue Manager -- Connections
12 Queue Manager -- Connections ► Total Connection Limit: Connection pool size of the QM object Default value is 250 Can tune the total-connection-limit for performance ► Local Address Local address for outbound connections Specific local interface and port For a range of ports, use (1414,1420) or x.x.x.x(1414,1420)
13 Queue Manager -- Connections ► Retry Interval Time interval between attempts to retry failed connections Recommend 10 to 15 seconds (default is 1 second) Low value can spike CPU and memory usage ► Retry Attempts Number of attempts to retry the failed connections After threshold reached, the Long Retry Interval is used instead Default value of 0 (zero) disables Long Retry Interval Recommend non-zero value
14 Queue Manager -- Connections ► Long Retry Interval Interval in seconds to retry connection after Retry Attempts Recommend 600 seconds (default is 1800) Must be greater than the Retry Interval (if less, ignored) ► Reporting Interval How often to log retries (seconds) Suppresses duplicate log entries Recommend setting this the same as Retry Interval
15 Queue Manager -- Connections
16 Queue Manager -- Connections ► Sharing Conversations Maximum conversations sharing single TCP/IP connection Value is negotiated between SVRCONN SHARECNV setting and DataPower (lower value takes effect) Value of 0 means NO Shared Conversations • Suppresses MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value of 1 means NO Shared Conversations • Allows MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value 1 means Shared Conversations permitted • Allows MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Can impact performance of clients (unless V7 features used)
17 MQ Front Side Handler
18 MQ Front Side Handler ► Get Queue Name of queue to get messages from Mandatory, unless Pub/Sub being used ► Put Queue Optional, because: • May be one-way messaging (fire and forget) • May be using Reply-To Queue • May be dynamically allocated by Response Rule code ► The number of concurrent MQ connections: Number of active and pending MQGETs for the Get Queue Recommend value of 1 If greater than 1, monitor Queue Manager for workload If using message ordering, set to 1
19 MQ Front Side Handler ► Get Message Options Allows the use of any MQGMO_Options parameters Overrides any specific parameters set elsewhere ► Polling Interval Pause period between MQGET calls (seconds) Low value increases network traffic Recommend default of 30 ► Retrieve Backout Settings Get BOTHRESH and BOQUEUE from the Get Queue Only relevant if queue parameters were set by administrator Recommend set off and use Queue Manager Object settings
20 MQ Front Side Handler ► Use Queue Manager in URL Defines the behavior of var://service/URL-in If on, the variable returns the name of the Queue Manager If off the variable returns the name of the Queue Manager Group Default off
21 MQ Front Side Handler
22 MQ Front Side Handler ► Subscribe Topic String Pub/Sub topic string for subscription If Get Queue also defined, this is ignored ► Subscription Name Used to resume a Durable Subscription ► Publish Topic String Pub/Sub topic string for response publication If Put Queue also defined, this is ignored ► Parse Properties Extracts MQ V7 (and above) Message Properties into Node Set Minor overhead, so leave off unless needed
23 MQ Front Side Handler ► Selector Allows selective retrieval of messages based on properties Forces sequential search of queue so may be inefficient ► Exclude Message Headers Strip of selected MQ header types ► Header to extract Content-Type Can obtain Content-Type from • MQMD • RFH • RFH2
24 MQ Front Side Handler ► Async Put Put message to queue without waiting for a response Do not use when Queue Manager units-of-work is 1 Recommend only use where performance is an issue ► Batch Size Number of messages to handled as a singe batch Recommend leave this as zero
25 Back End URL ► General Syntax: dpmq://mqQueueManagerObject/URI?parameters ► RequestQueue=requestQueueName Name of the backend MQ request queue ► ReplyQueue=replyQueueName Name of the backend MQ reply queue ► Sync=true Issues a Commit call when a message is put on Request Queue ► GMO=optionsValue MQGMO_Options parameter value when getting from Reply Queue ► PMO=optionsValue MQPMO_Options parameter value when putting to Request Queue
26 Back End URL ► ParseHeaders={true|false} Specifies whether to parse and strip headers from message ► SetReplyTo={true|false} Specifies whether to set MD ReplyToQ during Put ► AsyncPut={true|false} Specifies whether to use Asynchronous Put Only valid when using MQ V7 ► Browse={first|next|current} Controls non-destructive retrieval of messages
27 Back End URL ► ContentTypeHeader=header Which MQ header identifies the content type of the message ► ContentTypeXPath=expression XPath expression to extract the content type of message ► ParseProperties={on|off} Parse message properties ► PublishTopicString=string and SubscribeTopicString=string Specifies topic to use with Pub/Sub (MQ V7 and above) ► SubscriptionName=string Specifies name for a durable subscription (MQ V7 and above) ► Selector=expression SQL92 style query filtering on message properties
28 Message Header Manipulation ► Using Header Injection (Header Tab) ► Example setting Format and Persistence: Direction: Back (or Backend MQPUT) Header Name: MQMD Header Value: MQMDFormatMQSTR/FormatPersistence1/Persistence/MQMD
29 Message Header Manipulation ► Using the Stylesheet method (page 1 of 2)
30 Message Header Manipulation ► Using the Stylesheet method (page 2 of 2)
31 Message Header Manipulation ► Context variable method to inject the MQMD header For the following code to work: Set Transform Action's OUTPUT context to EVENTS Set Result Action's INPUT context to EVENTS
32 Error Handling ► MQ error handling example:
33 Conversational Processing ► Backend application must copy Msgid to CorrelId Back end retrieves reply using CorrelId ► If MQ Object Descriptor header is supplied, Request Rule issues remote open to back end Queue Manager ► If ReplyToQ is set, Response Rule sends message there ► If destination is a Cluster, no need to supply ReplyToQmgr
34 Conversational Processing ► XSL code snippet to set ReplyToQ and ReplyToQmgr in a Request Rule:
35 Conversational Processing ► XSL code snippet to set ReplyToQ and ReplyToQmgr in a Response Rule:
36 Transactional Control ► DataPower is considered an application by MQ Therefore, no inherent message integrity ► If the same Queue Manager at front and back: True message integrity Once and once-only delivery ► If different Queue Managers at front and back No possibility of two-phase ocmmit Message integrity assured if DataPower configured properly Possibility of messages sent more than once
No Transactional Control App 1 App 2 Queue Manager QM2 37 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 0 Sync=false; Transactional=false
Front Side Transactional Control only App 1 App 2 Queue Manager QM2 38 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=false
Back End Transactional Control only App 1 App 2 Queue Manager QM2 39 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 0 Sync=false; Transactional=true
Front Side and Back End Transactional Control App 1 App 2 Queue Manager QM2 40 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=true
41 Single Queue Manager App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN No Transactional Control Units of Work = 0 Sync=false; Transactional=false
42 Single Queue Manager Front Side Transactional Control only App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=false
43 Single Queue Manager Front Side Transactional Control only App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=true; Transactional=false
44 Single Queue Manager Front Side and Back End Transactional Control App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=true; Transactional=true
45 Poison Message FS.IN Queue Manager QM1 DataPower Request Rule App 1 Units of Work = 1 FS.ERR FSH BOTHRESH = 3 BOQUEUE = FS.ERR
46 Poison Message FS.IN Queue Manager QM1 DataPower Request Rule App 1 Units of Work = 1 FS.ERR FSH BOTHRESH = 3 BOQUEUE = FS.ERR
End of Session Thank You! To learn more, come to Lab IL5 – Wednesday 3:15pm – Room 7b Contact: robin.wiley@learnquest.com

More Related Content

DataPower-MQ Integration Deep Dive

  • 1. WebSphere Technical University, Dusseldorf, 2014 225 E. City Avenue Suite 106 Bala Cynwyd, PA 19004 610-206-0101 www.LearnQuest.com Presents DataPower-MQ Integration Deep Dive
  • 2. 2 Your Presenter: Robin Wiley ► Lead Instructor, IBM WebSphere Training, LearnQuest MQ Administration Application Development DataPower Administration Service Development ► IBM Certified: MQ V7 Administrator MQ V7 Solution Designer DataPower Solution Implementer ► Over 40 years IT Industry Experience Network Integration Managing Software Development ► Experience: 18 years with MQ; 8 with DataPower Technical Architecture, Project Management, Installation, Training
  • 3. 3 Agenda ► Queue Manager Object ► Queue Manager Group Object ► MQ Front Side Handler ► MQ URL ► Message Headers ► Error Handling ► Transaction Management
  • 5. 5 Queue Manager Object ► Host Name (IPv4) address:port 192.168.57.1:1414 address(port) 192.168.57.1(1414) address 192.168.57.1 default port 1414 ► Host Name (IPv6) [address]:port [2202::148:248]:1414 address(port) 2202::148:248(1414) address 2202::148:248 default port 1414 ► Host Name (DNS) hostname:port myserver:1414 hostname(port) myserver(1414) hostname myserver default port 1414
  • 8. 8 Queue Manager Object ► Channel Name SVRCONN name as defined on the Queue Manager ► Channel Heartbeat (seconds) Approximate time between heartbeat flows on the channel 0 = no heartbeat flow exchanged Does not set the heartbeat on the channel Negotiates heartbeat value with channel -- greater is used ► Cache Timeout (seconds) How long the appliance keeps alive a dynamic connection in the connection cache Must be greater than the negotiated heartbeat interval but less than the Queue Manager keep alive interval (defined on the host)
  • 9. 9 Queue Manager Object ► User Name Supplied to Queue Manager at connection Maximum 12 characters ► Alternate User Off (default) = use Message Descriptor User Identifier for queue authorization On = use Object Descriptor Alternate User Identifier for queue authorization (need to create Object Descriptor) ► XML Manager Recommend using a dedicated XML Manager per QM object
  • 10. 10 Queue Manager Object ► Maximum Message Size Limit the size of the MQ message payload ► Units of Work Controls syncpoint processing (transaction management) Affects MQ Front Side Handlers only Two values: 0 or 1 ► Units of Work = 0 No syncpoint control Front Side Handler MQGET is immediate and irrevocable If error, message integrity is responsibility of DataPower
  • 11. 11 Queue Manager -- Connections
  • 12. 12 Queue Manager -- Connections ► Total Connection Limit: Connection pool size of the QM object Default value is 250 Can tune the total-connection-limit for performance ► Local Address Local address for outbound connections Specific local interface and port For a range of ports, use (1414,1420) or x.x.x.x(1414,1420)
  • 13. 13 Queue Manager -- Connections ► Retry Interval Time interval between attempts to retry failed connections Recommend 10 to 15 seconds (default is 1 second) Low value can spike CPU and memory usage ► Retry Attempts Number of attempts to retry the failed connections After threshold reached, the Long Retry Interval is used instead Default value of 0 (zero) disables Long Retry Interval Recommend non-zero value
  • 14. 14 Queue Manager -- Connections ► Long Retry Interval Interval in seconds to retry connection after Retry Attempts Recommend 600 seconds (default is 1800) Must be greater than the Retry Interval (if less, ignored) ► Reporting Interval How often to log retries (seconds) Suppresses duplicate log entries Recommend setting this the same as Retry Interval
  • 15. 15 Queue Manager -- Connections
  • 16. 16 Queue Manager -- Connections ► Sharing Conversations Maximum conversations sharing single TCP/IP connection Value is negotiated between SVRCONN SHARECNV setting and DataPower (lower value takes effect) Value of 0 means NO Shared Conversations • Suppresses MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value of 1 means NO Shared Conversations • Allows MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value 1 means Shared Conversations permitted • Allows MQ V7 features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Can impact performance of clients (unless V7 features used)
  • 17. 17 MQ Front Side Handler
  • 18. 18 MQ Front Side Handler ► Get Queue Name of queue to get messages from Mandatory, unless Pub/Sub being used ► Put Queue Optional, because: • May be one-way messaging (fire and forget) • May be using Reply-To Queue • May be dynamically allocated by Response Rule code ► The number of concurrent MQ connections: Number of active and pending MQGETs for the Get Queue Recommend value of 1 If greater than 1, monitor Queue Manager for workload If using message ordering, set to 1
  • 19. 19 MQ Front Side Handler ► Get Message Options Allows the use of any MQGMO_Options parameters Overrides any specific parameters set elsewhere ► Polling Interval Pause period between MQGET calls (seconds) Low value increases network traffic Recommend default of 30 ► Retrieve Backout Settings Get BOTHRESH and BOQUEUE from the Get Queue Only relevant if queue parameters were set by administrator Recommend set off and use Queue Manager Object settings
  • 20. 20 MQ Front Side Handler ► Use Queue Manager in URL Defines the behavior of var://service/URL-in If on, the variable returns the name of the Queue Manager If off the variable returns the name of the Queue Manager Group Default off
  • 21. 21 MQ Front Side Handler
  • 22. 22 MQ Front Side Handler ► Subscribe Topic String Pub/Sub topic string for subscription If Get Queue also defined, this is ignored ► Subscription Name Used to resume a Durable Subscription ► Publish Topic String Pub/Sub topic string for response publication If Put Queue also defined, this is ignored ► Parse Properties Extracts MQ V7 (and above) Message Properties into Node Set Minor overhead, so leave off unless needed
  • 23. 23 MQ Front Side Handler ► Selector Allows selective retrieval of messages based on properties Forces sequential search of queue so may be inefficient ► Exclude Message Headers Strip of selected MQ header types ► Header to extract Content-Type Can obtain Content-Type from • MQMD • RFH • RFH2
  • 24. 24 MQ Front Side Handler ► Async Put Put message to queue without waiting for a response Do not use when Queue Manager units-of-work is 1 Recommend only use where performance is an issue ► Batch Size Number of messages to handled as a singe batch Recommend leave this as zero
  • 25. 25 Back End URL ► General Syntax: dpmq://mqQueueManagerObject/URI?parameters ► RequestQueue=requestQueueName Name of the backend MQ request queue ► ReplyQueue=replyQueueName Name of the backend MQ reply queue ► Sync=true Issues a Commit call when a message is put on Request Queue ► GMO=optionsValue MQGMO_Options parameter value when getting from Reply Queue ► PMO=optionsValue MQPMO_Options parameter value when putting to Request Queue
  • 26. 26 Back End URL ► ParseHeaders={true|false} Specifies whether to parse and strip headers from message ► SetReplyTo={true|false} Specifies whether to set MD ReplyToQ during Put ► AsyncPut={true|false} Specifies whether to use Asynchronous Put Only valid when using MQ V7 ► Browse={first|next|current} Controls non-destructive retrieval of messages
  • 27. 27 Back End URL ► ContentTypeHeader=header Which MQ header identifies the content type of the message ► ContentTypeXPath=expression XPath expression to extract the content type of message ► ParseProperties={on|off} Parse message properties ► PublishTopicString=string and SubscribeTopicString=string Specifies topic to use with Pub/Sub (MQ V7 and above) ► SubscriptionName=string Specifies name for a durable subscription (MQ V7 and above) ► Selector=expression SQL92 style query filtering on message properties
  • 28. 28 Message Header Manipulation ► Using Header Injection (Header Tab) ► Example setting Format and Persistence: Direction: Back (or Backend MQPUT) Header Name: MQMD Header Value: MQMDFormatMQSTR/FormatPersistence1/Persistence/MQMD
  • 29. 29 Message Header Manipulation ► Using the Stylesheet method (page 1 of 2)
  • 30. 30 Message Header Manipulation ► Using the Stylesheet method (page 2 of 2)
  • 31. 31 Message Header Manipulation ► Context variable method to inject the MQMD header For the following code to work: Set Transform Action's OUTPUT context to EVENTS Set Result Action's INPUT context to EVENTS
  • 32. 32 Error Handling ► MQ error handling example:
  • 33. 33 Conversational Processing ► Backend application must copy Msgid to CorrelId Back end retrieves reply using CorrelId ► If MQ Object Descriptor header is supplied, Request Rule issues remote open to back end Queue Manager ► If ReplyToQ is set, Response Rule sends message there ► If destination is a Cluster, no need to supply ReplyToQmgr
  • 34. 34 Conversational Processing ► XSL code snippet to set ReplyToQ and ReplyToQmgr in a Request Rule:
  • 35. 35 Conversational Processing ► XSL code snippet to set ReplyToQ and ReplyToQmgr in a Response Rule:
  • 36. 36 Transactional Control ► DataPower is considered an application by MQ Therefore, no inherent message integrity ► If the same Queue Manager at front and back: True message integrity Once and once-only delivery ► If different Queue Managers at front and back No possibility of two-phase ocmmit Message integrity assured if DataPower configured properly Possibility of messages sent more than once
  • 37. No Transactional Control App 1 App 2 Queue Manager QM2 37 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 0 Sync=false; Transactional=false
  • 38. Front Side Transactional Control only App 1 App 2 Queue Manager QM2 38 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=false
  • 39. Back End Transactional Control only App 1 App 2 Queue Manager QM2 39 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 0 Sync=false; Transactional=true
  • 40. Front Side and Back End Transactional Control App 1 App 2 Queue Manager QM2 40 Two Queue Managers Queue Manager QM1 FS.IN DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=true
  • 41. 41 Single Queue Manager App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN No Transactional Control Units of Work = 0 Sync=false; Transactional=false
  • 42. 42 Single Queue Manager Front Side Transactional Control only App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=false; Transactional=false
  • 43. 43 Single Queue Manager Front Side Transactional Control only App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=true; Transactional=false
  • 44. 44 Single Queue Manager Front Side and Back End Transactional Control App 1 App 2 FS.IN Queue Manager QM1 DataPower Request Rule Response Rule FS.OUT BE.OUT BE.IN Units of Work = 1 Sync=true; Transactional=true
  • 45. 45 Poison Message FS.IN Queue Manager QM1 DataPower Request Rule App 1 Units of Work = 1 FS.ERR FSH BOTHRESH = 3 BOQUEUE = FS.ERR
  • 46. 46 Poison Message FS.IN Queue Manager QM1 DataPower Request Rule App 1 Units of Work = 1 FS.ERR FSH BOTHRESH = 3 BOQUEUE = FS.ERR
  • 47. End of Session Thank You! To learn more, come to Lab IL5 – Wednesday 3:15pm – Room 7b Contact: robin.wiley@learnquest.com