Developing, Testing and Scaling with Apache Camel - UberConf 2015

Photos by

Developing, Testing and Scaling with Apache Camel
Matt Raible • http://raibledesigns.com

Blogger on raibledesigns.com
Founder of AppFuse
Father, Skier, Mountain
Biker, Whitewater Rafter
Web Framework Connoisseur
Who is Matt Raible?
Bus Lover

What about YOU?
Are you familiar with Enterprise Integration Patterns?

Have you used Apache Camel?

What about Spring Integration?

XML or JavaConﬁg?

How do you feel about testing?

Do you want to take a Camel ride?

What is Apache Camel?
Apache Camel is a versatile open-source
integration framework based on known
Enterprise Integration Patterns.

Enterprise Integration
Why is it necessary?

Businesses need to integration diﬀerent systems

Why a framework?

To allow you to focus on the business logic

Others have done it before, don’t reinvent the wheel

Simpliﬁed testing

Camel EIPs http://camel.apache.org/eip

Content Based Router
from newOrder
choice
when isWidget to widget
otherwise to gadget

from(newOrder)
choice()
when(isWidget) to(widget)
otherwise to(gadget)

from(newOrder)
.choice()
.when(isWidget).to(widget)
.otherwise().to(gadget);

Endpoint newOrder = endpoint(“activemq:queue:newOrder");
from(newOrder)
.choice()

Endpoint newOrder = endpoint("activemq:queue:newOrder");
Predicate isWidget = xpath("/order/product = 'widget"); 
 
from(newOrder)
.choice()

Predicate isWidget = xpath("/order/product = 'widget");
Endpoint widget = endpoint("activemq:queue:widget");
Endpoint gadget = endpoint("activemq:queue:gadget");
from(newOrder)
.choice()

import org.apache.camel.Endpoint;
import org.apache.camel.Predicate;
import org.apache.camel.builder.RouteBuilder;
public class MyRoute extends RouteBuilder {
public void configure() throws Exception {
Predicate isWidget = xpath("/order/product = 'widget");
Endpoint widget = endpoint("activemq:queue:widget");
Endpoint gadget = endpoint("activemq:queue:gadget");
from(newOrder)
.choice()
.otherwise().to(gadget)
.end();
}
}
}

Content Based Router: Java DSL
import org.apache.camel.builder.RouteBuilder;
public class MyRoute extends RouteBuilder {
from("activemq:queue:newOrder")
.choice()
.when(xpath("/order/product = 'widget"))
.to("activemq:queue:widget")
.otherwise()
.to("activemq:queue:gadget")
.end();
}
}
}

Content Based Router: XML DSL
<route>
<from uri="activemq:queue:newOrder"/>
<choice>
<when>
<xpath>/order/product = 'widget'</xpath>
<to uri="activemq:queue:widget"/>
</when>
<otherwise>
<to uri="activemq:queue:gadget"/>
</otherwise>
</choice>
</route>

Spring XML
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>

<from uri="stream:in?promptMessage=Enter something: "/>

<transform>
<simple>${body.toUpperCase()}</simple>
</transform>

<to uri="stream:out"/>
</route>
</camelContext>

public static void main
import org.apache.camel.spring.Main;
/**
* A main class to run the example from your editor.
*/
public final class CamelConsoleMain {
private CamelConsoleMain() {}
public static void main(String[] args) throws Exception {
// Main makes it easier to run a Spring application
Main main = new Main();
// configure the location of the Spring XML file
main.setApplicationContextUri("META-INF/spring/camel-context.xml");
// enable hangup support allows Camel to detect when the JVM is terminated
main.enableHangupSupport();
// run and block until Camel is stopped (or JVM terminated)
main.run();
}
}

Camel Components
Task Automation: Timer, Quartz2

Amazon: AWS-CW, AWS-DDB, AWS-SES, AWS-S3

Basics: Bean, Class, Context, Data Format, Exec, Language, Printer

Chat: IRC, XMPP

Endpoints: Direct, Direct-VM, Disruptor, SEDA

ESB: JBI, NMR, Vert.x

JMS: ActiveMQ, JMS, RabbitMQ, Stomp, ZeroMQ

Developing, Testing and Scaling with Apache Camel - UberConf 2015

What is Apache Camel?
An integration framework

Supports Enterprise Integration Patterns (EIP)

Routing (many DSLs to choose from)

Easy Conﬁguration (endpoints as URIs)

Just Java, XML, Scala, etc.

No Container Dependency

Hundreds of components

My Experience
First learned about Apache Camel
from Bruce Snyder in 2008 at
Colorado Software Summit

Asked to replace IBM Message
Broker last year (2014) for a client

James Strachan
Me: “Do you know of any particular guides
about migrating from IBM Message Broker
to Fuse?”

Strachan: “TBH the easiest thing really is to just start using Apache
Camel for this kinda stuﬀ”

Result: Recommended Apache Camel to my client and went to work

Getting Started
Started with a Camel Archetype

Used Java 7

Used Java DSL to deﬁne routes

Development Strategy

1. Write an integration test against existing service

2. Implement the service with Camel; unit test

3. Copy logic from step 1; write integration test

Legacy Integration Test
@Test
public void sendGPIRequestUsingSoapApi() throws Exception {
SOAPElement bodyChildOne = getBody(message).addChildElement("gpiRequest", "m");
SOAPElement bodyChildTwo = bodyChildOne.addChildElement("args0", "m");
bodyChildTwo.addChildElement("NDC", "ax22").addTextNode("54561237201");
SOAPMessage reply = connection.call(message, getUrlWithTimeout(SERVICE_NAME));
if (reply != null) {
Iterator itr = reply.getSOAPBody().getChildElements();
Map resultMap = TestUtils.getResults(itr);
assertEquals("66100525123130", resultMap.get("GPI"));
}
}

Drug Service Implementation
@WebService
public interface DrugService {
@WebMethod(operationName = "gpiRequest")
GpiResponse findGpiByNdc(GpiRequest request);
}

Spring Conﬁguration
@Configuration
@ImportResource("classpath:META-INF/cxf/cxf.xml")
@ComponentScan("com.raibledesigns.camel")
public class CamelConfig extends CamelConfiguration {
@Override
protected void setupCamelContext(CamelContext camelContext) throws Exception {
PropertiesComponent pc = new PropertiesComponent();
pc.setLocation("classpath:application.properties");
camelContext.addComponent("properties", pc);
super.setupCamelContext(camelContext);
}
}

CXF Servlet
@Override
public void onStartup(ServletContext servletContext) throws ServletException {
servletContext.addListener(new ContextLoaderListener(getContext()));
ServletRegistration.Dynamic servlet =
servletContext.addServlet("CXFServlet", new CXFServlet());
servlet.setLoadOnStartup(1);
servlet.setAsyncSupported(true);
servlet.addMapping("/api/*");
}
private AnnotationConfigWebApplicationContext getContext() {
AnnotationConfigWebApplicationContext context =
new AnnotationConfigWebApplicationContext();
context.setConfigLocation("com.raibledesigns.camel.config");
return context;
}
}

Drug Route
@Component
public class DrugRoute extends RouteBuilder {
private String uri = "cxf:/drugs?serviceClass=" + DrugService.class.getName();
@Override
from(uri)
.recipientList(simple("direct:${header.operationName}"));
from("direct:gpiRequest").routeId("gpiRequest")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
// get the ndc from the input
String ndc = exchange.getIn().getBody(GpiRequest.class).getNDC();
exchange.getOut().setBody(ndc);
}
})
.to("sql:{{sql.selectGpi}}")
.to("log:output")
.process(new Processor() {
// get the gpi from the input
List<HashMap> data = (ArrayList<HashMap>) exchange.getIn().getBody();
DrugInfo drug = new DrugInfo();
if (data.size() > 0) {
drug = new DrugInfo(String.valueOf(data.get(0).get("GPI")));
}
GpiResponse response = new GpiResponse(drug);
exchange.getOut().setBody(response);
}
});
}
}

Unit Testing
Hardest part was ﬁguring out Camel’s testing support

Bought Camel in Action, read chapter 6

Went to work

Eliminated dependency on datasource

Modiﬁed route and intercepted SQL calls

Unit Testing: DrugRouteTests.java
@RunWith(CamelSpringJUnit4ClassRunner.class)
@ContextConfiguration(loader = CamelSpringDelegatingTestContextLoader.class,
classes = CamelConfig.class)
@DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD)
@UseAdviceWith
public class DrugRouteTests {
@Autowired
CamelContext camelContext;
@Produce
ProducerTemplate template;
@EndpointInject(uri = "mock:result")
MockEndpoint result;
static List<Map> results = new ArrayList<Map>() {{
add(new HashMap<String, String>() {{
put("GPI", "123456789");
}});
}};
// continued on next slide
}

Unit Testing: @Before
@Before
public void before() throws Exception {
camelContext.setTracing(true);
ModelCamelContext context = (ModelCamelContext) camelContext;
RouteDefinition route = context.getRouteDefinition("gpiRequest");
route.adviceWith(context, new RouteBuilder() {
@Override
interceptSendToEndpoint("sql:*").skipSendToOriginalEndpoint().process(new Processor() {
@Override
exchange.getOut().setBody(results);
}
});
}
});
route.to(result);
camelContext.start();
}

Unit Testing: @Test
@Test
public void testMockSQLEndpoint() throws Exception {
result.expectedMessageCount(1);
GpiResponse expectedResult = new GpiResponse(new DrugInfo("123456789"));
result.allMessages().body().contains(expectedResult);
GpiRequest request = new GpiRequest();
request.setNDC("123");
template.sendBody("direct:gpiRequest", request);
MockEndpoint.assertIsSatisfied(camelContext);
}

Code Coverage
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<configuration>
<instrumentation>
<excludes>
<exclude>**/model/*.class</exclude>
<exclude>**/AppInitializer.class</exclude>
<exclude>**/StoredProcedureBean.class</exclude>
<exclude>**/SoapActionInterceptor.class</exclude>
</excludes>
</instrumentation>
<check/>
</configuration>
<version>2.6</version>
</plugin>
...
<reporting>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.6</version>
</plugin>

Integration Testing
public class DrugRouteITest {
private static final String URL = "http://localhost:8080/api/drugs";
protected static DrugService createCXFClient() {
JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setBindingId("http://schemas.xmlsoap.org/wsdl/soap12/");
factory.setServiceClass(DrugService.class);
factory.setAddress(getTestUrl(URL));
return (DrugService) factory.create();
}
@Test
public void findGpiByNdc() throws Exception {
// create input parameter
GpiRequest input = new GpiRequest();
input.setNDC("54561237201");
// create the webservice client and send the request
DrugService client = createCXFClient();
GpiResponse response = client.findGpiByNdc(input);
assertEquals("66100525123130", response.getDrugInfo().getGPI());
}
}

Integrating Spring Boot
Had to upgrade to Spring 4

Camel 2.13.1 didn’t support Spring 4

Camel 2.14-SNAPSHOT, CXF 3.0

Found issues with camel-test-spring, ﬁxed, created pull request
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- <camel.version>2.13.1</camel.version>
- <cxf.version>2.7.11</cxf.version>
- <spring.version>3.2.8.RELEASE</spring.version>
+ <camel.version>2.14-SNAPSHOT</camel.version>
+ <cxf.version>3.0.0</cxf.version>
+ <spring.version>4.0.5.RELEASE</spring.version>
</properties>
Learn more: http://raibledesigns.com/rd/entry/developing_services_with_apache_camel2

Microservices Deployment
http://martinfowler.com/articles/microservices.html

Camel 2.15 includes Spring Boot Support
Auto-conﬁguration of Camel context

Auto-detects Camel routes

Registers key Camel utilities

Producer Template

Consumer Template

Type Converter

Connects Spring Boot’s external conﬁguration with Camel properties
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring-boot</artifactId>
<version>${camel.version}</version>
</dependency>

High Performance

Scala, Akka and Netty

Nice HTML-based Reporting

Scenario Recorder
Load Testing with Gatling
val scn = scenario("Test the Blog entity") 
.exec(http("First unauthenticated request") 
.get("/api/account") 
.headers(headers_http) 
.check(status.is(401))) 
.pause(10) 
.exec(http("Authentication") 
.post("/api/authenticate") 
.headers(headers_http_authentication) 
.formParam("username", "admin") 
.formParam("password", "admin") 
.check(jsonPath("$.token").saveAs("x_auth_token"))) 
.pause(1) 
.exec(http("Authenticated request") 
.get("/api/account") 
.headers(headers_http_authenticated) 
.pause(10) 
.repeat(2) { 
exec(http("Get all blogs") 
.get("/api/blogs") 
.pause(10 seconds, 20 seconds) 
.exec(http("Create new blog") 
.put("/api/blogs") 
.body(StringBody("""{"id":null, "name":"SAMPLE_TEXT", "handle"

1. Write tests to run against current system. Find the number of
concurrent requests that make it fall over.

2. Run tests against new system and tune accordingly.

3. Throttle requests if there are remote connectivity issues with 3rd
parties. If I needed to throttle requests, I was planning to use
Camel's Throttler.
Gatling Approach

Used Gatling’s Recorder

Listened on port 8000

Changed DrugServiceTest to use same port

Ran integration test

Created AbstractSimulation.scala to allow changing parameters
My Gatling Strategy

AbstractSimulation.scala
import io.gatling.core.scenario.Simulation
import io.gatling.http.Predef._
/**
* Base Simulation class that allows passing in parameters.
*/
class AbstractSimulation extends Simulation {
val host = System.getProperty("host", "localhost:8080")
val serviceType = System.getProperty("service", "modern")
val nbUsers = Integer.getInteger("users", 10).toInt
val rampRate = java.lang.Long.getLong("ramp", 30L).toLong
val httpProtocol = http
.baseURL("http://" + host)
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.doNotTrackHeader("1")
.acceptLanguageHeader("en-US,en;q=0.5")
.acceptEncodingHeader("gzip, deflate")
.userAgentHeader("Gatling 2.0")
val headers = Map(
"""Cache-Control""" -> """no-cache""",
"""Content-Type""" -> """application/soap+xml; charset=UTF-8""",
"""Pragma""" -> """no-cache""")
}

DrugServiceSimulation.scala
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class DrugServiceSimulation extends AbstractSimulation {
val service = if ("modern".equals(serviceType)) "/api/drugs"
else "/axis2/services/DrugService"
val scn = scenario("Drug Service :: findGpiByNdc")
.exec(http(host)
.post(service)
.headers(headers)
.body(RawFileBody("DrugServiceSimulation_request.xml")))
setUp(scn.inject(ramp(nbUsers users) over (rampRate seconds))).protocols(httpProtocol)
}

Executing Simulations
Legacy service with 100 users over 60 seconds

mvn test -Dhost=legacy.server:7802 -Dservice=legacy -Dusers=100 -Dramp=60
Local drug service with 100 users over 30 seconds (defaults used)

mvn test -Dusers=100
Results

Legacy service started failing at 400 requests per second (rps)

Local service started throwing errors at 4000/rps

Data Feeders
JDBC Feeder allowed making requests contain unique data
for each user

ELFileBody allows substituting a ${NDC} variable in XML ﬁle
val feeder = jdbcFeeder("jdbc:db2://server:50002/database", "username", "password",
"SELECT NDC FROM GENERICS")
val scn = scenario("Drug Service")
.feed(feeder)
.exec(http(host)
.post(service)
.headers(headers)
.body(ELFileBody("DrugServiceSimulation_request.xml")))

Performance Results
100 users over 30 seconds

No failures

Max response time: 389ms for legacy, 172ms for Camel service

1000 users over 60 seconds

Legacy service: 50% of requests failed, avg. response time over 40s

New service: all requests succeeded, response mean time 100ms

Monitoring
 
“a modular web console for managing your Java stuﬀ”

Camel plugin shows routes and metrics, route source editable (XML)

Summary
Open source wins again!

Pleasant experience developing with Apache Camel and Spring

Integration tests were essential to verifying functionality

Gatling Feeders helped discover edge cases with real data

What about Spring Integration?

Never looked into it, Camel was good enough

Action!
Don’t be afraid to try new things

Try Apache Camel if you need
Enterprise Integration

Spring Boot Rocks! Try it if you’re
already using Spring

Test, Test, Test, then test some more

Gatling is a great way to ﬁnd our
performance and test with real data

Contact Information

http://raibledesigns.com

@mraible

Presentations

http://slideshare.net/mraible

Code

http://github.com/mraible
Questions?

Additional Information
Camel in Action

2nd edition on its way (summer 2016)!

Microservices with Spring Boot, Dropwizard, Jetty, CDI

Cloud, Docker, Kubernetes, Fabric8, Hawtio

Reactive with RxJava and Vert.x

Follow its authors: @davsclaus and @jon_anstey

http://camel.apache.org

Devoxx4Kids Denver
Teaching Kids to Program

Java, Minecraft, robots, oh my!

Non-proﬁt, looking for speakers!

http://www.meetup.com/Devoxx4Kids-Denver/

Developing, Testing and Scaling with Apache Camel - UberConf 2015

Related slideshows

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

Recommended for you

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Developing, Testing and Scaling with Apache Camel - UberConf 2015

Similar to Developing, Testing and Scaling with Apache Camel - UberConf 2015 (20)

More from Matt Raible

More from Matt Raible (20)

Recently uploaded

Recently uploaded (20)

Developing, Testing and Scaling with Apache Camel - UberConf 2015