More then 350.000 downloads for Pax Web

Now this is pretty amazing, just checked the recent download statistics at oss.sonatype and I have to say it’s pretty impressive.


Just within one year the downloads jumped from less then 150.000 to more then 350.000 downloads. That’s more then 200.000 more within the year for all org.ops4j.pax.web downloadable artifacts.

The same increase over the year can be found for the pax-web-jetty bundle. From about 20.000 downloads in November 2012 (I already thought of being pretty impressive), increased to more then 60.000 downloads in November 2013.

Thanks for using, it’s been a lot of fun enhancing Pax Web.

Testing Camel JPA routes with Pax-Exam and Karaf

This blog is about how to use JPA, CXF and ActiveMQ with Camel in a Karaf and how to do the testing best. I’ve had a couple of conversations about this already on IRC so here is my personal way of doing it best. You’ll find a sample of all of this in my repository at GitHub.

Project overview

Now to get things started a brief overview of how the structure of this project is.

Bildschirmfoto 2013-08-15 um 21.41.28

It’s a project containing 4 sub-modules. The project itself is only a reactor project to build all submodules. The submodules are entities, feature, pom and route-control. The entities module does contain all JPA related classes and DAOs as services, it uses Aries for working with JPA. The feature module contains a Karaf feature file descriptor for easy deployment in Karaf. The pom module contains the parent pom for all other modules for easy configuration of dependencies. The last module contains the camel routes using the JPA entities from the entities module, it also contains tests for the JPA and CXF services running inside Pax-Exam-Karaf.

I’ve chosen Pax-Exam with the Karaf container since it’s much easier for testing, one could try to rebuild all dependencies with Pax-Exam alone but still end up with most of the modules found in Karaf.

Feature

The features module contains the “manual” created features.xml file, since it’s still the best way of managing all required dependencies.

<features name="Camel-Demo" xmlns="http://karaf.apache.org/xmlns/features/v1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://karaf.apache.org/xmlns/features/v1.0.0 http://karaf.apache.org/xmlns/features/v1.0.0">
    <repository>mvn:org.apache.camel.karaf/apache-camel/${dependency.camel.version}/xml/features</repository>
    <repository>mvn:org.apache.activemq/activemq-karaf/${dependency.activemq.version}/xml/features</repository>
    <repository>mvn:org.apache.cxf.karaf/apache-cxf/${dependency.cxf.version}/xml/features</repository>

    <feature name='camel-exam-demo' version="${project.version}" resolver="(obr)">

        <!-- depending activemq features -->
        <feature version="${dependency.activemq.version}">activemq-blueprint</feature>
        <feature version="${dependency.activemq.version}">activemq-camel</feature>

        <!-- depending cxf features -->
        <feature version="${dependency.cxf.version}">cxf-jaxws</feature>

        <!-- Container dependencies -->
        <feature>transaction</feature>
        <feature>jpa</feature>
        <feature>jndi</feature>

        <!-- depending camel features -->
        <feature version="${dependency.camel.version}">camel-blueprint</feature>
        <feature version="${dependency.camel.version}">camel-jms</feature>
        <feature version="${dependency.camel.version}">camel-jpa</feature>
        <feature version="${dependency.camel.version}">camel-jdbc</feature>
        <feature version="${dependency.camel.version}">camel-mvel</feature>
        <feature version="${dependency.camel.version}">camel-cxf</feature>

        <bundle dependency="true">mvn:com.h2database/h2/${dependency.h2.version}</bundle>

        <bundle>mvn:${project.groupId}/route-control/${project.version}</bundle>
        <bundle>mvn:${project.groupId}/entities/${project.version}</bundle>
    </feature>
</features>

 Entities

The entities module contains the entities classes, persistence.xml and blueprint xml file for declaring services. As JPA framework OpenJPA is used since by the time of writing this sample application (not when I wrote this blog ;) ) it’s been the only one usable JPA implementation for OSGi.

Bildschirmfoto 2013-08-15 um 22.05.19

The Persistent entity class is defined in the root package de.nierbeck.demo.entities, the Interface for declaring a DAO to this entity is in the  de.nierbeck.demo.entities.dao package. The implementation which is hidden as it is a service is placed in the de.nierbeck.demo.entities.dao.impl package.

The persistence.xml does look like a usual persistence.xml though running in OSGi with Aries blueprint does take some specialties. Let’s take a detailed view of it.

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/persistence

http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"

    version="2.0" >
	<persistence-unit name="camelStore" transaction-type="JTA">
		<provider>org.apache.openjpa.persistence.PersistenceProviderImpl</provider>
        <jta-data-source>osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=jdbc/route-test-ds)</jta-data-source>

		<class>de.nierbeck.camel.exam.demo.entities.CamelMessage</class>

        <exclude-unlisted-classes>true</exclude-unlisted-classes>
        <validation-mode>NONE</validation-mode>
        <properties>
            <property name="openjpa.Log" value="slf4j" />
            <property name="openjpa.RuntimeUnenhancedClasses" value="supported" />
			<property name="openjpa.jdbc.SynchronizeMappings" value="buildSchema"/>
        </properties>
	</persistence-unit>
</persistence>

First of all the persistence unit needs a name since this is reference in the blueprint.xml. Since it’s running in a OSGi container with container managed transaction we need to set the transaction-type to JTA.

The next specialty is the jta-data-source reference. it’s a special JNDI lookup. It works since we also require the Aries JNDI project that enables to search for OSGi services through a JNDI lookup.

osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=jdbc/route-test-ds)

The jndi lookup is redirected to a service lookup and looks for the declared Interface which is appended with a OSGi ldap filter to make sure you only get that one service which is your desired datasource.

The blueprint xml is straightforward with it’s definitions, a bean is declare – the dao – and registered as a service in the service registry. Since it’s the dao bean we are declaring it needs a entity manager and the definition of the transitions. This is done through the aries jpa reference for the entity manager which will be injected into the property em and references the persistence defined with the name camelStore as it has been declared in the persistence.xml.

<blueprint default-activation="eager"
	xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:jpa="http://aries.apache.org/xmlns/jpa/v1.0.0" xmlns:tx="http://aries.apache.org/xmlns/transactions/v1.0.0"
    xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd http://camel.apache.org/schema/blueprint/camel-blueprint.xsd http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0/blueprint-cm-1.0.0.xsd http://cxf.apache.org/blueprint/jaxws http://cxf.apache.org/schemas/blueprint/jaxws.xsd http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd">

	<bean id="orderMergingJpaDao"
		class="de.nierbeck.camel.exam.demo.entities.dao.jpa.CamelMessageStoreJpaDao">
		<jpa:context property="em" unitname="camelStore" />
		<tx:transaction method="*" value="Required" />
	</bean>

	<service id="orderMergingDao" ref="orderMergingJpaDao"
		interface="de.nierbeck.camel.exam.demo.entities.dao.CamelMessageStoreDao" />

</blueprint>

The Transactions are declared per method, in this example it’s * as for all method and the transaction is set to required. For more details on what is possible here take a look a the aries documentation, though if your used to spring you’ll find some similarities.

Route-Control

The route-control module is the camel sample module containing routes and tests. The package de.nierbeck.camel.exam.demo.control contains the interfaces and model classes of this project. The WebServiceOder is the Web-Service interface using CXF, it’s transfer class is the CamelMessageBean. The de.nierbeck.camel.exam.demo.control.inernal package contains the Route and a Process class. The de.nierbeck.camel.exam.demo.control.internal.converter contains a Converter class.

Bildschirmfoto 2013-08-15 um 22.55.36

The Route accepts Messages from the cxf Web-Service which uses the OSGi HttpService. Those Incoming messages are converted and send to an JMS queue. For Logging a wire-tap EIP is used which stores the data asynchronously to the database with the JPA component.

		from("cxf:bean:messageService").routeId(RouteID.WEB_SERVICE_ORDER)
		.log(LoggingLevel.DEBUG, "Incoming Request: ${body}")
		.setBody(simple("${body[0]}"))
		.setHeader("MessageId").mvel("exchangeId")
		.wireTap("direct:logMessage")
		.to("activemq:queue:"+ JmsDestinations.QUEUE_MESSAGE_STORE+"?disableReplyTo=true")
		.setExchangePattern(ExchangePattern.InOut)
		.process(new OutMessageProcessor());

		from("direct:logMessage").routeId(RouteID.LOG_ORDER)
			.process(new MessageLogConverter())
			.to("jpa:de.nierbeck.camel.exam.demo.entities.CamelMessage");

The camel-context.xml contains the blueprint configuration for Camel. It defines the cxf endpoint, references the DAO service, entity manager and transaction Manager, needed for the JPA component and starts the Camel Context. The jms-config.xml is used for configuring ActiveMQ as queue.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:camel="http://camel.apache.org/schema/blueprint" xmlns:jpa="http://aries.apache.org/xmlns/jpa/v1.0.0"
    xmlns:tx="http://aries.apache.org/xmlns/transactions/v1.0.0" xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0"
    xmlns:camelcxf="http://camel.apache.org/schema/blueprint/cxf"
    xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd http://camel.apache.org/schema/blueprint/camel-blueprint.xsd http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0/blueprint-cm-1.0.0.xsd http://cxf.apache.org/blueprint/jaxws http://cxf.apache.org/schemas/blueprint/jaxws.xsd http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd">

    <camelcxf:cxfEndpoint id="messageService" address="/camel-test/contol"
        serviceClass="de.nierbeck.camel.exam.demo.control.WebServiceOrder" wsdlURL="wsdl/WebServiceOrder.wsdl" >
        <camelcxf:properties>
            <entry key="dataFormat" value="POJO" />
        </camelcxf:properties>
    </camelcxf:cxfEndpoint>

    <reference id="entityManagerFactory" interface="javax.persistence.EntityManagerFactory"/>
    <reference id="jtaTransactionManager" interface="javax.transaction.TransactionManager"/>

    <bean id="transactionManager" class="org.springframework.transaction.jta.JtaTransactionManager">
        <argument ref="jtaTransactionManager"/>
    </bean>

    <bean id="jpa" class="org.apache.camel.component.jpa.JpaComponent">
        <property name="entityManagerFactory" ref="entityManagerFactory" />
        <property name="transactionManager" ref="transactionManager" />
    </bean>

    <bean id="orderWebService"
        class="de.nierbeck.camel.exam.demo.control.internal.OrderWebServiceRoute">
    </bean>

    <reference id="dao" interface="de.nierbeck.camel.exam.demo.entities.dao.CamelMessageStoreDao" />

    <camelContext trace="true" id="route-control" xmlns="http://camel.apache.org/schema/blueprint">
        <routeBuilder ref="orderWebService" />
    </camelContext>

</blueprint>

The fun part starts with testing this setup in the desired Karaf container with Pax-Exam. This test uses Pax-Exam 3.2.0 the latest released version, Karaf 2.2.11, Camel 2.10.4 and ActiveMQ 5.7. One would wonder why the rather “old” version of Karaf. This is due to the fact that ActiveMQ 5.8.0 did mess up with OSGi dependencies and therefore isn’t really usable right now. (I might be wrong but I wasn’t able to get it running)
For running the Pax-exam test it’s sufficient to add the following annotation to the class so it’s discovered as an Pax-Exam Test.

@RunWith(PaxExam.class)

As usual you define the configuration of the Pax-Exam test through the configuration method identified by the @Configuration annotation. This configuration needs some deeper insights, at first the Karaf test container is defined and a directory for unpacking. Now the required features are defined, if this would be a real integration test the test would use the feature file from the feature module. Now some extra bundles are declared for this test, for example the entities module and the h2 bundle for the database.

The most important part of this configuration is the dynamic building of three additional bundles. The first one containing the required resources for a data-source bundle, the second containing the required resources for building a activemq broker bundle and last but not least a bundle containing all the classes and resources of the current module. This is needed due to the fact that the unit-test class loader doesn’t know of the classes built in “main” and therefore need to be added to the current test.

	@Configuration
	public static Option[] configure() throws Exception {
		return new Option[] {
				karafDistributionConfiguration()
						.frameworkUrl(
								maven().groupId("org.apache.karaf").artifactId("apache-karaf").type("zip")
										.versionAsInProject()).useDeployFolder(false).karafVersion("2.2.11")
						.unpackDirectory(new File("target/paxexam/unpack/")),

				logLevel(LogLevel.WARN),

				features(
						maven().groupId("org.apache.karaf.assemblies.features").artifactId("standard").type("xml")
								.classifier("features").versionAsInProject(), "http-whiteboard"),
				...
						maven().groupId("org.apache.camel.karaf").artifactId("apache-camel").type("xml")
								.classifier("features").versionAsInProject(), "camel-blueprint", "camel-jms",
						"camel-jpa", "camel-mvel", "camel-jdbc", "camel-cxf", "camel-test"),

				KarafDistributionOption.editConfigurationFilePut("etc/org.ops4j.pax.url.mvn.cfg",
						"org.ops4j.pax.url.mvn.proxySupport", "true"),
				keepRuntimeFolder(),

				mavenBundle().groupId("com.h2database").artifactId("h2").version("1.3.167"),
				mavenBundle().groupId("de.nierbeck.camel.exam.demo").artifactId("entities").versionAsInProject(),
				mavenBundle().groupId("org.ops4j.pax.tipi").artifactId("org.ops4j.pax.tipi.hamcrest.core")
						.versionAsInProject(),
				streamBundle(
						bundle().add("OSGI-INF/blueprint/datasource.xml",
								new File("src/sample/resources/datasource.xml").toURL())
								.set(Constants.BUNDLE_SYMBOLICNAME, "de.nierbeck.camel.exam.demo.datasource")
								.set(Constants.DYNAMICIMPORT_PACKAGE, "*").build()).start(),
				streamBundle(
						bundle().add("OSGI-INF/blueprint/mqbroker.xml",
								new File("src/sample/resources/mqbroker-test.xml").toURL())
								.set(Constants.BUNDLE_SYMBOLICNAME, "de.nierbeck.camel.exam.demo.broker")
								.set(Constants.DYNAMICIMPORT_PACKAGE, "*").build()).start(),
				streamBundle(
						bundle().add(JmsDestinations.class)
								.add(WebServiceOrder.class)
								.add(CamelMessageBean.class)
								.add(RouteID.class)
								.add(OrderWebServiceRoute.class)
								.add(OutMessageProcessor.class)
								.add(MessageLogConverter.class)
								.add("OSGI-INF/blueprint/camel-main-context.xml",
										new File("src/main/resources/OSGI-INF/blueprint/camel-context.xml")
												.toURL())
								.add("OSGI-INF/blueprint/jms-context.xml",
										new File("src/main/resources/OSGI-INF/blueprint/jms-config.xml").toURL())
								.add("wsdl/WebServiceOrder.wsdl",
										new File("target/generated/wsdl/WebServiceOrder.wsdl").toURL())
								.set(Constants.BUNDLE_SYMBOLICNAME, "de.nierbeck.camel.exam.demo.route-control")
								.set(Constants.DYNAMICIMPORT_PACKAGE, "*")
								.set(Constants.EXPORT_PACKAGE, "wsdl, de.nierbeck.camel.exam.demo.control").build())
						.start() };
	}

Something else is need for a successful test with Pax-Exam inside the same module (instead of a integration test of the complete module) You need to adapt the manifest of the test-bundle that is produced from the Test class. This is done by adding the @ProbeBuilder method and defining additional Manifest headers.

	@ProbeBuilder
	public TestProbeBuilder probeConfiguration(TestProbeBuilder probe) {
		// makes sure the generated Test-Bundle contains this import!
		probe.setHeader(Constants.BUNDLE_SYMBOLICNAME, "de.nierbeck.camel.exam.demo.route-control-test");
		probe.setHeader(Constants.DYNAMICIMPORT_PACKAGE, "de.nierbeck.camel.exam.demo.control,*,org.apache.felix.service.*;status=provisional");
		return probe;
	}

To reduce timing issues during startup it’s best to have a doPreSetup test, which tries to look for the camel context as a service of the current test bundle. Due to this waiting you make sure the Test container is fully functional and it’s one of the first test to make.

	@Override
	protected void doPreSetup() throws Exception {
		controlContext = getOsgiService(CamelContext.class, "(camel.context.name=route-control)", 10000);
		assertNotNull(controlContext);

		testContext = getOsgiService(CamelContext.class, "(camel.context.name=route-test)", 10000);
		assertNotNull(testContext);

		for (CamelMessage orderMerging : orderMergingDao.findAll()) {
			orderMergingDao.makeTransient(orderMerging);
		}

	}

The test class also contains some pretty handy methods for either waiting on a service or, sending shell commands to the karaf shell running in the container. With those commands you’re able to send some “debugging” commands to your unit test that usually is helpful in finding misconfigurations inside the test that sometimes are hard to find.

In the src/sample/resources you’ll find some additional xml files that might be helpful. It’s a xml file that is used to build a datasource, this could be deployed in the Karaf or used by the test. Also there is a active-mq broker configuration file that also could be used directly in Karaf. Both files do also use property placeholders for configuring file locations for either the location of the H2 database file or the storage directory for ActiveMQ.

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0"
    xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
           http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd">

    <ext:property-placeholder />

    <bean id="h2DataSource" class="org.h2.jdbcx.JdbcDataSource">
        <property name="URL" value="jdbc:h2:${karaf.data}/database/h2Test" />
        <property name="user" value="sa" />
        <property name="password" value="" />
    </bean>

    <service id="dataSourceService" interface="javax.sql.DataSource" ref="h2DataSource">
        <service-properties>
            <entry key="osgi.jndi.service.name" value="jdbc/route-test-ds" />
        </service-properties>
    </service>

</blueprint>

 

A developers scrapbook