Dan Allen on Arquillian Testing Framework
Arquillian is an integration and functional testing platform that can be used for Java middleware testing. With the main goal of making integration (and functional) tests as simple to write as unit tests, it brings the tests to the runtime environment, freeing developers from managing the runtime from within the test.
Arquillian supports integration with Java EE containers like JBoss AS and GlassFish and servlet containers like Tomcat and Jetty, and supports running tests in cloud services. The container support allows developers to target a variety of technology platforms, including Java EE 5 and 6, servlet environments, OSGi, Embedded EJB and standalone CDI.
Arquillian team announced the release of 1.0.0.Final version last month. They are also working on a point release that will be out shortly.
Some of the features in the new release include:
- Orchestration of multiple deployments across multiple containers and domain controllers in a single test
- A new configuration schema that supports multiple configurations per container
- Expression Language (EL) like evaluation in properties and configuration overrides via Java properties
- Explicit ordering of test methods
- Fine-grained control over the container lifecycle
InfoQ caught up with Arquillian spokesperson Dan Allen about the testing framework’s features, how its approach differs from other testing strategies and what new features will be available in the future releases.
InfoQ: What types of testing (unit, integration, functional, performance, security etc) does Arquillian help with?
Dan Allen: Arquillian picks up where unit tests leave off and provides a platform to cover the remainder of the test spectrum. As soon as you begin integrating components inside a runtime, whether it be embedded or standalone, you need a service to manage that runtime. That's a core concern of Arquillian. Arquillian puts your tests inside the runtime and then delegates to extensions that help integrate components, invoke services, navigate web pages, measure performance characteristics, report code coverage and pass tests through the cloud, to name a few. Arquillian is the engine of test automation.
What Arquillian exposes is how small of a part unit tests play in the grand scheme of testing and how inadequate JUnit and TestNG are alone. You end up having to extend one base class for one type of testing and another base class for another type of testing. Arquillian aligns disparate testing services (e.g., Selenium, DBUnit, JaCoCo) into a single pipeline.
If you're using a programming model like CDI (or Spring), you may even decide to use Arquillian for testing units inside an embedded service container. This blurs the line between unit and integration tests, but it's good because you focus on the goal of the test and not how it's classified.
InfoQ: What are the sub-projects of Arquillian framework?
Dan: Arquillian doesn't have sub-projects, per se. Instead, it's organized as a network graph of modules that plug into the core platform. This arrangement allows integrations to come from both inside and outside of Arquillian proper. In fact, we prefer if projects host their own Arquillian extension as it ensures tighter integration and remains in sync with the release of that project.
There are (currently) 5 types of modules:
- container adapter
- test enricher
- test runner
Arquillian also integrates with the three ShrinkWrap modules: archives, descriptors and resolvers.
Container adapters are the most prevalent type of module in Arquillian since we aim to support every major Java container (i.e., server) and eventually non-Java ones too. The project currently host adapters for 12 vendor containers, including JBoss AS, GlassFish, Apache Tomcat, Jetty, WebLogic Server, Websphere AS, and 2 cloud providers, OpenShift and CloudBees. The adapters provide up to three management styles (embedded, managed and remote) each and cover several major release lines. It's a lot of adapters :)
The other major module is Arquillian Drone and its companion, Arquillian Graphene. These extensions take away all the hassle of setting up and managing Selenium, WebDriver and other browser drivers. All you have to do is inject the browser API into your test and send commands to it. By bringing together application deployment and browser control, you can quite literally test end-to-end. OpenShift just announced an integration with SauceLabs, which enables you to use Arquillian to run cross-platform browser tests from Jenkins running in the cloud.
A similar extension to Drone is the Android extension, which installs an application on the device and then controls the Android UI rather than a browser.
Two emerging modules we're watching closely are the Arquillian Persistence Extension, which provides declarative control and add-ons for DBUnit, and the Arquillian Spring Extension, which allows Spring developers to replace the Spring test framework with Arquillian and gain access to all the Arquillian extensions, in particular Drone. The Arquillian Spring Extension is being developed as a Google Summer of Code project.
I'll cap off this list by mentioning the module that provides an entry point into the platform, the Arquillian plugin for JBoss Forge. Simply put, there's no quicker way to get Arquillian setup in your project. One command and you're ready to go.
InfoQ: Have testing practices like BDD and TDD influenced the design of Arquillian framework?
Dan: I'd say that testing practices like BDD, TDD and ATDD have influenced who and what Arquillian is designed for.
We want developers to be able to test early, test often and test for real. As a developer, you can write a battery of unit tests or integration tests that uses an embedded or mock runtime, only to find out later you were violating a fundamental assumption about the runtime (classloading behavior, visibility, file system schema) that invalidates everything you worked on. You've just got to test the real thing. That's been scary up until now because it meant packaging your project and deploying the whole thing. By leveraging ShrinkWrap's micro-deployments, you can create very targeted integration tests that serve as check points along the way. No more face-palm moments mid-project.
I often say that Arquillian is a learning environment. You can test your assumptions or understanding of a framework in a "true" runtime very early on in the development process. You'd be amazed how quickly you learn things this way.
Arquillian is also supporting these testing practices by pursuing integrations with BDD and ATDD frameworks such as Spock, JBehave and Thucydides. This will be a major area of growth in the coming year.
InfoQ: Since Arquillian provides support for testing the applications inside the container, is there a risk that this approach will lead to developers not paying attention to design practices like loose coupling between classes, isolation, and modularity?
Dan: That's a very good question. While it's true that Arquillian puts your test code in the container where you can conveniently ignore the design practices you mentioned, it's balanced by ShrinkWrap. When writing each test, the developer has to consciously decide what to put in the test archive, which becomes the microcosm of the test (and hence the scope of the classpath visibility). While some developers find this tedious at first, they soon recognize that it provides a unparalleled level of isolation to closely study the dependencies between classes and libraries.
The developer's desire to want to run the tests in an embedded service container (e.g., Weld Embedded) encourages them to avoid getting too coupled with container resources except at the boundary layer. And while they may use the embedded container during development, they get the peace of mind that the tests run in a "true" container in continuous integration.
In summary, Arquillian and ShrinkWrap pull from opposite ends of the rope to maintain the balance between meaningful feedback and good design.
InfoQ: What are the limitations of Arquillian test framework?
Dan: Classpath management and lack of Java modularity. Developers often encounter these problem when writing tests for code that runs inside a container. Java tools (build tools, IDEs) and runtimes (application servers) are often very careless about the classpath or have no other choice. As a result, the developer starts running into very bizarre problems at runtime, which they attribute to Arquillian or ShrinkWrap. But it's really the absence of Java modularity to blame. This limitation is especially prevalent when using embedded containers. I wrote about the problem recently on the Arquillian blog. Fortunately, the workaround is pretty straightforward. Use a managed container that runs in a separate VM.
Another limitation is one of usability. It's still a bit awkward for developers to switch between different container adapters, partly because it requires using classpath profiles and partly because container configuration selection is not intelligent enough yet. For now, these are just the ropes of Arquillian that developers have to learn. If they read the Arquillian guides, they'll start off on the right track and not likely to get stuck.
We've put a lot of effort into writing guides to help developers get started. What we've done so far is very useful. But we are still short on documentation, so that's an area we need to continue to focus to help developers avoid getting stuck.
InfoQ: What is the future roadmap in terms of new features and enhancements?
Dan: What isn't? Arquillian has a culture of innovation and wild imagination. Just to give you an appreciation for how much we think about the future, check out the dozen project ideas the team proposed for the Google Summer of Code 2012 program. That list paints a pretty good picture of where we are headed. I'll list a few key items here:
- Spring beans and MVC testing
- Automatic visual comparison of browser screenshots from a test run
- More powerful persistence testing
- Hot redeploy of Java classes and resources using JRebel
- Easier control over deployment assembly by enhancing ShrinkWrap Descriptors and Resolvers
- Support for non-Java servers like Apache HTTPD
The interested readers can check out the guides designed exclusively to take them from an empty workspace to the first green bar with Arquillian.
About the Interviewee
Dan Allen is an open source advocate, community catalyst, author and speaker. He's currently pursuing these interests as a Principal Software Engineer at Red Hat. In that role, he serves as a JBoss Community liaison, contributes to several JBoss Community projects, including Arquillian, ShrinkWrap, Seam 3 / DeltaSpike and JBoss Forge, and participates in the JCP on behalf of Red Hat. Dan is the author Seam in Action (Manning, 2008), writes for IBM developerWorks, NFJS magazine and JAXenter and is an internationally recognized speaker. He's presented at major software conference series including JavaOne, Devoxx, NFJS, JAX and Jazoon. After a long conference day, you'll likely find Dan enjoying tech talk with fellow community members over a Belgian Trappist beer.
Randy Shoup Jul 03, 2015