According to the 2020 Java InfoQ Trends Report, Quarkus has grown in popularity, moving quickly into the Early Adopters space. The framework that calls itself supersonic, subatomic Java, has added more features to facilitate the development and operation of software applications. In the 17 months since the release of Quarkus 1.0, its built-in development mode has added state preserving hot deploy and redeploy features and now targets to add yet another feature inspired by Ruby. Quarkus 2.0 will provide developers the ability to run tests in a continuous manner every time a class is modified. This feature promises to run smoothly behind the scenes, notifying the developer if new code will break any tests.
InfoQ spoke to Stuart Douglas, senior principal engineer at Red Hat, for a deeper understanding of this promising feature.
InfoQ: What was the motivation for implementing continuous testing? What benefits do you think it will bring to the development process?
Stuart Douglas: It actually came about because of a question on the mailing list from a developer who primarily develops in Ruby, but was trying out Java. He said he loved the Quarkus live reload in quarkus:dev mode, but was disappointed that running tests were so slow in comparison, and pointed out that scripting language developers often use continuous testing.
I must admit my initial reaction was just "that's just how it is in Java," but after I slept on it, I realised that we actually had all the building blocks we needed in Quarkus to do our own continuous testing; all we really needed was to write a JUnit test runner and wire everything together.
From a developer’s point of view, this gives you almost instant feedback on changes that you make. As soon as you save your code, the tests run and you can see if your changes were effective. We use a very targeted approach to running tests, so we only run tests that your change might affect, and the result is a super quick feedback cycle that is like nothing I have ever experienced with Java. I actually feel disappointed that I can’t use it when working on Quarkus itself.
From the beginning, developer experience has been one of our main focuses in Quarkus, and this takes it to the next level. Because this is integrated into the core of Quarkus, every user has access to it out-of-the-box. This feature also does not bloat the resulting Quarkus application as none of the testing code ends up in your final application. This means that we can create lightweight cloud-focused applications while still providing a great developer experience.
InfoQ: This new feature is currently part of Quarkus’s alpha release. When will Quarkus 2.0 be available? Do you anticipate any changes before the GA release?
Douglas: At this stage, we are looking at mid-June.
At the moment, we are trying to get as much feedback as possible from the community, and have already made some changes based on the feedback we have received. I don’t expect the core functionality to change much between now and the release, but the more feedback we receive, the more likely we will be able to iron out any issues. There is still some work to be done on how we present the results, both in the console and the dev UI, so this will likely be the most user-visible change between now and the final release.
InfoQ: Are there any known limitations of continuous testing?
Douglas: I guess the biggest limitation is that it needs to run all the tests on startup in order to know which tests to run when code changes. So if you have a huge test suite, this may be an issue. There are things we can do to mitigate this, such as caching this information between launches, so this will likely improve in future releases.
InfoQ: Are you happy with the outcome so far? Would you implement anything differently?
Douglas: I think it is working better than I hoped when I first started experimenting. There will no doubt be improvements as time goes on, but I am really happy with the core architecture.
InfoQ: Can you give us an overview on its implementation details? Is there anything that you are particularly proud of?
Douglas: The first challenge when testing is enabled is to actually resolve the test artifacts and build a ClassLoader that can load the classes. When you are running your app normally, things like JUnit are not present, so we need to manually resolve them. This is relatively complex and needs to be done for both Maven and Gradle. But luckily, we already had code that we use to resolve the build time part of our extensions.
Next, we need to discover the tests, because development mode already tracked the current workspace to deal with changes that just required adding some additional metadata to our existing workspace model. Using this metadata, we can discover the user’s test classes and load them into our new ClassLoader. When we load the tests and application classes, we actually perform some bytecode transformations that allow us to track when methods are invoked. This is how we know which tests touch their corresponding classes.
The next thing we needed was a way to run the tests, which is the only bit that is completely new. This involved writing a JUnit runner, which I had not looked at before, but the API is really quite nice so the implementation is straightforward. When we run the tests on startup, we use the transformation mentioned above to track which tests use the application classes.
The three items above are all you need to actually run the tests, but we also need a way to rerun tests upon code change. Development mode already has support for detecting changed files and automatically compiling them, so we just had to modify this to handle tests as well. When source files are changed, we automatically compile them and look at the class usage data we gathered from our first test run. From this, we can figure out every test that uses the modified classes, and automatically run them, while not running any tests that are not relevant.
Out of all of this, the test runner is the only completely new part. Everything else is just an enhancement of a feature we already had. The bit I am most proud of is the way it can trace exactly which tests are affected by code. Without this, you would not accurately know which tests to run after a change.
InfoQ: Is continuous testing automatically enabled when development mode is enabled?
Douglas: It’s automatically available, but you need to press a key to enable it. This can be changed via configuration, so you can configure Quarkus to always use continuous testing. Quarkus provides several key bindings to control continuous testing, so you can quickly pause/resume testing, rerun tests, etc.
We are also providing a quarkus:test command that will only run the tests and not launch development mode. This gives users more flexibility. For example, if you are just doing test-driven development, you may only care about running the tests.
Targeted for a GA release in mid-June, Quarkus 2.0 brings, among other features, the ability to continuously run tests while code is written. Similar to the continuous deploy functionality added in Quarkus 1.11, continuous testing provides developers with quick feedback for modified code, promising to quickly detect any issues and potentially fix them. Part of Quarkus’s development mode runs out-of-the box allowing changes to be made via configuration. Even though still in alpha, only the UI side of the reporting tool will most-likely be affected with the core functionality remaining the same. More details on continuous testing may be found in Douglas’s YouTube video. Developers are encouraged to experiment with this new feature before the GA release.