Mission Control and Flight Recorder on HotSpot JVM
Since the Java 7 Update 40 release, Mission Control and Flight Recorder are shipped with the JDK as InfoQ reported on its release. Mission Control is the starting place for monitoring, management and troubleshooting, while Flight Recorder is the facility to collect and evaluate profiling data. Both tools have been available for JRockit and are now finally ported to HotSpot after being hinted at by Mark Reinhold in a webcast in 2010 and already announced by Mission Control Team Lead Marcus Hirt at JavaOne 2011. In multiple sessions during this year's JavaOne, Marcus Hirt and Staffan Larsen gave an introduction into how they work and can be utilized. The version number for Mission Control is 5.2.0, but as it is actually the first release for HotSpot, Marcus Hirt prefers to call it an 1.0.0 release.
Mission Control provides largely the same functionality as JVisual VM. Both tools allow connecting to local or remote Java processes to collect JMX data. Mission Control supports automatic discovery of running remote JVMs via the Java Discovery Protocol. To use it the JVM needs to be started with
Similarly to JVisual VM, Mission Control has a plugin mechanism, allowing for customization. But unlike VisualVM, Mission Control can also be used to create new views on the data collected. Two experimental plugins available today are JOverflow Heap Analyzer for finding inefficient use of Collections and DTrace Recorder for correlating DTrace profiles. Mission Control has a JMX browser as part of its core features and offers slightly more powerful functionality across the board. For example, the thread monitoring can provide per thread allocation information and on the fly stack traces. Because Mission Control is based on the Eclipse Platform, it is not only available as standalone tool within the JDK, but also as Eclipse plugin which can be obtained on the Oracle Mission Control Update Site.
Tools wanting to collect debug and especially performance data out of a JVM needed to implement the JVMPI/JVMTI interfaces. While most profilers work fine in development, getting them running with very low overhead in production has been difficult.
Flight Recorder brings functionality like CPU time or object allocation profiling with minimal overhead by implementing its own event based monitoring interface directly into the JVM. This new interface for example allows taking samples of threads without them being in safepoints, reducing overhead and skew of the measurements. There are only a few events that use bytecode instrumentation and thus have impact on the running code. Most of the capturing technology is new and not available to third parties. Flight Recorder is recording the data locally in the JVM, but off-heap, so it is not affecting the memory characteristics or garbage collection. When it is configured to persist the data, it is periodically dumped to a file.
The collected data consists mainly of 4 types of events: "instant", which are recorded when they happen, "requestable", which are polled, "duration", which is representing a metric of a timespan and "timed", which is same as "duration" but applies thresholds to filter data. There are two predefined configurations "continuous", which is intended to always run, and "profiling", which collects more data and is intended for short term analysis. The overhead however is always extremely low unless an event explicitly states otherwise.
Besides events generated by the JVM, frameworks and application servers are encouraged to provide own events. The interface is unsupported right now, but Weblogic and Glassfish already ship with events, which basically de-facto finalizes the interface. Marcus Hirt provides instructions on how to use the API on his blog post about Using the (Very Unsupported) Java Flight Recorder API The basic steps are to extend the appropriate Event class, annotate it to indicate values and invoke it from the event generating code. The custom events are no different from other events and can be used and dashboard creation and charted alongside other events. More details on the highlights of this release can be found in another blog by Marcus Hirt.
To use Flight Recorder, the server VM needs to be started with:
-XX:+UnlockCommercialFeatures -XX:+FlightRecorder This already indicates that these new capabilities require a license. Technically however there isn't a check, and a license file doesn't have to exist. Rather the license is a contractual agreement, which requires a Java SE Advanced or Java SE Suite, which are priced according to the Oracle Price List for 5k / 15k USD per processor for production systems. Test or development systems do not need a license, but still require the UnlockCommercialFeatures flag. It is unclear whether Mission Control itself requires a license. The documentation suggests that it can be freely used also on OpenJDK VMs, as long as Mission Control is not run on a production system which requires the per processor license. Hopefully Oracle can clarify the situation.
Now that Mission Control and Flight Recorder are in HotSpot, development focus is on adding plugins, like the Memory Leak Detector, as well as building heuristics into the tools, so they can automatically point out performance problems. There have been discussions on the future of JConsole or VisualVM, which both have overlap with Mission Control. They are likely to stay, as they are part of OpenJDK, while Mission Control is not. If Oracle follows the path which has been taken when VisualVM could have retired JConsole, it can be assumed that both JConsole and JVisual VM are going to stay and development will focus on Mission Control. Jack Shirazi speculates in his Java Performance Tuning newsletter that JConsole could be retired, while Kirk Pepperdine commented, that he believes that Mission Control will move to the Netbeans platform, similar to VisualVM. Marcus Hirt told InfoQ that there are no such plans.
This is hilarious
That's USD 5k for an entry level server with 2 physical CPUs (standard Intel and AMD cores are charged at 50% of the list price).
It's way, waaay cheaper to get a 3rd party monitoring solution, that can not only monitor but generate alerts, trends, historic data, etc. This might be a benefit for a large organization which already has a support contract with Oracle, otherwise it's irrelevant.
I don't want to endorse any company, but we do use a enterprise APM solution which costs a fraction of what oracle charges and provides full visibility on a JVM. We've even used this 3rd party product to find performance bottleneck in other java products(eg. terracotta)
Re: This is hilarious
Moreover, "Test or development systems do not need a license, but still require the UnlockCommercialFeatures flag.", so developers can use it for free in their everyday work.
Eclipse Kepler (4.3)
Is 4.3 supported? The error that I'm seeing is "Provisioning Exception" with the details below:
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_US
Framework arguments: -product org.eclipse.epp.package.standard.product -keyring /Users/msheinbaum/.eclipse_keyring -showlocation
Command-line arguments: -os macosx -ws cocoa -arch x86_64 -product org.eclipse.epp.package.standard.product -keyring /Users/msheinbaum/.eclipse_keyring -showlocation
Tue Oct 15 12:13:24 PDT 2013
Unable to read repository at download.oracle.com/technology/products/mission....
org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at download.oracle.com/technology/products/mission....
Caused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <download.oracle.com> != <a248.e.akamai.net> OR <*.akamaihd.net> OR <*.akamaihd-staging.net> OR <a248.e.akamai.net>
... 1 more
Re: This is hilarious
Once this data is exposed through a java api maybe APM vendors can offer extra views based on data from the flight recorder / mission control api. Oracle could then licence these api's at a cheaper rate to the existing APM vendors, so that they can call these apis and include the cost of the business java virtual machine within the cost of the APM or as an addon.
Existing APM customers could be upsold by their APM vendor at price that is a relatively small increase over what the customer pays to get the extra data from flight recorder, say 5% extra over current vendor price.
Since oracle would not have had to spend the money to sell the client and service the contract ... etc the lower rate could be profitable enough relative to selling directly that they would offer it to the APM vendors.
Since the APM vendors can't add to the JVM the native code that oracle adds and still call the final result oracle java, oracle might just have an excellent distribution channel to collect money from java users who are already paying for an APM solution.
Eyes wide shut
Now flight recording done right can use the event stream to recreate an offline or online simulation of one or more JVMs. This is light years away from anything the Oracle team could possibly hope to achieve and yet we did not need any special hooks beyond the bytecode instrumentation API already standardizes. The real innovation is not in instrumentation it is the layers above related to measurement, analytics and control. This work is outside the scope of the JVM.
Some further reading that tears back the curtain on what it means to monitor and manage applications.
Helen Walton Dec 17, 2014
Heather Hassebroek and Kent McDonald on Positive Politics, Organisational Change, Leadership Engagement and Sharing Experiences through Stories
Heather Hassebroek and Kent McDonald Dec 17, 2014
Monica Beckwith Dec 16, 2014