BT

Mission Control and Flight Recorder on HotSpot JVM

by Fabian Lange on Oct 10, 2013 |

Since the Java 7 Update 40 release, Mission Control and Flight Recorder are shipped with the JDK as InfoQ reported on its release. Mission Control is the starting place for monitoring, management and troubleshooting, while Flight Recorder is the facility to collect and evaluate profiling data. Both tools have been available for JRockit and are now finally ported to HotSpot after being hinted at by Mark Reinhold in a webcast in 2010 and already announced by Mission Control Team Lead Marcus Hirt at JavaOne 2011. In multiple sessions during this year's JavaOne, Marcus Hirt and Staffan Larsen gave an introduction into how they work and can be utilized. The version number for Mission Control is 5.2.0, but as it is actually the first release for HotSpot, Marcus Hirt prefers to call it an 1.0.0 release.

Mission Control

Mission Control provides largely the same functionality as JVisual VM. Both tools allow connecting to local or remote Java processes to collect JMX data. Mission Control supports automatic discovery of running remote JVMs via the Java Discovery Protocol. To use it the JVM needs to be started with -Dcom.sun.management.jmxremote.autodiscovery=true -Dcom.sun.management.jdp.name=JVM_Name

Similarly to JVisual VM, Mission Control has a plugin mechanism, allowing for customization. But unlike VisualVM, Mission Control can also be used to create new views on the data collected. Two experimental plugins available today are JOverflow Heap Analyzer for finding inefficient use of Collections and DTrace Recorder for correlating DTrace profiles. Mission Control has a JMX browser as part of its core features and offers slightly more powerful functionality across the board. For example, the thread monitoring can provide per thread allocation information and on the fly stack traces. Because Mission Control is based on the Eclipse Platform, it is not only available as standalone tool within the JDK, but also as Eclipse plugin which can be obtained on the Oracle Mission Control Update Site.

Flight Recorder

Tools wanting to collect debug and especially performance data out of a JVM needed to implement the JVMPI/JVMTI interfaces. While most profilers work fine in development, getting them running with very low overhead in production has been difficult.

Flight Recorder brings functionality like CPU time or object allocation profiling with minimal overhead by implementing its own event based monitoring interface directly into the JVM. This new interface for example allows taking samples of threads without them being in safepoints, reducing overhead and skew of the measurements. There are only a few events that use bytecode instrumentation and thus have impact on the running code. Most of the capturing technology is new and not available to third parties. Flight Recorder is recording the data locally in the JVM, but off-heap, so it is not affecting the memory characteristics or garbage collection. When it is configured to persist the data, it is periodically dumped to a file.

The collected data consists mainly of 4 types of events: "instant", which are recorded when they happen, "requestable", which are polled, "duration", which is representing a metric of a timespan and "timed", which is same as "duration" but applies thresholds to filter data. There are two predefined configurations "continuous", which is intended to always run, and "profiling", which collects more data and is intended for short term analysis. The overhead however is always extremely low unless an event explicitly states otherwise.

Besides events generated by the JVM, frameworks and application servers are encouraged to provide own events. The interface is unsupported right now, but Weblogic and Glassfish already ship with events, which basically de-facto finalizes the interface. Marcus Hirt provides instructions on how to use the API on his blog post about Using the (Very Unsupported) Java Flight Recorder API The basic steps are to extend the appropriate Event class, annotate it to indicate values and invoke it from the event generating code. The custom events are no different from other events and can be used and dashboard creation and charted alongside other events. More details on the highlights of this release can be found in another blog by Marcus Hirt.

Licensing

To use Flight Recorder, the server VM needs to be started with: -XX:+UnlockCommercialFeatures -XX:+FlightRecorder This already indicates that these new capabilities require a license. Technically however there isn't a check, and a license file doesn't have to exist. Rather the license is a contractual agreement, which requires a Java SE Advanced or Java SE Suite, which are priced according to the Oracle Price List for 5k / 15k USD per processor for production systems. Test or development systems do not need a license, but still require the UnlockCommercialFeatures flag. It is unclear whether Mission Control itself requires a license. The documentation suggests that it can be freely used also on OpenJDK VMs, as long as Mission Control is not run on a production system which requires the per processor license. Hopefully Oracle can clarify the situation.

The Future

Now that Mission Control and Flight Recorder are in HotSpot, development focus is on adding plugins, like the Memory Leak Detector, as well as building heuristics into the tools, so they can automatically point out performance problems. There have been discussions on the future of JConsole or VisualVM, which both have overlap with Mission Control. They are likely to stay, as they are part of OpenJDK, while Mission Control is not. If Oracle follows the path which has been taken when VisualVM could have retired JConsole, it can be assumed that both JConsole and JVisual VM are going to stay and development will focus on Mission Control. Jack Shirazi speculates in his Java Performance Tuning newsletter that JConsole could be retired, while Kirk Pepperdine commented, that he believes that Mission Control will move to the Netbeans platform, similar to VisualVM. Marcus Hirt told InfoQ that there are no such plans.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

This is hilarious by Augusto Rodriguez

"Oracle Price List for 5k / 15k USD per processor for production systems."

That's USD 5k for an entry level server with 2 physical CPUs (standard Intel and AMD cores are charged at 50% of the list price).

It's way, waaay cheaper to get a 3rd party monitoring solution, that can not only monitor but generate alerts, trends, historic data, etc. This might be a benefit for a large organization which already has a support contract with Oracle, otherwise it's irrelevant.

I don't want to endorse any company, but we do use a enterprise APM solution which costs a fraction of what oracle charges and provides full visibility on a JVM. We've even used this 3rd party product to find performance bottleneck in other java products(eg. terracotta)

Re: This is hilarious by Ant hony

It's easy to make things hilarious by ripping them out of context: companies pay for Oracle Java SE Advanced/Suite, which a.o. includes 24x7 support, the JRockit VM with Java Flight Recorder, etc. Now they are able to use Java Flight Recorder with the HotSpot VM as well, provided as a free add-on to their Java SE Advanced/Suite license.

Moreover, "Test or development systems do not need a license, but still require the UnlockCommercialFeatures flag.", so developers can use it for free in their everyday work.

Eclipse Kepler (4.3) by Mark Sheinbaum

I failed to install the Eclipse plugin from download.oracle.com/technology/products/mission...
Is 4.3 supported? The error that I'm seeing is "Provisioning Exception" with the details below:
eclipse.buildId=4.3.0.I20130605-2000
java.version=1.7.0_25
java.vendor=Oracle Corporation
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_US
Framework arguments: -product org.eclipse.epp.package.standard.product -keyring /Users/msheinbaum/.eclipse_keyring -showlocation
Command-line arguments: -os macosx -ws cocoa -arch x86_64 -product org.eclipse.epp.package.standard.product -keyring /Users/msheinbaum/.eclipse_keyring -showlocation

Error
Tue Oct 15 12:13:24 PDT 2013
Unable to read repository at download.oracle.com/technology/products/mission....

org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at download.oracle.com/technology/products/mission....
at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache(CacheManager.java:192)
at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.getLocalFile(SimpleMetadataRepositoryFactory.java:66)
at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.load(SimpleMetadataRepositoryFactory.java:88)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:57)
at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:768)
at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:668)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:96)
at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:92)
at org.eclipse.equinox.p2.ui.ProvisioningUI.loadMetadataRepository(ProvisioningUI.java:439)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getMetadataRepository(MetadataRepositoryElement.java:120)
at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.fetchChildren(MetadataRepositoryElement.java:70)
at org.eclipse.equinox.internal.p2.ui.model.RemoteQueriedElement.fetchDeferredChildren(RemoteQueriedElement.java:34)
at org.eclipse.ui.progress.DeferredTreeContentManager$1.run(DeferredTreeContentManager.java:235)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53)
Caused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <download.oracle.com> != <a248.e.akamai.net> OR <*.akamaihd.net> OR <*.akamaihd-staging.net> OR <a248.e.akamai.net>
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:228)
at org.apache.http.conn.ssl.BrowserCompatHostnameVerifier.verify(BrowserCompatHostnameVerifier.java:54)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:149)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:130)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:397)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:263)
at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69)
... 1 more

Re: Eclipse Kepler (4.3) by Mark Sheinbaum

never mind. no need to use secure http.

Re: This is hilarious by Adib Saikali

Given that Mission control extracts data from deep within the virtual machine through oracle only additions to the JVM perhaps they can get at data that other techniques / tools won't be able to get at.

Once this data is exposed through a java api maybe APM vendors can offer extra views based on data from the flight recorder / mission control api. Oracle could then licence these api's at a cheaper rate to the existing APM vendors, so that they can call these apis and include the cost of the business java virtual machine within the cost of the APM or as an addon.

Existing APM customers could be upsold by their APM vendor at price that is a relatively small increase over what the customer pays to get the extra data from flight recorder, say 5% extra over current vendor price.

Since oracle would not have had to spend the money to sell the client and service the contract ... etc the lower rate could be profitable enough relative to selling directly that they would offer it to the APM vendors.

Since the APM vendors can't add to the JVM the native code that oracle adds and still call the final result oracle java, oracle might just have an excellent distribution channel to collect money from java users who are already paying for an APM solution.

Eyes wide shut by William Louth

I am sorry but this view is flawed and lacking of knowledge of what other tools provide far beyond what is being offered by mission control. Lets take "flight recording". The Oracle tool sees this as merely collecting some high level metrics & thread state samples every second or so and logging them....it calls them events when in fact most are are effectively measurement samples.

Now flight recording done right can use the event stream to recreate an offline or online simulation of one or more JVMs. This is light years away from anything the Oracle team could possibly hope to achieve and yet we did not need any special hooks beyond the bytecode instrumentation API already standardizes. The real innovation is not in instrumentation it is the layers above related to measurement, analytics and control. This work is outside the scope of the JVM.

Some further reading that tears back the curtain on what it means to monitor and manage applications.

www.jinspired.com/site/mirroring-mindreading-an...

www.jinspired.com/site/changing-space-and-time-...

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

6 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT