As part of Oracle's ongoing project to merge the HotSpot and JRockit codebases, Oracle has announced that they will remove PermGen from the Java 8 version of the HotSpot JVM. Many people however have interpreted this announcement in a way that all their PermGen errors will disappear. Because the effects of the removal of PermGen can now be checked with Java 8 Early Access builds, it is time to find out if all PermGen problems have been resolved.
What is the PermGen?
Jon Masamitsu, JVM developer at Oracle, explained 2006 in his blog the purpose of the permanent generation: The permanent generation contains information about classes, such as bytecode, names and JIT information. It is stored in a separate space, because it is mostly static and garbage collection can be much more optimized by separating it.
The Problems with PermGen
Many developers find their system showing "java.lang.OutOfMemoryError: PermGen space" at some time. When it does, it is very often caused by a memory leak related to classloaders, and creation of new classloaders, which generally happens during hot deployments of code. This is why it is more frequent on development machines, than in production. When it occurs in production, developers can take the generated heap dump and use a tool like Eclipse Memory Analyzer Toolkit to look for classloaders that should be gone but are not. The permanent collection is garbage collected, unless specific configuration prevents it. However, in the case of leaks, there is just nothing to collect. In production the most common "problem" is that the default value 64MB is way too low. Setting it to 256MB is the usual band aid to resolve it.
What Java 8 changes
In his mail on the HotSpot development list, Jon now explains what Java 8 will change: with Java 8, there is no PermGen anymore. Some parts of it, like the interned Strings, have been moved to regular heap already in Java 7. In 8 the remaining structures will be moved to a native memory region called "Metaspace", which will grow automatically by default and will be garbage collected. There will be two flags: MetaspaceSize and MaxMetaspaceSize.
Jon Masamitsu explains the design goal behind this on request by InfoQ:
A goal for removing perm gen was so that users do not have to think about correctly sizing it.
Set MetaspaceSize to a value larger than the default, if you know that your applications needs more space for class data. Setting it to a larger size will avoid some number of GC's at startup. It is not necessary and I do not particularly recommend it unless you want to avoid as many GC's as possible.
Set MaxMetaspaceSize if you want to limit the space for class data. You might want to do this if you suspect you are leaking classloaders and want the application to stop before it uses up too much native memory. Another case might be where you have multiple applications running on a server and you want to limit how much space each uses for class data.
So MetaspaceSize falls in the category of potentially affecting the first few Garbage Collections and should not be important in most cases. In that way it reflects the purpose of the old PermSize; flag.
When MaxMetaspaceSize is set, the space can still run out of memory, which will cause a "java.lang.OutOfMemoryError: Metadata space". Due to the fact that classloader leaks exist, and consuming unlimited native memory is something Java typically does not do, it seems sensible to set a similar limit to what was set with MaxPermSize. Similar to the PermGen, verbose GC logging will print the current consumption of Metaspace. Using the command line flags PermSize or MaxPermSize will result in a warning, instructing the user to switch to the Metaspace flags.
Conclusion
Because the concept of Metaspace and perm is mostly the same, an administrator performing a Java 7 to Java 8 upgrade can change the flags simply by running sed 'e/Perm/Metaspace/g'.
Overall, this change looks underwhelming. For most cases, it is just a name change. Making the Metaspace unbounded by default prevents choosing a too small default, but requires setting a maximum to ensure system stability. Luckily, we can reuse the configuration of PermSize and MaxPermSize almost everybody already uses and just rename the flag. Unfortunately the move from managed java heap to native memory means that there is a lot less valuable troubleshooting information in a heap dump, as also Kirk Pepperdine raised as concern.
In the end, classloader leaks can still occur as before.
Community comments
What magic solution did you expect?
by Ronald Miura /
Re: What magic solution did you expect?
by Cameron Purdy /
Re: What magic solution did you expect?
by peter lin /
Re: What magic solution did you expect?
by Ronald Miura /
Re: What magic solution did you expect?
by peter lin /
Re: What magic solution did you expect?
by Kirk Pepperdine /
Re: What magic solution did you expect?
by Marc Logemann /
Magic and Leaks
by Fabian Lange /
Re: Magic and Leaks
by Ronald Miura /
Re: Magic and Leaks
by Cameron Purdy /
Finding the classloader leaks
by Ivo Mägi /
What magic solution did you expect?
by Ronald Miura /
Your message is awaiting moderation. Thank you for participating in the discussion.
If you load classes into a bounded section, you'll eventually get the OutOfMemoryError. If you don't, you'll get a 'unstable system'.
Underwhelming... what did you expect, the JVM correcting your memory leak bugs automatically?
Re: What magic solution did you expect?
by Cameron Purdy /
Your message is awaiting moderation. Thank you for participating in the discussion.
It's a bit better than that. Previously, the JVM chewed up that space whether or not it needed it ... so if you sized it too small, the JVM died, and if you sized it too large, the JVM wasted memory. Now you should theoretically just ignore the whole topic, because it will self-tune itself at runtime to the "right size".
Peace,
Cameron Purdy | Oracle
Re: What magic solution did you expect?
by peter lin /
Your message is awaiting moderation. Thank you for participating in the discussion.
I was going to reply to mr miura, but you already said it much more eloquently than I could have.
Having used and tested JRockit in the past, having the JVM manage the size of permGen makes life easier for developers. I've lost count of how many times tomcat users asked that question on the mailing list.
Re: What magic solution did you expect?
by Ronald Miura /
Your message is awaiting moderation. Thank you for participating in the discussion.
You guys got the wrong impression (that is, I failed to transmit the message). I think this change is a huge step forward.
What I disliked is the article conclusion, which undermined the move as 'underwhelming'. It's only 'underwhelming' if you are expecting magic solutions to unsolvable problems (automatically fixing your bugs).
Re: What magic solution did you expect?
by peter lin /
Your message is awaiting moderation. Thank you for participating in the discussion.
agree, there's no magic solution to fixing bad code and bugs.
Magic and Leaks
by Fabian Lange /
Your message is awaiting moderation. Thank you for participating in the discussion.
Thanks for your comments.
Let us consider two applications:
a) a well written one, which does in fact load and unload classloaders (like using JBoss rules).
b) a badly written one, which in fact does not do much classloader magic at all (but might leak one during deployment).
What I see in production are mostly applications of type "b". Peter confirms that, when he says people on the Tomcat mailing list complain.
You could argue that they complain about the 64MB default limit, but in fact more often than not they HAVE a leaking classloader during redeployment.
And the "PermGen removal" will NOT help them. Because now they will be filling up native memory.
And, given the fact that you do have classloader leaks, the change will make it worse to debug them.
And yes the last line might be provoking, but as your comments show, the general understanding of the PermGen removal was, that it improves the situation. And I respectfully disagree.
Re: Magic and Leaks
by Ronald Miura /
Your message is awaiting moderation. Thank you for participating in the discussion.
It does improve the situation, it just doesn't improve that particular situation.
Again, it won't fix your bugs. Or Tomcat's bugs. Or JBoss' bugs. Or Hibernate's bugs. It just makes an unnecessary parameter unnecessary, when you do things right.
The only things that could possibly solve this kind of memory leak (despite your incompetence in avoiding it) are:
- to change the semantics of the Java memory model and classloader architecture (and breaking everything under the sun with that);
- JREs to implement JSR-121 and app servers take advantage of it (thus, being able to 'kill' a whole section of the application);
- using a Tomcat per application, and restarting it on every deploy (which is what we do anyway).
No other JVM would solve this classloader leak problem, and if you find this change 'underwhelming', it's because you were the one expecting too much.
That said, Tomcat 7 made great improvements around this problem, detecting the most common issues, and fixing it, when possible, or warning you, when not. No JVM change required (although they probably can talk about changes that would make their life easier).
Re: Magic and Leaks
by Cameron Purdy /
Your message is awaiting moderation. Thank you for participating in the discussion.
Fabian -
Let's consider a third application:
c) A reasonably well written application that uses Spring, Hibernate and some other large libraries on a moderate-to-large application server (JBoss, WLS, etc.)
What I see in production are server+application+framework footprints that far exceed the defaults that PermGen has allocated by the JVM, and thus blow up when one runs the application. So one must tune those defaults, and likely tunes them too large or too small (since it is manual tuning).
What the elimination of PermGen does is to make the JVM work "out of the box", regardless of what the correct setting of PermGen should be. In other words, the JVM figures it out on the fly, with no JVM crash ("PermGen Out Of Space" exception), and with no manual twiddling of obscure knobs -- knobs that waste memory on one end and cause crashes on the other!
Sure, it's not magic. It's a very nice step forward, though :-)
Peace,
Cameron Purdy | Oracle
Working for Oracle, but the opinions expressed are my own and may not necessarily reflect the views of my employer. Some assembly required. Batteries not included. Do not try this at home.
Re: What magic solution did you expect?
by Kirk Pepperdine /
Your message is awaiting moderation. Thank you for participating in the discussion.
sorry to disagree with you Cam as I know you have to toe the party line but sweeping dirt under the carpet just moves it to under the carpet. I've always felt that there were better things to spend engineering effort. I say this because getting rid of Perm doesn't solve the underlying issues with classloading that you and IBM, and Azul all face. It's a process that is very susceptible to leaks but At least in the Oracle JVM the problems were contained in a useful way. By exposing this to Java and C heap you hide the problem, directly expose it to the garbage collector, and move the problem into a space where we have little if any tooling to identify and then diagnosis what's going on.
Regards,
Kirk
Re: What magic solution did you expect?
by Marc Logemann /
Your message is awaiting moderation. Thank you for participating in the discussion.
I think Cameron is right in saying that most perm issues are related to big libraries usage. we dont have issues with leaks in perm with business software. But we often forget to increase default perm on new machines and this annoying todo is now gone. So on this matter, its an improvement.
Marc
Finding the classloader leaks
by Ivo Mägi /
Your message is awaiting moderation. Thank you for participating in the discussion.
When you suspect a classloader leak, either in PermGen or Metaspace, give plumbr.eu a try - the tool will find the leak, give the exact location and solution guidelines to fix it.