BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Presentations A Year with Java 11 in Production!

A Year with Java 11 in Production!

Bookmarks
50:11

Summary

Andrzej Grzesik talks about Revolut’s experience in running Java 11 in production for over a year. He talks about the doubts they had, some pain points and gains, as well as surprises that surprised them. He discusses tools, alternative JVM languages, and some 3rd party products.

Bio

Andrzej Grzesik works at at Revolut, UK. He likes distributed systems in all shapes and form. While he has written in many languages, he favours the JVM. Since "most software problems are people problems", he stirs communities, organizes and speaks at conferences. He is a Java Champion.

About the conference

Software is changing the world. QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.

Transcript

Andrzej Grzesik (ags): My name is Andrzej Grzesik. I go by Ags. I'm an engineer at Revolut, which means I write code on a daily basis. I get frustrated when things don't work. Usually, they work. I'm very proud to be a Java champion. Very proud to be old enough to have gone to Java 1, in time to become a Java 1 Rockstar. I also happen to run a Polish Java User Group, a conference called GeeCON, two Meetups. Geographically, I'm from Poland.

About Revolut

Which version of Java are people on? Most of you are on the LTS editions. Congratulations to the brave people who are on the more modern versions. We are looking at doing something similar later. What we are going to talk about is how we went from Java upgrade in the context of a particular company, because the context is always very important. What you probably know about Revolut is that it's a financial company. We have a product that's a credit card or a card, and an accompanying mobile application. The important facts from a software engineer's point of view is that the company is 4 years and a bit old. It has over 10 million users, which means it is growing like crazy. The usage on the systems is growing like crazy. The number of developers keeps on increasing. All the problems that we can avoid having, if there is anything that gives us that, we're very happy. This allows people to focus not on production issues, but instead on building and improving the codebase, and building out new features, and scaling globally and expanding the reach. For us, that's very important.

The codebase is very opinionated. For example, we don't use Spring. Some people might have heard about it. Why? Because debugging annotation driven problems is not fun. As people spend more time with computers, they prefer less creative solutions to some problems. I even describe my job that I keep doing the same year in, year out. I put boredom in places. I don't want certain things to be exciting. It might be counterintuitive, but there is a lot of fun in building stuff in general. I don't want surprises. Especially, if I've had an amazing evening somewhere, I don't want to receive a phone call, and then I have to look at squiggly lines and I have no idea what happens. If you've read some very esoteric looking languages, or if you've seen somebody who was trying to do a very smart version of Scala or Haskell code. I find Java much more readable in places. This is the attitude that we have in Revolut's codebase everywhere. We prefer to use code driven everything, not annotation driven. We prefer to have compile time checks rather than runtime explosions. Also, we don't have 1500 microservices. We have much less services. It's more controlled.

Deployment

In terms of deployment. What does deployment for a typical application look like? We have CoreOS, which is coming to an end-of-life. That will be a fun bit. We run applications in Docker. There is one application in a container, which means we don't share the instances between things. There is Fluentd, which represents also all the other daemons that would sit on any of the instances. We use a framework called Java Spark. Why? Because it's extremely simple. Very easy to debug. Very easy to reason about. If something goes wrong, that means you don't have to worry too much, because it's all in there. We also use jOOQ, which is a database interaction layer. We don't use Hibernate, because that gives us more control. People that I work with like to have this explicit control. Of course, that means that sometimes you have to write a bit more. If you want to build distributed systems, at least from my experience, you would have to get over your fear of SQL or whatever the actual language of the platform that you're running with.

Why Upgrade?

Why should we upgrade? If we're moving fast, if we stay to Java 8, probably, it might be a nuisance. It might be useless. Some corporate entities choose not to upgrade. Some corporate entities, as has been already demonstrated, are still before Java 8. That's of course their choice. Then comes the aspect of culture. The aspect of culture that is visible by those letters that are shining, GET IT DONE. People in a startup have a mentality of building things that they can get shipped, which means if there is something that can allow you to go faster, you probably want to do it. We are trying to move fast because we have to move fast, otherwise, the product will not happen. We cannot, obviously, break anything, because it's a financial company and we are put under a lot of scrutiny. Because if you're dealing with people's money, it would be very bad if we lost any of that. There is a lot of auditing. There is a lot of processes that we have to follow because we're dealing with money. That's obvious.

If something needs to be updated, if it's on the core path, because people see it, people are eager to make those changes because they remove friction. Why upgrade? To get it done. Also, so that we can move fast, because Java 11 allows us to move faster, worrying less. We also would have to upgrade anyway. It's a fact of life that software gets new, exciting features, towards the end-of-life cycle. Everything that Ben described, hopefully, is coming in your Java versions. This is something that everybody would like to benefit from. There are changes in JDK 11 that we have benefited from. If those features are out there, and you're on the previous JDK, it's a bit hard to use them. We want to upgrade.

There is this guy. This guy represents the almighty JSON, if you're dealing with applications that talk JSON to themselves, which most of us do, probably. There is one feature that happened between Java 8 and Java 9 that we also really wanted to benefit from, which is java.lang.String. It does not have a char array internally, but they have a byte array, which means you do get half the size of the maximum length of a string that you can store. The more important part is, you are storing bytes, which means it's half the size. Which means if your strings are ASCII, basically, you will use half the memory, which is an extremely important feature. It's a very nice feature. If you're doing JSON manipulation, which many of us are, because this is how the web communicates because the internet still runs mostly in debug mode, plain text, and so on, we benefited from this. If you need to do something more with strings, string deduplication with G1 even in JDK 8 might save you. There is a price to pay with string deduplication. This change you get for free just by switching from 8 to a newer JDK. JSON is now happy because obviously it takes less time and less memory.

Upgrade and Ops

We're a financial company, how could we do it? What groups did we have to reach out to? How did it actually happen? We do a lot of automation. A lot of automation in our context means a lot of YAML files. We automate a lot. The infrastructure, the images, how applications get built are expressed in code, which means anybody who is capable of using a text editor can change it up and can influence the process. Also, we are trying to keep this startup mentality in which we want to push and build things as quickly, with as little friction as possible, which means developers take care of many infrastructure tasks. Which also means if you build a piece of software, you get to maintain it. If you get to maintain it, you should probably have a say in what environment it is running on. Because, for example, the usage of your software has outgrown the resources that have been dedicated, and you know that you need to upgrade, why should you require a separate team to do something about it if you can do it yourself? If you have this self-service culture built into infrastructure in many of the places, of course, where it's applicable. I mentioned FinTech. There are some limitations which don't allow us to do absolutely everything that we wanted. If it's possible, we want to automate it and we want to empower people that are closest to the problem. There is a PR that will get a code review. TeamCity will hopefully run a build that will update whatever needs to be updated. That's how things end up in production. That's automated. That happens almost by itself. People don't need to do it all the way manually. Which would probably raise the question further, do we have any ops people? Of course, we do. They work on more specialized tasks. Tasks that developers, people who write Java code on a daily basis would probably take a bit more time to learn and figure out how to do correctly. That's all.

Java 11 Upgrade - From jdk8u212-b03 to 11

What did we do? We upgraded to Java 11, from JDK 8 update 212, and that particular build. Not super important. We did it to all the apps. All applications within Revolut doing Java in the backend run JDK 11 now. Of course, the team that runs Android, I have no idea what they do because I don't deal with Android. There might be some limitations there. Why 11? Because 9 and 10 were no longer maintained. I assume you know about the release cycle. Why not 12? 12 wasn't available when we got to it. The title of the talk is, "A Year in Production with JDK 11." This, in the case of Revolut, means Revolut has been running JDK 11 in production for over a year now. We are quite happy with what we have seen. It wasn't a journey without some hurdles. There were some thorns on the way as well.

Why not Oracle JDK?

Why not Oracle JDK? We went away from Oracle JDK. We are using Docker, so we bake images. With Oracle JDK, there is a license that is very complicated and lawyers like to read that language. I find it tiresome. English is not my first language. Lawyer English is like a special version of English. How I understand this license is I cannot build my own images. I cannot give people a copy of the Oracle JDK. People should download it from Oracle always, because Oracle should be the distributor. I can't. It also requires a license to run in production. Getting a license from Oracle, I don't know what the process is for 11. We decided if there is an alternative, we can use an alternative. We probably will use an alternative.

Java 11

If you want to go with Java 11, you have two choices. You have a choice of, let's do everything 11. We can compile on 11. We can run on 11. If you can, it's awesome. Absolutely. You can also choose to compile on 8 and run on 11. This is what I used to do before I joined Revolut, to some very good results. Because there are new features, there are new fixes to the runtime happening continuously. We are using this version. Java 11 is everywhere. It's what it gets compiled with. This is what it runs on. People don't use JDK 8, I don't think, mostly anywhere. Also, if you are not in that situation, it might be that you will have to use Java 11 without the full Java 11 tool chain. When would that be? If you have dependencies. There are some dependencies which are very nice and useful, but you might not have upgraded. The version numbers I'm using are from the project's respective websites. I didn't verify whether the numbers are actually correct. I used their information. If you are running, for example, older Cassandra, older Grails, especially older Hadoop, planning an upgrade to a version that actually supports 11 might take time. It might be that you will be forced for some time to compile with 8. Especially, if you have a dependency that uses the older version of Thrift somewhere. It can be finicky. It can be very particular about the JDK version that you run it on. That's the fact of life. Then, what you get to do is compile that piece of code with 8, until you manage to upgrade whatever your environment is on the version numbers. Then you're stuck. This is also possible. This happened to me before. We were having a lot of Spark related code. We would compile on 8, and then it would run on 11. A bit convoluted. A bit funny. Some unusual situations might happen just purely because you switch from 8 to 9, which starts to do modules and then removes a bunch of things. It is also an option. It's absolutely a viable option. It's absolutely ok. It's socially and morally acceptable to compile on 8 and run on 11.

Build Tools - jdeps

If you choose to do so, there are two tools called jdeps and jdeprscan. Jdeps shows module dependencies. It has this amazing command line, especially the --++ thing is great for any enterprise project. If you can imagine typing all the JARs that you would have to type in there. I couldn't and nobody else can, so there is a Gradle plugin, there is a Maven plugin. You can put that as part of your build.

jdeprscan

There is also jdeprscan. It also has a classpath option. It also comes with a Gradle and a Maven plugin. The hidden fact of life is you also get the features that these two good people deliver by using Javac. We didn't use those. I mentioned them for the sake of completeness. If you need to go through that process, the tools are under the bin directory of your JDK. You don't have to.

Maven

If you're using Maven, what you will have to do for sure is upgrade compiler plugin to 3.8.0, which is dated now. You can go up a few versions on top of that, but this will allow you to run with more modern JDKs. Also, since I mentioned Maven. Maven users probably might have seen a Maven versions plugin. Maven versions plugin has this awesome feature, which is called display plugin updates, which can tell you which plugins you have within your Mavenized view of the world, can be upgraded and what requirements do they come with. Obviously, the list that this plugin produced has been much longer. We're on the fifth floor so I would probably end up somewhere underground, because it was a really long list. If you require Maven 2.0.2, you can upgrade Maven Java.plugin from 2.10 to 2.2. Does it really make sense? It's discussable. It will tell you a lot of things that you can actually upgrade. I think it has an option in which it can actually upgrade to whatever it thinks is the correct choice. I don't trust Maven that much. I prefer to change versions manually and verify that things work. At least it can give you an automated report about what's possible, what can happen. Another awesome thing that Maven can do and can help your pains when upgrading software or following what software updates are out there already, is versions display dependency updates. As for plugins, it can do the same for your dependencies, which is very useful.

Gradle

If you're using Gradle, like for example we are, if you want to run Java 11, v5 is good, v6 is good. If you're on v4, my recommendation would be go to v5 first, go to v6 second, because if you try to upgrade bravely from v4 to v6, I will probably wait for you to say that you actually decided to go with 5 in between because it's just easier. V4 to v6 is a bit of a rocky ride, directly. Do it with the interim step. People have tried. I think it's a popular opinion that putting a v5 in between is a good step. You can stay on v5 and you don't have to go v6 immediately.

Libraries - Mockito

Now we'll go through some libraries, just to show you what happens. For example, Mockito. An amazing library originating back to an office I used to work with. Formally, it supports JDK 11 since 2.20.1. If you looked at release notes in the Doc directory of Mockito, you'll see releases that tack some JDK support, bug fixes, or changes with a few numbers. If you look at JDK 9, official support happens. Then there are some other bug fixes that happen. What I'm trying to say is the official support, that's not sometimes equal, that everything will work as expected. Because those JDKs change reflection. They change bytecode generation, quite significantly. If you have libraries that do that within your environments, you might get unexpected surprises. It's a part of life. It happens to Mockito. It happens to all the other libraries as well. Also, it might be that because we did it a year ago, if you take the latest versions, you will not have to go and discover all the fun problems that people have solved, because the world keeps moving forward and people upgrade libraries. Most things seem to work on 11 reasonably well, with the exception of modules because modules are not fully supported by some languages, but we'll get there.

jOOQ

We also use jOOQ. jOOQ officially supports JDK 11 since 3.12.0. We actually are still on 3.11-something. We probably should have upgraded but it turns out we didn't have to because it still works.

Flyway

If you do your database migrations manually, Flyway is something that helps you manage database migrations. Super nice. Super easy. Runs really great. Gives you repeatable, testable databases. If you combine it with testcontainers, you can run database tests and all the other tests within a container in a repeatable environment. We use Postgres, so that works out of the box. If you have to use Oracle database, I have no idea what the case is there. This is a talk from our context. Flyway supports JDK 11 since 5.2.0. We actually encountered some issues when we tried to upgrade Flyway and we decided to stick back to 4.2.0. We are still there, because it works for us without any problems whatsoever.

JEP 320

Somebody once joked that if you want to come across as an expert in the Java land, you should mention JEPs by numbers. I am not in that space. I had to Google it. JEP 320 is the one that removes some components of Java EE from the JDK. There has been JAXB that was part of your JDK, in 8. It's no longer there. If you want to use JAXB, or if you want to generate some code based on WSDLs, you will have to explicitly add those tools to your build. That's all this JEP does. That's all the modification that this JEP forces, potentially, on your applications.

Languages

Which languages do we use? We primarily use Java everywhere. We default to Java. We're quite happy with that. We also use Groovy in places. Why Groovy? Because it has Spock. Spock is awesome for writing tests. That's my personal opinion. We have upgraded Groovy to a version that supports JDK 11. This just worked. We also use Kotlin in some places. I personally prefer Scala. There are a lot of people and a lot of opinions and I will not tell people what to do, because everybody has their own. That's fine. Kotlin supports JDK 9 to 12 since 1.3.30. This just worked. Scala, we are 2.12.7. We just switched versions of the underlying JDK. It just worked. If all features, all changes happened like that, life would be beautiful.

JDK 11 compatibility or support in terms of Scala is not fully complete because it doesn't fully follow the Java Platform Module system. This is a very popular thing that's not fully out there yet. If you want to heavily rely on modules in Java and tie it with the language support, that might not be there. We are not doing that because our applications are isolated. They sit in their own containers. We don't need it. If this is a killer feature that you would like to see from Scala, check the current state because the last time I checked they were still working on it.

Testing with Postgres

I mentioned Postgres and testing with Postgres. Historically, a year ago, we used to use something called PostgreSQL Embedded. PostgreSQL Embedded was a very nice project with the tiny caveat of, it stopped being actively maintained. Then, somebody after they stopped maintaining it, upgraded it to make Java 9 class compatible. Because both of those commits in the red frames happened after we migrated, we actually switched the testcontainers and we never looked back. Also, because this project is not really developed anymore on testcontainers, we are very happy with. My recommendation is, use testcontainers. If you've never tried it, if you have to mock a database, bring up a database for integration testing, absolutely do it. It's an awesome tool. The authors of frameworks are also frequent visitors to many conferences. They are very approachable on Twitter and other mediums. Wholeheartedly recommend.

Dependencies

Let's say you want to tackle the problem of upgrading, and you have three components, component A, B, and C. A depends on B, B depends on C. In the case of what I deal with on a daily basis, there is the application that actually delivers the service that depends on an Alpha framework. Alpha is our own event sourcing CQRS framework. Commons is a commons library. Everybody needs one. Everybody has one. What we started doing, how the migration actually worked is we took the outermost layer first, making it compile without any problems on Java 11. It does require some library changes. It does require some build file changes. It does require some Gradle file modifications. This took some time. It wasn't particularly painful. After that, we started testing using 11. Then we started running this using 11. Previous components were being compiled, and everything was there happening with JDK 8. After we did with the outermost layer, we went one level deeper, and so on. You get the pattern.

BouncyCastle

Now I should mention some of the funny quirks that we encountered on the journey, for example, BouncyCastle. In JDK 8 I know a lot of people that used to install this into the JRE lib\ext directory. This directory is no longer present. If you want to install it to the same effect in JDK 9+, you have to do this thing. What we do with BouncyCastle now is add it to the dependencies normally. It just works. We're happy. No problem. No magic required.

VM Options

There is also the case of VM options. You've probably heard about Unified GC logging. The syntax of getting output from GC logs changed. The formats changed as well. What this means in practice, is we replaced top with the bottom, more or less, of course. What happens is the line that tells the JVM to lock everything from GC becomes simpler. That's it. All changes that there are. If you have very big command lines, if you think it might be a problem in your environment, somebody was very kind and developed this tool called jaCoLine, which can tell you what's good in JDK 8. Then if you switch to JDK 11, it can tell you what's no longer good. Super awesome. Super neat. We didn't have to use it. I found it much later. It's only fair I mention it. Also, whenever I mention anything to do with command lines, I always like to say a huge thanks to Chriswhocodes, who is the person behind this awesome website that takes all of the flags that different versions of JDKs contain, and it gives you a human readable description. It tells you what the default is. Which component, which architecture, which operating system is that specific to. When has it appeared? When it has disappeared. If you want to explore that area, Chriswhocodes. Java 9 came with a new Zip file implementation, which means one flag we could kill, -Dsun.zip.disableMemoryMapping. It's gone.

Time Precision

Time precision is a bit of a problem, or a bit of a source of interesting things and phenomena. JDK 9 has increased the precision of the Java time clock depending on the precision of the underlying operating system source. That seems like a very innocuous change, but in reality, it's a very profound one. Because what it means is if your operating system can be more precise than milliseconds, it can actually give you microseconds in that clock. Of course, depending on the operating system. I don't know what the level of support is on Windows. On Linux, it works and it's more precise. I don't use Windows. I just don't know. Since I mentioned time before. I told you what the bug is. I have to always recommend Shipilev's excellent writing about time. There is a lot of complexity with dealing with time in computers. The article is from 2014. It's an awesome read. I do recommend reading that. It's not a very sleep-inducing article. I found it extremely exciting, because there are things out there.

What happens when you do Instant look now and print it? Instead of getting three digits after the dot, you now get six. What's the meaning of this? Postgres stores timestamps with microsecond precision. Before, that will be truncated to just three, which means all your tests will just work. Now things start to slightly go out of sync. Some code assumes three digits. Some code assumes six digits. The code that has three digits gets expanded and doesn't really merge with six, which means you get a lot of strange exceptions, or bugs, or test failures in your application. This took us quite some time to fix.

Recommendation

What we did and what I usually recommend people do anyway, is if you're dealing with time, insert the time through a piece of code that you control. If you rely on Instant look now, and things like that, you don't have any control. Mocking becomes tricky, or convoluted, and hard. We do something like a time provider. A time provider is the entry point to how applications in Revolut get time. This has a similar API, but not the same as the java.time package. Why? An IDE will not get surprised, and it will not give you suggestions of using the wrong methods and the wrong static methods. We use the types. We didn't do anything. It's just an entry point. If somebody needs to acquire time they get it from their time provider. Time provider can say what precision is expected of a clock. For us, in most cases, millis is actually what we want, because of all the provider integrations. Millis is what we get on the entry. Millis is what we also should provide on the outbound.

DateTimeFormatter

Another bug that we discovered, not only us of course, DateTimeFormatter. When you were using DateTimeFormatter with locale UK, and given one of the offices in the UK, and we have a lot of customers in the UK, we got a NullPointer. Strange, but this happens. What you have to do is instead of doing that with locale of UK, you have to do locale English. We replaced everything with locale English because that at least allows some consistency because not using defaults in locale is a source of joy.

Compiler Bugs

There have been compiler bugs. The easy one, sometimes some builds, almost at random, would fail with a bug saying message file broken. You don't even see what the problem is. If you explore the JDK bug database, you will see that that bug is a duplicate of something, which has only been fixed in version 12. In here at the bottom, there is a list of backports. It's not complete because it's growing. This actually is highlighting why the support model and release discussion. It is an important thing to keep in mind because if there is a bug in JDK 11, and it's fixed in 12, 9 and 10 will not see this fix ever. 11 will see it because it's an LTS edition. The fix is in 11.0.4. It is publicly available, but to use it in production you need to have a license, and so on. Using the supported LTS goes a long way. Anonymous type class inference gives you a NullPointer. What does it mean? That if you try to do something like that, you'll get a NullPointer. Our approach to the problem, don't use anonymous classes as frequently. We just did it and the bug went away.

Actually, the biggest problem that we had with Javac and generics, is Javac has a problem with captures. The problem is still unresolved. The problem manifests itself in a runtime problem. Compilation sometimes goes ok, sometimes not. If you have a generic which extends something, what we had to do was replace it just with a question mark. That bug is still open. That bug is still happening. Some of the results of this bug were runtime failures. Our compilation went fine. Then things exploded at runtime in production. Not what you want. I have to mention it. What can you do if you actually care about the type? Other languages come to the rescue. I am a fan of Scala's type system, because it allows you to express a lot. If you want to capture those types, you can use Scala as a workaround, and ask Scala calls to give you hints and give you a lot of help. If you can live with just a question mark, that's also an option.

Life with G1 by Default

Then people who are performance oriented will probably say, G1. There has been a lot of conversations about G1 in 11. The default GC is G1. It's a different G1 than the G1 that comes in JDK 8. Completely not the same thing. We are happy with this. We use New Relic. We see garbage collection expressed as CPU time instead of pause time. I would like to see the pause time because it's important and useful. Hopefully, this will happen soon. In JDK 11, full GCs with G1 are multi-threaded, actually since JDK 10. We don't have to do magic, almost any fiddling with GC tuning, to achieve pauseless on a human perception level for our services. This is such an empowering life-changer compared to JDK 8 that is hard to describe. Our heaps and our application usage has kept growing month by month. What we do is just change the container size, give it more memory. That's it. It still works reliably. We still don't have problems. We don't have crazy pauses. It works well enough that we don't have to pay too much attention. With a caveat, we don't run Cassandra. We don't run too much latency sensitive code. This is in the context of what we do here.

Basically, if you run web services, or microservices that have to reply within a few hundred millis, from our point of view, we just kept increasing heap sizes, and there were no issues. Another nice feature is we had less problems that required GC logs, to the point in which, in most cases, we actually don't need them. I would still recommend you do take GC logs because they are useful when they are needed. Another nice feature of G1 in 11 that we have observed, is if an application is to do an out of memory exception, you will see it faster. We are seeing it much faster than it used to be in 8. In 8, that behavior is still present, but there is a case in which you can get an out of memory exception GC overhead limit exceeded. Is there anybody who has never seen an exception like this? What happens here, GC will try to run. It will try to reclaim memory. It will say I give up, after it passes a certain threshold. I know a lot of people don't know that this flag even exists. The application spends 98% time in GC. 98% time in GC, all cores are burning. Your machine is steaming hot for speed. You're waiting for an exception to happen. Our exceptions come much faster in 11. We're happy.

Memory Appetite

Caveat now to 11. We noticed that containers started dying at first. Imagine I have logged into Oracle Cloud because it's the most blessed environment that I can. I'm comparing JDK 8, at the bottom, with JDK 11 at the top. What you can probably see is that RSS for a similar memory usage is slightly higher in 11. That's basically what we have observed across the board that Java 11 requires slightly more memory from the operating system than 8. Which means we just changed the max and start heap size for a container so that everything fits into a precisely calculated container size. It worked. Faster out of memories, I already mentioned.

Language Features Adoption

Java 11 comes with new language features, which you already know very well. Some people on Twitter claim that var is an absolutely useless thing, and they don't want to use it. Some people hate it. They are absolutely entitled to their opinion. I've analyzed one of our projects, I see that at first people were a bit cautious with using var, but then the feature quickly caught on. People actually like using that. People like replacing final something with vars because it's just less characters. It's just less reading. Usually, the code that it replaces is quite obvious. We've seen great adoption with this. You might see a dip. The dip happens because we took a module out of the main codebase.

Also, you probably know that as of release 9, underscore is a keyword. You cannot use this as an identifier. Don't worry, people are creative. This is what I just said. You can use double underscore. Double underscore is still legal. If you have used other languages, if you want to have underscore in Java, you can do double underscore, it works to the same effect.

Java Flight Recorder and Mission Control

There is another awesome thing, Java Flight Recorder and Mission Control are open source. That means that you don't need to get a license to use them. You can just download it. You can compile them yourself. You can analyze. You're running JDK 11 application with those excellent tools. If you've never used it, I would recommend. Give them a try because it's going to change your life for the better.

What Is Next?

What do we plan to do next? One option is go Java 13. Java 13 has a lot of improvements in G1 space. Much better behavior from the garbage collector. For example, much more controlled pause times. If you were to graph 11, it can be all over the place. 13 can be following a trend. Less memory hunger, so smaller RSS size compared to 11. However, it's not supported anymore. It's going to be time for 14 which comes out very soon. How about 14? 14 has an amazing feature for observability. Flight recorder, event streaming, we want it. We would like to have it now even. However, Gradle does not work with Java 14. Gradle doesn't work with Java 14 because of Groovy. What we are planning to do is wait them out. Let them solve their issues. When it's available, we will go there because we don't need that feature. It's not a killer feature for us. We will probably get towards that stage, or we can hope for some very kind people to backport those features, hopefully, to maybe all their JDKs. That would be awesome. What we are going to try next is Graal, especially with native image. It's a thing we want to test. I am aware that there are some changes to how GC works in native images. We want to give it a try. How should you go about upgrading your apps to JDK 11 or later? Just try to compile them. Try working with that.

Conclusion

I want you to have great Java, if you're before 11. It's probably a good idea. It has worked for us very well, and many of you can have a good experience, even in a big organization. The whole process will require some planning. The list of changes and the list required to the application is not just changing the underlying JDK, in most cases. It will require some work. It will require some planning and scheduling, and working with a product owner or whoever organizes your work. This should be done. This will put you on the path to greater, newest features.

Questions and Answers

Participant 1: You mentioned that one of the driving factors behind the migration was that you wanted to move faster. You migrated, are you moving faster now?

Andrzej Grzesik (ags): Yes, we are. How is that exactly happening? For example, if we need to react to load increasing, because our customer base is increasing, we just give containers more memory. It just works. We don't have to do anything there. Also, another important thing is hiring and finding talented people who are inspired to do good stuff. Those people are probably already curious about 11. They want to work with an organization that's already using new features. It's a very important thing, because talent in tech is very hard to find, because there is a lot of competition.

One question that you might have asked, the performance. We were happy about the performance, no problems there. It's a human perception latency that we are trying to optimize with most services, which means it's not millisecond or sub-millisecond latencies that we have. We're happy with having replies with 10, 20, 30, in that range.

Participant 2: You mentioned that you moved away from Embedded Postgres to testcontainers. What's the reset time between tests for testcontainers?

Andrzej Grzesik (ags): I don't know the individual time for a reset. I could maybe try to check it. Out of the blue, I don't know the whole test suite. There are a lot of tests. It takes almost an hour. The number of test cases and tests is counted in tens of thousands. It's an effort. It takes time, but we are at least quite sure of our applications.

Participant 3: Just wondering with your Docker images, did you find any particular base image for Java 11 that you recommend?

Andrzej Grzesik (ags): We build our own. The way the application is assembled, there is a base image that has Java runtime in it. Then we marry that image with a JAR file that contains the application and all the other things that we have. Look at what OpenJDK people are doing. That's probably a good source for inspiration at first. I cannot recommend anything in particular because we're baking our own.

Participant 4: I was just wondering, I don't know how much independence the teams within your organization have. How did you coordinate it across the firm? Did you have one rollout for all the teams together or were there some pioneering teams that drove it and the others followed?

Andrzej Grzesik (ags): We did have to coordinate. There is a bunch of applications that make Revolut do what Revolut does. We started with one application and then we moved on. When we were quite certain of the image that has been upgraded, and that the outer layer of the onion already works, we started to switch individual applications. Verify that they run correctly. Because of course, we have tested. We were quite sure that it's going to be fine. With the generic bug, that was an unexpected one. That still happens because it's still out there. We had to modify individual applications, one by one. There was a blessing from the CTO. That was required.

Which JDK?

We've obviously adopted OpenJDK.

 

See more presentations with transcripts

 

Recorded at:

Oct 01, 2020

BT