Solving Fat JAR Woes at HubSpot

They're doing this the wrong way.

The answer is "simple". It's a "maven aware" class loader.

Out of the box, maven bundles the pom in the META-INF of a Java jar.

Maven already has code to "run" an application based on its pom, standing up the class path before executing the prescribed main routine.

That code need to "simply" be ported as a stand alone class loader, invocable by the main method. It will essentially do what maven does - it will reify the dependencies, use the classic local maven repository, build up an internal class path that based on those dependencies, and then fire off whatever method the developer desires.

MavenClassloader.run(MyAppClassWithMainMethod.class, args);

Its perhaps reasonable to desire not running an assemble(sort-of) job in production where build once deploy many(same artefact) is the philosophy..

What I fail to understand is the 1000s of micro-services bit, why are there soo many micro-services?
What is the bandwidth gain thats being talked about? If there is an artefact server which is mirroring some internet maven repos, and is within the same environment. That could potentially reduce bandwidth dramatically isn't it? Unless I've mis-understood the bandwidth concerns.
Looks like an invented problem to me!

Well the you're at an impasse, then, aren't you? The artifacts need to get to the machines somehow. So they either have to already be out there, or you have to drag them with you each time. It is straightforward to add a flag to the command that simply tells the process to satisfy its requirements before "running it for real", but for whatever reason, folks itch at that too.

This is no different than pre-staging all of the jars, setting the class path up, and firing off the code. But we did that back in 1999, and had to manage all of that. It was awful and brittle and nasty back then.

This is a natural extension to what Maven can already do, using tools and infrastructure everyone already has.

Of course, you don't have to do anything at all, you can simply run mvn directly using the exec plugin, but folks cry and whine about that as well.

Hey Will, thanks for the feedback. The way I think about it is that before we would upload a single large artifact at build time and download a single artifact at deploy time. Now, we upload lots of small artifacts at build time and download lots of small artifacts at deploy time. These are functionallity equivalent and you'll notice in either case there's nothing happening at runtime, by that point all of this work is already done. So my philosophical objection to your suggestion is that we don't have Maven installed on our application servers and we like it that way. We let our Mesos scheduler, Singularity, handle artifact downloads for us at deploy time in a language agnostic manner. Previously we gave it a single S3 URL to download (fat JAR), but now we give it a list of S3 URLs (the app plus its dependencies) and it handles downloading these artifacts, caching on disk, retrying, verifying checksums, signatures, etc. Since this was already implemented in Singularity, it's hardly any more complexity at deploy time and it ended up being really easy to integrate into our deploy process. By the time the app starts up, all of its dependencies are guaranteed to be present (if an S3 download failed, the deploy would have failed) so it's totally transparent to the application.

I also have some more pragmatic concerns, such as Maven not using locking or atomic operations when interacting with the local repository, so if we had concurrent deploys accessing the local repository we would run into issues. Also, we use snapshot versions extensively for our internal dependencies and our Nexus instance is configured to only keep the latest snapshot to keep disk space under control. But to make sure our dependencies aren't shifting at deploy time, SlimFast always uses resolved snapshot versions. So if we were fetching artifacts via Maven at runtime, these resolved snapshots would quickly point at purged artifacts and the app would fail to start up (ie, my app depends on artifact A snapshot version 1, artifact A builds again and publishes snapshot version 2 to Nexus which causes snapshot version 1 to get purged, now my app can't start because it can't fetch artifact A snapshot version 1 from Nexus). Additionally, for dependencies that come from 3rd party repositories there's also concerns that those repositories could disappear at any time. This is acceptable when it causes a build time failure, but if we fetched via Maven at runtime it could cause a serious outage.

Hey Bruce, thanks for the questions! We have lots of microservices mainly because of the breadth of our product and our team structure. The HubSpot product is extremely broad: our customers can manage their leads, segment them into lists, build automated marketing workflows, score them with predictive lead scoring, build their website, run their blog, send emails, publish to various social media networks, view analytics across all of these channels, and much much more. Each of these pieces of the product is owned by a different product team, and each product team owns the potentially dozens of microservices that power that piece of the product. For example on the email side, there's an API to send an email, there's an API to record email opens and clicks, there are jobs to handle bounces and spam reports, there's an API to fetch statistics about an email send for display to the customer (open percentage, click percentage, etc.), and many more. These aren't integrated into a single service because they have very different performance characteristics and reliability needs. For example, the click tracking service needs to respond within a few ms and emphasizes availability and eventual consistency because if it's down the email links won't work. This is very different from the API that sends emails, which may take a few seconds to complete all of the SMTP operations and favors consistency over availability to make sure we never double-send the same email.

On the bandwidth side, the bandwidth savings are from not uploading a fat JAR on build and downloading it on deploy. Instead, we only upload the dependencies that have changed, which are usually none, and download the dependencies that aren't already cached on the application server, again usually none. Previously, the transfers were happening within the same environment so the bandwidth was free, but the real problem was the time being wasted. It could take ~30 seconds to assemble the fat JAR, another 30 seconds to upload it to S3, and another 30 seconds to download it at deploy time. Now all of these times are measured in milliseconds. For some background, the metrics the HubSpot platform optimizes for are developer time, productivity, and happiness. If we found a way to save 30 seconds for our developers on builds and deploys but it used 3x the bandwidth and storage we would do it in a heartbeat. It just so happened that this approach saves bandwidth and storage in addition to time.