Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Kubernetes Workloads in the Serverless Era: Architecture, Platforms, and Trends

Kubernetes Workloads in the Serverless Era: Architecture, Platforms, and Trends

This item in japanese

Key Takeaways

  • Microservices architecture has evolved into cloud-native architecture, where many of the infrastructure concerns are provided by Kubernetes, in combination with additional abstractions provided by service mesh and serverless frameworks.
  • The brilliance of Kubernetes lies in its extensibility for new workloads through the Pod abstraction, which also supports the emergence of new cloud-native application patterns.
  • Kubernetes supports not only stateless applications, but also stateful workloads, singletons, batch jobs, cron jobs, and even serverless and custom workloads via CRDs and operators.
  • Today’s serverless platforms have significant limitations preventing them from being adoption by enterprise companies, which have strong interoperability and portability constraints.
  • The serverless ecosystem is evolving by exploring standard and open packaging, runtimes, and event formats. The advances in these areas are blurring the differences between cloud-native and serverless workloads, and are already pushing serverless offerings towards open, portable, and interoperable frameworks. 


My "Integration Patterns in the Serverless World" talk at Red Hat Summit 2019  was inspired by customers asking where serverless workloads fit in the current enterprise cloud-native landscape, which is dominated by Kubernetes.

This article summarizes that talk but with less focus on integration and a deeper look into different Kubernetes workloads and the limitations of serverless offerings. If you would like to attend to my next webinar on a related topic, please check out the details at the end of this article.

Progressions of Distributed Architectures

Before predicting where serverless is heading and where it fits, we need to analyze how and why we got here in the first place.

Monolithic Architecture

My career in large-scale integration projects started when service-oriented architecture (SOA) was the most popular architecture and an enterprise service bus (ESB) was its most common implementation.

SOA was based on good principles, of which the majority are still valid: contract-first development, and loosely coupled, composable, and stateless services that are also autonomous and reusable.

ESB frameworks provided a good set of capabilities, such as protocol translation, technology connectors, routing and orchestration mechanisms, error handling, and high-availability primitives.

Progressions of distributed architectures

The main issue with SOA and ESBs was centralization, from both architectural and organizational points of view. An important principle of SOA was reuse of services and components, which led to the creation of layered services architectures that enabled reuse but caused tight architectural service coupling. Organizationally, ESBs were owned by a single team, turning the middleware into a technological and organization bottleneck for scalability and, more importantly, for rapid evolution. Complex industry specifications evolved slowly, defined by a small groups of people. 

Microservices Architecture 1.0

While SOA provided a good foundation for tackling enterprise-level complexity, technology evolved rapidly. The open-source model led to faster innovation and became the new distribution and standardization mechanism for technology.

In parallel, agile development practices such as XP, Scrum, and kanban produced iterative software development but this approach clashed with the existing monolithic architectures that were incapable of coping with quickly deployed incremental design. As a result, features were developed in two-week-long iterations but only deployed to production once or twice per year.

Microservices architecture emerged as a panacea at the right time, promising to address these challenges. Microservices architecture allowed faster change thanks to its guiding principles. Services became modeled around a business domain, which helped contain the change within service boundaries more effectively than the reusable SOA services. Autonomous and independently deployable services permitted each service to evolve and scale at its own pace.

While these new principles optimized SOA for the era of fast iterations and experimentation, they also introduced challenges, by turning every application into a distributed system. As a consequence, microservices also must be highly automatable, highly observable, fault tolerant, and everything else that a distributed system component must be. This was a price that not everyone realized at first, nor were they necessarily ready to pay.

The microservices architecture was born as a technological answer for the age of fast, iterative development and experimentation but we built the first generation of microservices on top of archaic tooling and they were difficult to manage. During the ESB era, business logic leaked into the platform, causing all kinds of couplings, and during the early microservices era, we observed the opposite: too many infrastructure concerns leaked into every microservice. Early microservices had their own challenges; they had to do their own service discovery, configuration management, network resiliency, log distribution, etc. Having the ability to create tens or hundreds of new services didn’t necessarily mean that an organization was ready to manage and deploy them into production with their existing tooling. The processes and supporting tooling for releasing, managing, and handling over services to operations teams had to be improved. All that pushed us to the Kubernetes era.

Cloud-Native Architecture (a.k.a Microservices 2.0)

Microservices architecture optimized SOA for rapid change, but it achieved that by trading code complexity for operational complexity. Its challenges led to the adoption of containers, which took over the industry overnight. Containers came about as a technical cure for the pain of deploying a large number of microservices uniformly and effectively enabled the modern DevOps practice. With containers, we could package applications and run them in a format that development and operations teams could both understand and use. It was clear even early on that managing tens or hundreds of containers would require automation, and Kubernetes came from the heavens and swept away all the competition. Kubernetes addressed the technological challenges and the DevOps practices addressed the cultural aspects of microservices. This cloud-native tooling was so fundamental that it caused the emergence of a second generation of microservice platforms.

This was the start of a new shift, a shift of infrastructure responsibilities moving from the application layer into the platform layer. Cloud-native platforms (initially many, but eventually Kubernetes primarily) provided features such as resource isolation and management, automated placement, declarative deployment, health probes and self-healing, configuration management, service discovery, autoscaling and more. These features allowed application developers to focus on the business logic and use out-of-the-box platform features to address the infrastructure concerns uniformly.

At the time, I compared popular microservices 1.0 tooling (Spring Cloud and Netflix OSS) with microservices 2.0 tooling (Kubernetes as the de facto standard) and I received mixed reactions from readers. Today, it is a more widely understood and accepted transition, confirmed by the complete dominance of Kubernetes as the microservices management platform and deprecation of many Netflix OSS libraries from the earlier generation.

But all of that is history. Let's explore what is coming next with Kubernetes.

The brilliance of Kubernetes

There are many fascinating elements of the Kubernetes architecture: the containers providing common packaging, runtime and resource isolation model within its foundation; the simple control loop mechanism that monitors the actual state of components and reconciles this with the desired state; the custom resource definitions. But the true enabler for extending Kubernetes to support diverse workloads is the concept of the pod.

A pod provides two sets of guarantees. The deployment guarantee ensures that the containers of a pod are always placed on the same node. This behavior has some useful properties such as allowing containers to communicate synchronously or asynchronously over localhost, over inter-process communication (IPC), or using the local file system.

The pod’s lifecycle guarantee makes sure that the containers in a pod are actually managed in two groups: init containers and application containers. Init containers run first; they run one after another, and only if the previous container has completed successfully. Init containers enable a sequential pipeline behavior, and a single container performs each step. On the other hand, application containers run in parallel and without any ordering guarantees. This group of containers enables the popular sidecar pattern for extending and enhancing the functionality of a preexisting containers with orthogonal functionalities.

Deployment and lifecycle guarantees of a Pod

The extensible control-loop mechanism, combined with the generic pod characteristics, enables Kubernetes to handle diverse workloads, including serverless. Let’s examine these diverse workloads and see what the suitable use cases for each are.

Cloud-Native Workloads

Proving why Kubernetes is a universal platform capable of supporting diverse workloads and use cases requires exploring the different workload types and their needs.

Stateful services

Let’s start from a workload that is the least exciting, but almost always present in enterprise environments: stateful services. Stateful services with business logic can be turned into scalable stateless services by moving the state into external data stores. Such a design moves the constraints from the business services into the data sources, which become the stateful components in the architecture. These data stores are typically off-the-shelf relational databases, distributed caches, key-value stores, search indexes, etc. Managing distributed stateful components on dynamic cloud infrastructures requires certain guarantees such as:

  • Persistent storage —state typically lives on a disk, and distributed stateful applications require dedicated, persistent storage for every instance to store its state.
  • Stable networking ID — similar to the storage requirement, a distributed stateful application requires a stable network identity. In addition to storing application-specific data in the storage space, stateful applications also store configuration details such as hostname and connection details of their peers. This means that every instance should be reachable via a predictable address that should not change dynamically, as is the case with pod IP addresses in a ReplicaSet.
  • Stable identity — as we can see from the preceding requirements, clustered stateful applications depend heavily on every instance holding on to its long-lived storage and network identity. That is because in a stateful application, every instance is unique and knows its own identity, and the main ingredients of that identity are the long-lived storage and networking coordinates. To this list, we could also add the identity/name of the instance (some stateful applications require unique persistent names as well), which in Kubernetes would be the pod name. 
  • Ordinality — in addition to a unique and long-lived identity, every instance of a clustered stateful application also has a fixed position relative to others that matter. This ordering typically impacts the sequence in which the instances are scaled up and down, but it can also be used as a basis for consistent hashing algorithms, data distribution and access, and in-cluster behaviors placement such as locks, singletons, or masters.


A distributed stateful application on Kubernetes

These are exactly the guarantees that a Kubernetes StatefulSet offers. A StatefulSet provides generic primitives for managing pods with stateful characteristics. Apart from the typical ZooKeeper, Redis, and Hazelcast deployments with a StatefulSet, other use cases include message brokers and even transaction managers. 

For example, Narayana transaction manager uses StatefulSet to make sure it does not miss any JTA logs during the scaling down of services using distributed transactions. Apache Artemis message broker relies on StatefulSet to drain messages when scaling down a clustered message broker. The StatefulSet is a powerful generic abstraction, useful for complex stateful use cases.

Global Singletons

The singleton pattern from the Gang of Four is an old and well-understood concept. The equivalent in the distributed, cloud-native world is the concept of a singleton component (a whole service or only part of it) that is a global singleton (among all distributed services) but still highly available. The use case for this workload type arises typically from the technical constraints of other systems that we have to interact with, for example, APIs, data sources, and file systems that allow only a single client (singleton) access at a time. Another use case is when the message ordering must be preserved by the consuming services, limiting it to be a singleton. Kubernetes has a few options with which to support these kinds of use cases.

The simplest option is to rely on Kubernetes to run a single instance of a service. We can easily achieve this by using a ReplicaSet or StatefulSet with replicas=1. The difference between the two alternatives is whether you require a strongly consistent singleton with “at most one” guarantees or a weak singleton with “at least one” guarantees. A ReplicaSet favors availability and prioritizes keeping a single instance up (“at least one” semantics). That could occasionally lead to more than one instance running at the same time, for example when a node is disconnected from the cluster and the ReplicaSet starts another pod on a different node without confirming that the first pod is stopped. A StatefulSet favors consistency over availability, and provides “at most one” semantics. In the case of a node being disconnected from the cluster, it won’t start the pod on a healthy node. That can happen only after an operator confirms the disconnected node or the pod is really shut down. This could sometimes lead to service downtime but will never lead to multiple instances running at the same time.

There is also the option of implementing a self-managed singleton from within the application. While in the previous use cases, the application wasn’t aware of being managed as a singleton, a self-managed singleton ensures that only a single component is activated, regardless of the number of service instances (pods) started. This singleton approach requires runtime-specific implementation to acquire a lock and act as a singleton, but it has a few advantages. First, there is no danger of accidental misconfiguration, and increasing the number of replicas still results in only a single component being active at any given time. Second, it allows the scaling of services while being able to ensure singleton behavior for only a part of the service, such as an endpoint. This is useful when the only part of a microservice rather than the whole thing must be a singleton due to external technical limitations on a specific operation or endpoint. An sample implementation for this use case is the singleton feature of Apache Camel’s Kubernetes connector, which is able to use a Kubernetes ConfigMap as a distributed key and activates only a single Camel consumer across multiple Camel services deployed into Kubernetes.

Singleton workloads on Kubernetes

Singletons are another workload type that come up in small numbers, but they are common enough to be called out. Singleton and high availability are two conflicting requirements, but Kubernetes is flexible enough to offer both with acceptable compromises.

Batch Jobs

The use case of batch jobs is suited for managing workloads that process isolated atomic units of work. In Kubernetes primitives, it is implemented as the job abstraction, which reliably runs short-lived pods to completion on a distributed environment.


Batch and recurring workloads on Kubernetes

From a lifecycle point of view, batch workloads have few characteristics that are similar to asynchronous serverless workloads, as they are focused on a single operation, and they are short-lived, lasting until a task is completed. But although the job-based workloads are asynchronous in nature, they do not take direct input from consumers, and they are not directly started in response to consumer requests. They typically know where to retrieve input data from, and where to write the result. If the job has a time dimension i.e. it is scheduled, its execution is triggered by a temporal event on a regular basis.

Stateless Workloads (a.k.a 12-Factor-Apps)

Stateless workloads are the most widely used workload type on Kubernetes. This is the typical 12-factor application or microservices-based system managed on top of Kubernetes using a ReplicaSet. Typically, a ReplicaSet will manage multiple instances of such a service and will use different autoscaling strategies to scale such workloads horizontally and vertically.

Stateless workloads with service discovery on Kubernetes

A common requirement for services managed by a ReplicaSet is service discovery and load balancing. And here Kubernetes offers a variety of options out of the box.

Service discovery mechanisms on Kubernetes

The point here is that even if there are a variety of service-discovery mechanisms that can dynamically detect healthy and unhealthy pod instances, the different service types are relatively static in nature. The Kubernetes Service primitive does not offer dynamic traffic monitoring and shifting capabilities. This is where a service mesh comes into the picture.

Service Mesh

One of the challenges when implementing a microservices-based system is building the commodity features that are not part of the business logic, such as resilient communication, tracing, monitoring, etc. This logic used to reside at the central ESB layer and now has to be segregated and repeated among the smart clients of microservices. The service-mesh technology aims to solve this problem by offering additional enhanced networking capabilities such as: 

  • Traffic routing — A/B tests, staged rollouts.
  • Resilience — retries, circuit breakers, connection limits, health checks.
  • Security — authentication, authorization, encryption (mTLS).
  • Observability — metrics, tracing.
  • Testing — fault injection, traffic mirroring.
  • Platform independence — polyglot, allowing runtime configuration.

If we look closely at these features, we will notice that there is significant overlap in the functionalities provided by integration frameworks.

Service mesh and integration framework responsibility overlap

There are differing opinions on whether moving all of these responsibilities outside of the services is a good approach or not. While there is a clear move of networking responsibilities from the application layer into the common cloud-native platform, not every networking responsibility moves out of the application:

  • A service mesh is good for connection-based traffic routing and an integration framework from within a service is good for content-based routing. 
  • A service mesh can do protocol translation and an integration framework can do content transformation.
  • A service mesh can do a dark launch and an integration framework does wire-tapping.
  • A service mesh does connection-based encryption and an integration framework can do content encryption.

Certain requirements are better handled within the service and some from outside using a service mesh. And some have to still be handled at both layers: a connection timeout can be configured from service-mesh layer but it still has to be configured from within the service. That is true for other behaviors such as recovery through retries and any other error-handling logic. A service mesh, Kubernetes, and other cloud services are the tools of today but the ultimate responsibility for the end-to-end reliability and correctness of an application lies within the service implementation and its development and design teams. This does not change.

The service-mesh technology further highlights the main difference between microservices from the first and second generations: the shift of certain operational responsibilities to the platform. Kubernetes shifts deployment responsibilities to the platform and service meshes shift networking responsibilities to the platform. But that is not the end state; these changes are only paving the way to a serverless world where deployment and traffic-based instant scalability are a prerequisite.

Serverless Concepts

It’s All About Perspective

To discuss the characteristics of serverless, I will use the definition from the serverless working group of the Cloud Native Computing Foundation (CNCF), as it is one of the most widely agreed definitions across many different software vendors:

Serverless computing refers to the concept of building and running applications that don’t require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions, are uploaded to a platform and executed, scaled, billed in response to the exact demand needed.

If we look at this definition as developers who write code that will benefit from a serverless platform, we could summarize serverless as an architecture that enables “running finer-grained functions on demand without server management”. Serverless is usually considered from the developer’s perspective but there is also another, less discussed perspective. Every serverless platform has providers who manage the platform and servers: they have to manage coarse-grained compute units and their platform incurs costs 24x7 regardless of demand. The providers are the teams behind AWS Lambda, Azure Functions, and Google Cloud Functions or the team in your company managing Apache OpenWhisk, Kubernetes with Knative, or something else. In either case, the providers enable developers to consume compute and storage as a workload type that does not have any notion of servers. Depending on the organizational and business factors, the providers can be another team/department in the same organization (imagine an Amazon team using AWS Lambda for its needs) or can be another organization (when an AWS customer uses Lambda and other services). Whatever the business arrangement between providers and consumers, consumers have no responsibility for the servers; providers do.

Serverless Architecture

The definition above refers only to “serverless computing”. But an application’s architecture is composed of compute and data combined. A more complete definition of the serverless architecture is the one that includes both serverless compute and serverless data. Typically, such applications will incorporate cloud-hosted services to manage state and generic server-side logic, such as authentication, API gateways, monitoring, alerting, logging, etc. We typically refer to these hosted services as “back-end as a service” (BaaS) — think of services such as DynamoDB, SQS, SNS, API Gateway, CloudWatch, etc. In hindsight, the term “serviceful” rather than “serverless” could have been a more accurate description of the resulting architecture. But not everything can be replaced with third-party services; if this was the case and there was a service for your business logic, you wouldn’t be in business! For this reason, a serverless architecture usually also has “functions as a service” (FaaS) elements that allow the execution of a custom stateless compute triggered by events. The most popular example here is AWS Lambda.

A complete serverless architecture is composed of BaaS and FaaS, without the notion of servers from the consumer/developer point of view. No servers to manage or provision also means consumption-based pricing (not provisioned capacity), built-in autoscaling (up to a limit), built-in availability and fault tolerance, built-in patching and security hardening (with built-in support policy constraints), monitoring and logging (as additional paid services), etc. All of that is consumed by the serverless developers, and provided by serverless providers.

Pure Serverless

If the serverless architecture sounds so good, why not have a pure serverless architecture with all components 100% serverless and no notion of servers? The reason can be explained by the following quote from Ellen Ullman: "We build our computer systems the way we build our cities: over time, without a plan, on top of ruins." An enterprise system is like an old city; usually, it has existed for over a decade, and this is where its value and criticality come from. This is what makes it "enterprisey". Imagine London, a city that has existed for over 2,000 years, with its century-old Tube system, narrow streets, palaces, Victorian neighborhoods and supply systems; such a complex system in use can never be fully replaced by a new one, and it will be always in undergoing restoration and renewal (refactoring, upgrades, migrations, and re-platforming in IT terms). In such systems, the state of change is the norm; the state of mixing old and new existences is the norm. It is how these systems are supposed to exist.

Serverless 1.0

Container technology has existed for many years in various forms, but Docker popularized it and Kubernetes made it the norm for deployments. Similarly, serverless technologies have existed for many years but AWS Lambda made it popular, and we are yet to see who is going to take it to the next level.

Serverless 1.0 was basically defined by AWS and represented by AWS Lambda for the FaaS component and by other AWS services such as API Gateway, SQS, SNS, etc. for the BaaS components. With AWS defining the trends and others such as Google and Azure trying to catch up, here are some of the characteristics of serverless from the current generation that are less than ideal, and are potential candidates for improvements.

The non-deterministic execution model has:

  • unpredictable container lifecycle and reuse semantics with implications on cold start;
  • constraints on the programming model, which influences code initialization logic, causes callback leaks, creates additive recursive call costs, etc.;
  • unpredictable resource lifecycles such as /tmp file storage;
  • arbitrary limits on memory, timeouts, payload, package, temp file system, and env variables; and
  • an overall, non-standardized execution model across the industry, with non-standardized constraints that influence the programming model for the specific serverless vendors.

Limited runtime support means:

  • The serverless software stack, which is the combination of an operating system, language runtime and libraries version, is limited to (a single version of) an OS, JDK, and application libraries (such as AWS SDK).
  • The platform support policy often dictates under which terms any serverless stack components can be deprecated and updated, forcing all serverless users to follow strict timelines at the same pace.
  • Programming APIs can cause issues. While I have had a good experience using AWS SDK for other services, I don’t like the fact that all functions have to be heavily coupled with the package and its programming model.
  • We use non-standard packaging. Using .zip, uber-JAR, or lib directory with a custom AWS-specific layering and dependency model does not make me confident that the packaging is future proof or that it will work across serverless platforms.
  • Custom environment variables (that start with AWS_) would not work on other serverless platforms.

Proprietary Data Formats

Proprietary data formats are an obstacle. Events are the primary connectivity mechanism for the functions in a serverless architecture. They connect every function with every other function and BaaS. They are effectively the API and the data format for the functions. Using events defined in the package in all functions guarantees zero interoaparability.

No Java Support

While there is a Java runtime for AWS Lambda, there is such a mismatch between Java and Lambda that even AWS does not recommend it. In Tim Bray’s “Inside AWS: Technology Choices for Modern Applications” talk, he suggests using Go and Python instead. Rather than trying to re-educate millions of Java developers and change an ecosystem of millions of Java libraries, I’d expect serverless providers to do better and improve their runtimes. Java is already as light and fast as Go (if not even better), so building serverless with this language is inevitable.

Possible Implications

In review, the combination of a non-deterministic and non-standardized execution model, proprietary runtime, proprietary data formats, proprietary API, and lack of Java support means that all of these issues leak into the application code and influence the way we implement the business logic. This is the ultimate delegation of control from one organization to another. Because serverless as it is today provides build-time artifacts — in the form of SDKs and packaging formats, event formats, and software stacks — and provides the runtime environment, serverless consumers commit to the support policy and the proprietary limitations that the provider imposes. The consumers commit to stick to and keep up with the provider’s language runtime, SDK, upgrades, and deprecations. They commit to these terms by writing functions that have zero interoperability across serverless platforms. If we are using BaaS for everything, and we have just coupled our organization’s business logic, written in functions, to the proprietary execution model, runtime, API, and data formats, there is nowhere else we can ever go. While we might not want to go anywhere else, having the option is important for some.

Coupling and lock-in by themselves are not bad, but the high cost of migration is. The use of AWS AMIs, AWS RDS, popular open-source projects as managed services, and even SQS are examples of consumers who don’t mind being locked in, as migrating to an alternative service or provider is a viable alternative. That is not the same as coupling our business logic to immature serverless technologies and the serverless provider’s characteristics, specifically. Here, the migration effort is a complete rewrite and testing of business logic and glue code, which are particularly costly considering the highly distributed nature of the serverless architecture.

Microservices trades code complexity for operational complexity. Serverless trades control for velocity. Choose a modern architecture, but read the small print. Every architecture choice is a trade-off.

Serverless 1.5

AWS has done an amazing job of bringing serverless to where it is but it would be sad if the current state of serverless is its high point. It would also be sad if it is only AWS that is able to innovate within serverless and define its future. Considering that AWS’s background in the open-source ecosystem is relatively limited, it would be a stretch to expect AWS to standardize the serverless paradigm, affecting the whole industry on such a fundamental level. The AWS business model and market positions are good for identifying market trends and initial closed innovation, but the open-source model is better for generalization, standardization, and non-forceful industry-wide acceptance. I expect the next generation of serverless to be created using the open-source model with a wider collaboration across the industry, which will help its adoption and interoperability. That process has started, and the industry is slowly exploring interoperable and portable alternatives of the proprietary serverless offerings of today.

Let’s discuss some of the industry trends, in no particular order, that I believe will drive and influence the serverless technology of tomorrow.

Uniform Packaging and Execution Model

Containers are established as the industry standard for application packaging and runtime. A containerized application combined with a powerful orchestration engine enables a rich set of workloads, as we have seen earlier. There is no reason for serverless workloads to be an exception, as that would move us back to a mix of packaging formats and execution models. Knative is an open, joint effort from multiple vendors that is challenging the status quo by offering serverless characteristics (scaling to zero, autoscaling based on HTTP requests, subscription, delivery, binding, and management of events) to container-based workloads on Kubernetes. Container-based packaging and Kubernetes-based execution would allow an open execution model that could be standardized across multiple serverless providers. It would enable a richer set of runtimes with better support for Java, custom software stacks, limits, and customization possibilities. 

Some might argue that including the language runtime and the event handler in the function package is not FaaS per the original definition of serverless but that is an implementation detail and this is a much-needed option for the serverless of tomorrow.

Industry-Accepted Event Formats

The serverless architecture is event-driven by definition, and events play a central role. The more event types there are in a serverless environment, the richer the developer experience and the more logic that can be replaced with off-the-shelf services. But that comes with a price as the business logic becomes coupled with event format and structure. While you can read AWS best practices for separating core business logic and the event-handling logic into separate methods, it is far from decoupling. This coupling of the business logic with the serverless platform’s data formats is preventing interoperability. CloudEvents is an effort to create standardized event formats that would operate across all serverless platforms. Apart from being an awesome idea, it has a huge industry interest including that of AWS, which is probably the ultimate validation of its significance and adoption potential.

Portability and Interoperability

Once there is a standard packaging format and standard events, the next level of freedom is the ability to run serverless workloads cross serverless providers on a public or private cloud, on premises, or on the edge, and mix and match all of it into a hybrid as needed. A function should be runnable on multi-cloud, hybrid cloud, any cloud, non-cloud, or mixed, and should only require a few configurations and mapping. In the same way we used to write Java applications to implement abstract interfaces and deploy them to different web containers, I want to be able to write my functions for a non-proprietary API, events, and programming model, and deploy them to any serverless platform, and for it to behave in a predictable and deterministic manner. 

In addition to portability, I want to see interoperability, with functions able to consume events from any platform regardless of where the function is running. Projects such as KEDA let us run custom functions, such as Azure Functions, in response to AWS, Azure, and other event triggers. Projects such as TriggerMesh allow us to deploy AWS Lambda-compatible functions on top of Kubernetes and OpenShift. These are signs that the functions of the future will be portable and interoperable at multiple levels: packaging, execution environment, event formats, event sources, tooling, etc.

Treat Java as First-Class

While serverless workloads are suitable for many use cases, preventing the use of Java, the most popular programming language of enterprises, is a major limitation. Thanks to Substrate VM and frameworks such as Quarkus, Java is already light, fast, cloud native, and serverless friendly. And there are indications that such Java runtimes soon will be available for serverless too, including for AWS Lambda, hopefully.

Containerized workloads with serverless characteristics, function portability and interoperability with standardized events, and ultra-light and fast Java runtimes created for cloud-native and serverless environments are all signals that serverless is about to change. I don’t yet want to label these indicators as “second-generation serverless” but it is not first-generation serverless, so 1.5 feels about right.

I remember when many thought that Cloud Foundry had won the PaaS war, but then Kubernetes happened. Many now claim that AWS Lambda has won the FaaS war. I hope Kubernetes (or something better) proves them wrong.

Serverless Workloads

We saw how microservices architecture improved the deployment cycles for monolithic applications by modeling services around business domains and encapsulating the change within the services. A naive description of serverless would be to represent this as even smaller microservices, where every operation is a function. While that is technically possible, that would be the worst of both architectures, leading to a large number of functions calling each other in a synchronous manner without gaining any benefit from the resulting architecture.

The value of a microservice comes from the fact that it can encapsulate complex business-domain logic and persistence logic behind a series of request/response-style operations with a web-based API (typically REST style). On the other hand, serverless and functions focus on events and triggers. While functions can be placed behind an API gateway and act in request/response style, the API is not intended to be the primary interface: events and triggers are. Serverless applications tend to work best when the application is asynchronous (unidirectional fire-and-forget style, rather than request/response) and connects through queues or other data and event sources. As a result, each function is intended to perform only one action and should avoid directly calling other functions directly, and write its result to an event store. Due to the execution model, functions are short-lived and are supposed to be used with other serverless data sources that are not connection oriented, as is the case with the typical RDBMS, be light in terms of deployment size, and have fast startup. All of the following use cases make serverless more suitable, as it acts as glue code that connects various event-driven systems:

  • on-demand functionality such as batch processing, stream processing, and extract-transform-load (ETL);
  • task scheduling for divisible work performed for a short time, such as batch jobs;
  • event-driven architecture that executes logic in response to data-source changes;
  • handling non-uniform traffic such as inconsistent traffic that doesn’t happen often or traffic with unpredictable load;
  • general-purpose “glue” code in operations;
  • continuous-integration pipelines with on-demand resources for build jobs; and
  • automating operational tasks such as triggering actions or notifying a person on call when an incident happens.

I discussed some of the major innovations that are happening in the serverless world but didn’t describe what they would look like on Kubernetes. Many efforts have tried to bring serverless to Kubernetes, but the project with the widest industry support and best chance of success is Knative. The primary goal of Knative is to offer a focused API with higher-level abstractions for common serverless use cases. It is still a young project (version 0.5 as of writing) and changing quickly. Let’s explore the serverless workloads that Knative currently supports.

Request Serving

A low-friction transition approach from microservices to serverless is to use single-operation functions to handle HTTP requests. We expect a serverless platform to be able to stand up a stateless, scalable function in seconds — and this is what Knative Serving aims to achieve by providing a common toolkit and API framework for serverless workloads. A serverless workload in this context is a single-container, stateless pod, primarily driven by application-level (L7) request traffic.

The Knative Serving project provides primitives that enable:

  • rapid deployment of serverless containers by providing higher-level, opinionated primitives;
  • activation, scaling up and down to zero driven by requests;
  • automatic routing and configuration of low-level primitives; and
  • immutable snapshots of revisions (deployed code and configurations).

All of the above is achievable within certain limitations, such as a single container per pod, single port, no persistence, and several other constraints.


The Knative Eventing project provides the building blocks for creating reliable, scalable, asynchronous-event-driven applications. It aims to create a standard experience around consumption of and creation of events using the CloudEvents standard. The high-level features of Knative Eventing include:

  • extendable and pluggable architecture that allows different implementations of importers (such as GitHub, Kafka, SQS, Apache Camel, etc.) and channel implementations (such as Kafka, Google Pub/Sub, NATS, in memory, etc.);
  • event registry for maintaining a catalogue of event types;
  • declarative API for event orchestration by binding event sources, triggers, and services; and
  • trigger capability that allows subscribing to events from a specific broker and optional filtering before routing the events to downstream Knative services.

These features are interesting, but how do they help cloud-native developers to be more productive on Kubernetes?

Let’s assume we have implemented a function, built it as a container, and tested it thoroughly. It is basically a service with a single operation that accepts CloudEvents over HTTP. Using Knative, we can deploy the container to Kubernetes as a workload with serverless characteristics. For example, using Knative Serving primitives, the container can activate only when there are HTTP requests and scale rapidly if necessary. In addition, the same pod can also be configured to accept CloudEvents from a broker by subscribing to a channel. That pod can also act as a step in a more complex event orchestration flow defined via a Knative Sequence. All of this is possible without modifying the already-built container, using only declarative Knative configurations. Knative will ensure the routing, activation, scalability, reliability, subscription, redelivery, and broker resiliency of the serverless infrastructure. It is not all there yet, but it is getting there.

Custom Workloads

And that is not all. If you have an application with very specific needs that none of the standard workload primitives provides you, Kubernetes has more options available. In such a case, a Custom Controller can add bespoke functionality to the behavior of the cluster by actively monitoring and maintaining a set of Kubernetes resources in a desired state.

At a high level, a controller is an active reconciliation process performing the “Observe -> Analyze -> Act” steps. It monitors objects of interest for the desired state and compares them to the world’s actual state. The process then sends instructions to attempt to change the world’s current state to be more like the desired state.

A more advanced approach for handling custom workloads would be to utilize another brilliant Kubernetes extension mechanism: CustomResourceDefinitions. Combing a Kubernetes Operator with CustomResourceDefinitions can encapsulate the operational knowledge for the specific application needs within an “algorithmic” form. An Operator is a Kubernetes controller that understands Kubernetes and an application domain -- by combining knowledge of both areas, it can automate tasks that usually require a human operator.

Controllers and Operators are turning into the standard mechanism for extending the platform and enabling complex application lifecycles on Kubernetes. And as a result, an ecosystem of controllers for managing the full lifecycle of more sophisticated cloud-native workloads is forming out.

The Operator pattern allows us to extend the Controller pattern for more flexibility and greater expressiveness. All of the workload types discussed in this article, and other related patterns are covered in the Kubernetes Patterns book I recently co-authored. Check it out for further details on these topics.

Cloud-native Trends

The tendency of moving more and more commodity features that are not part of the business logic to the platform layer continues in the Kubernetes ecosystem: 

  • Deployment, placement, health checks, recovery, scaling, service discovery, and configuration management have all moved to the Kubernetes layer.
  • Service meshes continue this trend by moving the network-related responsibilities such as resilient communication, tracing, monitoring, transport-level security, and traffic management to the platform.
  • Knative adds specialized serverless primitives, and moves the responsibility of rapid scaling up, scaling to zero, routing, eventing infrastructure abstractions, event publishing, subscription mechanism, and workflow composition to the platform as well.

This leaves mainly business-logic concerns for the application developers to implement. The platform takes care of the rest.

Responsibilities move to the platform. More abstractions enable diverse workloads

We shift more and more commodity responsibilities from the application layer into the platform by adding higher-level abstractions to Kubernetes. For example, Istio provides higher-level networking abstractions that depend on lower-level Kubernetes primitives. Knative adds higher-level serverless abstractions that depend on the lower-level abstractions from Kubernetes and Istio (note that this is about to change, and Knative will not depend on Istio going forward, although it will need to implement a similar functionality).

These additional abstractions enable Kuberntetes to uniformly support a variety of workloads, including serverless, with the same open packaging and runtime format.

Runtimes and Application Design

With the transition from monolithic architectures to microservices and serverless, the runtimes are evolving as well; so much so that the two-decade-old Java runtime is moving away from its “write once, run anywhere” mantra to become native, light, fast, and serverless first.
Java runtime and application design trends

Over the years, Moore’s Law and increasing compute power guided Java in building one of the most advanced runtimes, with an advanced garbage collector, JIT compiler, and many other things. The end of Moore’s law led Java to introduce non-blocking primitives and libraries that benefit from multiprocessor and multi-core systems. In the same spirit, with platform trends such as cloud native and serverless, distributed system components are becoming light, fast, and optimized for a single task.


The serverless paradigm is recognized to be the next fundamental architectural evolution. But the current generation of serverless has emerged from among early adopters and disruptors, whose priority is velocity. That is not the priority of enterprise companies that have complex business and technological constraints. Among these organizations, the industry standard is container orchestration. Kubernetes is trying to link cloud native and serverless though the Knative initiative. Other projects such as CloudEvents and Substrate VM are also influencing the serverless ecosystem and pushing it to an era of openness, portability, interoperability, hybrid cloud, and global adoption.


On Thursday, October 10, 2019, I will be presenting “Designing Cloud Native Applications with Kubernetes Patterns” at a virtual event organized by Red Hat. If you found this topic and the content interesting, check the agenda, register, and listen to me live.

About the Author

Bilgin Ibryam is a principal architect at Red Hat, and a committer to multiple Apache Software Foundation projects. He is a regular blogger, open-source evangelist, blockchain enthusiast, and speaker. He has written the books Camel Design Patterns and Kubernetes Patterns. He has over a decade of experience in designing and building highly scalable and resilient distributed systems. In his day-to-day job, Ibryam enjoys guiding teams in enterprise companies to build successful open-source solutions at scale through proven and repeatable patterns and practices. His current interests include enterprise blockchains and cloud-native and serverless paradigms. Follow him @bibryam for regular updates on these topics.


Rate this Article