Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News HashiCorp Releases Nomad 0.9 with Additional Scheduling Features

HashiCorp Releases Nomad 0.9 with Additional Scheduling Features

HashiCorp has released version 0.9 of Nomad, their distributed scheduler platform. This release includes enhancements to the scheduling features that determine how Nomad places applications across the infrastructure. The other major release is the groundwork for a plugin-based feature strategy to enable easier integrations with a number of technologies.

The enhancements to the scheduling features allow for additional methodologies for instructing Nomad on how to handle job placement decisions. By default, Nomad uses bin packing when making decision on job placement. Bin packing attempts to maximize the usage of resources on a set of client nodes before moving onto additional nodes. This has the advantage of typically smaller fleets and better cost savings.

With this release, a new spread stanza has been added to permit specifying a distribution for jobs based on a specific attribute or client metadata. This includes attributes such as datacenter, availability zone, or physical datacenter racks. The spread stanza allows for specification of a distribution ratio across a set of target values for a specified attribute:

spread {

  attribute = “${node.dc}”
  weight = 20

  target “us-east1” {
    percent = 50

  target “us-east2” {
    percent = 30

  target “us-west1” {
    percent = 20

In the above example, the spread stanza will tell the Nomad scheduler to attempt to distribute 50% of the workload into us-east1, 30% into us-east2, and the remaining 20% into us-west1. As the spread scores are combined with other scoring factors, such as bin packing, the end result may not align exactly to the defined stanza. Nomad treats spread criteria as a soft preference, meaning even if no nodes match the given criteria the placement will still be successful. A job or task can have multiple spread criteria with a weight parameter used to describe the relative preference between them.

Similar to the new spread stanza, an affinity stanza has been added that permits expressing a preference for a given workload based on the runtime state of the environment. This new stanza allows for assigning a preference for job placement based on any node property that is available to Nomad clients. This is in contrast to the constraint stanza which outright restricts the set of eligible nodes.

job "docs" {

  # Prefer nodes in the us-west1 datacenter
  affinity {
    attribute = "${node.datacenter}"
    value = "us-west1"
    weight = 100

  group "example" {

    # Prefer the "r1" rack
    affinity {
      attribute = "${meta.rack}"
      value = "r1"
      weight = 50

    task "server" {

      # Prefer nodes where "my_custom_value" is greater than 5

      affinity {
        attribute = "${meta.my_custom_value}"
        operator = ">"
        value = "3"
        weight = 50

In this example there is a strong affinity at the job level for the us-west1 datacenter. At the group level it will prefer the r1 rack while at the task level a preference for nodes where my_custom_value is greater than five is expressed.

Affinity scores can be negative, known as an anti-affinity, and will express a preference to not use that node based on the provided attribute. As with the spread stanza, the weight property can be used to assign relative priority over a series of stanzas.

This release also includes a new feature allowing for system jobs to displace lower priority workloads. Preemption allows for Nomad to kill allocations in order to place higher priority jobs when resources are limited. In version 0.9 support for system jobs was added, and support for service and batch jobs is scheduled for a future Enterprise release.

The Nomad client was refactored in this release to enable plugin-based support for Task Driver Plugins and Device Plugins. The task driver subsystem allows support for a range of workload types and provides drivers for tasks like deployment automation and automatic secret injection. With this release, external plugin support was added, allowing for external contributions by the community. The LXC task driver plugin was externalized and provides an example of how the new plugin system works.

Also included in this release is a new device plugin system that allows support for specialized hardware devices like GPUs, TPUs, and FPGAs. This new plugin system enables the Nomad client to detect a device, determine its capabilities, and make it available to the scheduler. The Nomad 0.9 release includes a native Nvidia CPU device plugin to allow support for these devices.

For more details and additional features included in this release, please review the official announcement on the HashiCorp blog. The CHANGELOG provides the full list of changes in the release. The open source version of Nomad is available free for download.

Rate this Article