With the help of AWS and HashiCorp consultants, Nomura created a solution for service discovery for complex microservices environments. The solution leverages HashiCorp Consul and is based on a hierarchical, rule-based algorithm. It supports discovery by service name, DNS latency, and custom tags.
Nomura needed a flexible service discovery solution to support its microservices and cloud adoption journey. Tom Palmer, head of e-trading strategy FX at Nomura, highlights challenges associated with service discovery in heterogeneous deployment environments:
These days, large organizations typically have microservices environments that span across cloud platforms, on-premises data centers, and colocation facilities. [...] Due to the complex nature of these environments, efficient mechanisms for service discovery and configuration management must be implemented to support operations at scale.
The solution involves deploying a HashiCorp Consul cluster spanning multiple networks with the aid of AWS Transit Gateway that can connect AWS VPC and on-premise networks through a central hub that acts as a highly scalable cloud router. AWS Transit Gateway allows sharing VPCs, Domain Name System (DNS), Microsoft Active Directory and IPS/IDS across regions with inter-region peering.
AWS Transit Gateway Overview (Source: AWS Documentation)
With wide-area network (WAN) topology, Consul clusters gain visibility of microservice instances deployed in distinct data centers, both on-premise and in the cloud. Microservices running on EC2 use the Spring Cloud Consult plugin to interact with the Consul cluster and fetch instance metadata from the EC2 instance metadata service using the custom configuration provider.
When the microservice starts, it retrieves tags and other metadata from the EC2 metadata service and registers itself with the Consul cluster. Once registered, the service can query for other services and be discovered by other services. The solution supports rule-based discovery based on tags and explicit precedence definitions. This allows discovering fallback instances in case the primary instance is not available. Rules can also be used to enforce boundaries between environments and data center locations and implement custom discovery scopes.
Service Discovery Flow (Source: AWS Architecture Blog)
In a typical service discovery flow, the Consul Agent instance registers itself with the local Consul cluster, allowing the service itself to interact with Consul. The service can then register itself with the agent and query for other services via API calls. Once service discovery data is returned with the list of available service instances based on tags and configured rules, the service can make direct calls to one or many discovered services.
AWS and HashiCorp put together a demo project containing all components and deployment steps. The project uses HashiCorp Terraform and Packer to package and provision necessary infrastructure and AWS CLI and AWS Systems Manager to gain access to the Consul cluster and interact with the Consul UI.