Cloudflare recently detailed how it manages its servers' egress traffic using "soft-unicast". Soft-unicast allows multiple servers to share a single IPv4 address for their egress traffic while redirecting the response packets to the correct physical server. It provides a scalable, cost-efficient solution for Cloudflare to offer various products that require tagged egress IP addresses.
Cloudflare chose to split an egress IP across servers by a port range. Each server owns a small portion of available source ports for a given egress IP - a port slice. With a port slice of 2048 ports, Cloudflare can share one IP among 31 servers. It implemented methods to reuse the egress ports efficiently to address the possibility of running out of ports. To route return packets to the correct machine, Cloudflare customised "Unimog", its L4 XDP-based load balancer, to be aware of this technique.
Source: https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-anymore/
Source-NAT on a router can traditionally share an IP address between servers. However, the number of egress IPs Cloudflare needs prevents it from relying on stateful firewall / NAT at the router level. Cloudflare also chose to avoid a distributed NAT installation for undisclosed reasons.
Typically, it's only possible to route subnets with a granularity of /24 or 256 IP addresses on the public Internet. For Cloudflare, this would lead to a waste of IP space. To improve the utilisation of Cloudflare's IP space, it deployed the egress IP addresses as anycast addresses, which it typically uses for ingress traffic. Cloudflare customised Unimog and taught it to forward the packets over its backbone network to the right data centre.
Source: https://blog.cloudflare.com/cloudflare-servers-dont-own-ips-anymore/
With this design, an IP address semantically identifies a data centre, and an IP and port range identifies a specific machine. It behaves almost like a unicast.
Wikipedia defines anycast as follows:
Anycast is a network addressing and routing methodology in which a single destination IP address is shared by devices (generally servers) in multiple locations. Routers direct packets addressed to this destination to the location nearest the sender, using their normal decision-making algorithms, typically the lowest number of BGP network hops. Anycast routing is widely used by content delivery networks, such as web and DNS hosts, to bring their content closer to end users."
Previously, Cloudflare used anycast only for its ingress traffic. By customising its L4 load balancer, it was able to utilise it for egress traffic as well, taking care of the "last-mile" route over its backbone network.