AWS makes flat network design default for new data centres

Amazon Web Services has made a quasi-random flat network architecture the default for most new data centres globally. The system is already in use in AWS facilities.

Known as RNG, for resilient network graphs, the design replaces the conventional fat-tree approach long used in large data-centre networks. AWS says the architecture uses 69% fewer routers than a traditional fat-tree network, delivers up to 33% better throughput and is projected to cut electricity use from network equipment by 40%.

Data-centre networks typically rely on layered router structures that pass traffic up through a hierarchy and then back down to its destination. AWS says that model is straightforward to implement, but it adds overhead, can concentrate congestion in upper layers and exposes larger parts of the network if a single router fails.

By contrast, a flat network links routers more directly. Researchers have long argued that random topologies can improve resilience and performance by creating many possible paths between points in the network, but those designs have been difficult to route efficiently and to cable in physical facilities.

AWS says it addressed those obstacles with a quasi-random topology and a passive optical device called a ShuffleBox. The device links local routers to other ShuffleBoxes in a pattern that gives the overall network many of the properties of a random graph while keeping installation complexity similar to that of a fat-tree design.

Under that arrangement, technicians can connect a new rack to a local ShuffleBox without rewiring the wider network. The number of cable runs and the installation process remain broadly comparable with conventional systems, even though the network's logical layout is different.

Routing changes

AWS also developed a routing method called Spraypoint to work with the flatter network design. According to the company, standard multipath routing methods for flat topologies usually require far more memory than commodity routers can provide.

Spraypoint sends traffic first to a random neighbouring router, then uses waypoints and shortest-path methods to guide packets to their destination. AWS says that approach nearly doubles the number of independent paths between routers compared with standard shortest-path techniques, improving the chances that traffic can avoid congestion or failed hardware.

AWS also built mathematical models to predict performance before construction. Those models estimate factors including path length, route counts and likely traffic loads on individual links, giving operators a way to choose design parameters before committing to a build.

The company tested those models through simulations equivalent to 530 processor-years on Amazon EC2. The goal was to let planners calculate the lowest-cost topology that would still meet server count and performance targets.

Production rollout

The first production deployment of the quasi-random network went live near Dublin and carried live traffic before the wider rollout. AWS compared real-world performance with its mathematical forecasts, refined the design and then expanded it to two further deployments.

In production benchmarking, AWS says the flat topology matched fat-tree performance for multipath transport workloads and latency-sensitive storage operations. Customer applications did not need to change because the network sits beneath existing software stacks.

The shift has implications for the economics of data-centre construction as operators seek to add computing capacity while limiting power consumption and hardware costs. Reducing the number of routers can lower not only equipment spending but also related cooling and operational demands across a site.

The move also reflects a broader effort within hyperscale infrastructure to challenge established networking assumptions. Fat-tree designs became common because they are predictable and easier to model, but AWS argues that advances in routing and physical interconnects now make flatter structures practical at industrial scale.

Amazon credited the work to Giacomo Bernardi, Principal Applied Scientist in AWS Core Networking; Ratul Mahajan, Professor at the University of Washington and Amazon Scholar with the AWS Network Engineering group; and Seshadhri Comandur, Professor of Computer Science at the University of California, Santa Cruz and Amazon Scholar with AWS Core Networking.

"The resulting network design - which we call RNG, for resilient network graphs - is now used in AWS data centers and is the default for most new builds globally," Bernardi, Mahajan and Comandur said.