Several hundred thousand GPUs spread across several buildings, a network stretching over 100 km and a target of around one million GPUs: Meta offers a behind-the-scenes look at Prometheus.
Meta has published a technical presentation on Prometheus, its gigawatt-scale supercomputer based in New Albany, Ohio. Two years ago, this site housed a conventional data centre dedicated to web traffic; it has since been repurposed for training state-of-the-art models. The first phase is now complete and was used to train the Muse family of models, including Muse Wonder. Meta’s long-term goal is to have around one million GPUs, spread across conventional buildings, AI-optimised buildings and rapid-deployment facilities, all connected to function as a single machine.
Reliability above all
On this scale, a failure on a single server can slow down or halt an entire training run. Meta reports a ‘goodput’ of over 85 per cent for pre-training and 90 per cent for reinforcement learning with Muse Wonder, with the mean time between interruptions increasing from around 47,000 to over 54,000 GPU-hours thanks to firmware patches. On Nvidia’s NVL72 racks, each cluster of 72 GPUs uses only 64 of them, with the remaining eight serving as immediate spares; approximately 2 per cent of additional racks are kept in reserve.
On the hardware side, Meta has deployed two platforms, Catalina (GB200 NVL72) and then Clemente (GB300 NVL72), both featuring closed-loop liquid cooling. The networks in each building, capable of connecting up to 20,736 GPUs, are interconnected by an aggregation network spanning from 2 to over 100 km.
Read the full article on the original source
Go to the source →