Meta의 Prometheus는 대규모 AI 학습 작업에 오브젝트 스토리지를 사용하고 있는 것으로 전해졌다.

Meta has begun discussing its Prometheus gigawatt-scale AI training cluster. Its storage system has migrated from a file-based design to multi-tiered, cached object storage, which reduces GPU idle time.

Prometheus is a gigawatt-scale GPU supercluster in New Albany, Ohio, used to build Meta's Muse family of foundation AI models. It is being scaled from its current hundreds of thousands of GPUs to around 1 million GPUs working in lockstep to train the next frontier models of Meta Supercluster Labs. Today the cluster is an array of hundreds of thousands of GPUs spread across multiple interconnected buildings.

Prometheus uses two Nvidia GPU platforms:

- **Catalina** uses the Ariel variant of Nvidia's GB200 NVL72 (Blackwell). It is a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain.

- **Clemente** uses Nvidia's GB300 NVL72 (Blackwell Ultra) and improves on Catalina at both the GPU and platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and increases dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. A full 72-GPU domain is deployed in a single rack.

A third platform, San Miguel, is due. It is based on Vera Rubin NVL72 GPUs, will deliver a generational leap in per-GPU performance, and will integrate directly into Prometheus's existing network fabric. As the cluster matures, augmenting its primarily interconnected GB300 capacity with Vera Rubin will deliver even more compute per watt.

Meta is building a second supercluster, called Hyperion, in Richmond Parish, Louisiana.

**Network**

Several Prometheus datacenter buildings are located at New Albany. Inside each building, an Ethernet-based fabric is used. The cluster buildings are interconnected by a multi-layer, hub-and-spoke Backend Aggregation (BAG) network spanning distances from 2 km to over 100 km. The network uses Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps of bandwidth per region pair.

**Prometheus Storage Scheme Outline**

The storage system is a custom, multi-tier stack built on Meta's Tectonic foundational layer. It provides BLOB (Binary Large Object) storage interfaces and uses specialized hardware platforms for bulk and high-performance needs. The subsystem has a four-layer hardware stack:

- **L1/L2:** Memory and local flash on GPU hosts.

- **L3:** Regional flash-backed BLOB storage.

- **L4:** Source of truth, consisting of durable HDD-backed BLOB/Tectonic storage (global or regional).

The design also includes distributed caching, which leverages spare GPU-host memory and reuses components from Meta's Owl system; metadata caching; prefetching in the data loader; hedged reads; and dynamic concurrency control. The design prioritizes predictable low tail latencies so that training steps are not stalled by storage.

**Owl**

Owl is Meta's production system for moving large volumes of hot internal content efficiently by combining peer-to-peer data transfer with intelligent centralized control. It is a high-fanout content distribution system, scaling from a few clients to millions of processes, for large, frequently accessed ("hot") objects inside Meta's private cloud. It delivers executables and binaries, code artifacts, AI models, and search indexes to millions of hosts. The delivered content can range from a few GB to several TB in size. Owl has separate data and control planes:

- **Decentralized data plane:** Ephemeral peer-to-peer distribution trees. Data streams from sources through peers as soon as chunks arrive, avoiding full store-and-forward delays. Trees are short-lived and per-chunk.

- **Centralized control plane:** Tracker services maintain detailed, up-to-date metadata on peers, their cache contents, locations, and ongoing downloads. Trackers decide sources, retry behavior, caching and eviction, and tree construction.

**Prometheus Storage Software Stack**

The Prometheus storage stack has moved away from the design used for Llama training, which exposed an NFS-like file system interface on top of Tectonic (see bootnote). That architecture remains widely used within Meta, but Prometheus is moving to BLOB-based storage. According to Meta, this "is motivated by the need for unified storage access to massive data lakes in the BLOB-storage layer as well as the need for high performance," with the goal of maximizing GPU utilization.

The storage software stack has two main layers:

- **Tectonic:** The foundational, regional, multi-tenant block storage fabric. It provides high durability and availability through erasure coding, media tiering (HDD and flash), and smart placement of hot, warm, and cold data.

- **BLOB-storage layers on top of Tectonic:** These expose object-storage APIs and, historically, file-system APIs.

The stack features a unified flat metadata schema backed by ZippyDB, which enables fast O(1) path-to-address lookups. It eliminates a data-plane proxy in favor of a "fat client" SDK that streams data directly from storage servers. Regional deployments are colocated with GPUs, and Meta now deploys a regional BLOB-storage stack colocated with GPUs in every AI region.

The read path works as follows. When the client SDK receives an API call, it issues a getReadPlan("/bucket/path") request to the API server. The API server performs an O(1) lookup per chunk in the new metadata store to map the path to (blockId, offset, size) tuples, then returns a ReadPlanResult to the SDK. The SDK has the Tectonic BlockClient embedded within it, so it can stream data from these blocks directly from Tectonic.

Two caches are used:

- **Distributed data cache:** Prometheus leverages spare memory on the GPU hosts as a distributed data cache for frequently and concurrently accessed data. It reuses components from Meta's Owl subsystem by integrating Owl peers directly into the BLOB-storage client SDK, so that all data access goes through this cache.

- **Readplan metadata cache:** A readplan is the mapping from path to storage address. This cache stores the read plans for frequently accessed BLOBs in a distributed-memory store similar to memcache.

**Prometheus Tiered Caching**

Meta says the core idea is a dataloading scheme that uses the memory and flash on the GPU host as L1 and L2 caches, with a regional BLOB-storage fabric backed by flash serving as the L3 cache. The global, HDD-based BLOB-storage data lake is the backing store, which Meta calls "the ultimate source of truth." Dataloaders access the caches and this backing-store system through the BLOB-storage SDK.

Several mechanisms support this scheme:

- **Dataloader prefetch:** Dataloaders prefetch the next batch of datasets into memory while processing the current batch. This prefetch surfaces as a read operation at the BLOB-storage SDK level.

- **Deep prefetch:** The BLOB-storage SDK exposes an explicit prefetch() API. The dataloader invokes it in the background to trigger explicit prefetch of the data needed over the next few minutes. This hydrates data from remote storage onto the local regional L3 cache and prewarms the metadata cache.

- **Automatic data lifecycle:** Data in the L3 regional disaggregated flash tier is typically held for a configured period so it can be reused across epochs of a training cycle. Custom eviction policies, including TTL and LRU, are supported. The eviction policies are also capacity- and quota-aware.

After Meta adopted this dataloading scheme, average job ingestion time dropped from 150 minutes to 10 minutes. The maximum ingestion time for very large jobs fell from 89 hours to just over 3 hours, or 182 minutes.

**Storage Hardware**

The storage hardware includes:

- **Olympic and Yosemite V3.5 Crater Lake:** Flash- and SSD-based platforms optimized for lower-latency, higher-throughput access.

- **Grand Canyon:** An HDD-based system for bulk training-data storage.

- **General-purpose compute platforms** (for example, Yosemite variants) for cluster management.

These components allow capacity, latency, and cost to be optimized independently across the data pipeline.

Yosemite V3.5 has a blade and sled chassis design, with up to a dozen 10 RU servers in a 40 RU rack. It supports various compute, memory/CXL, PCIe, accelerator, and SSD expansions. It uses OCP-aligned datacenter NVMe SSDs (E1.S for capacity and performance storage, and M.2 for boot), not commodity consumer drives. These are purpose-built or qualified datacenter NVMe devices.

Crater Lake is a server designed for the Yosemite V3.5 architecture. It supports 4th Gen Intel Xeon (Sapphire Rapids / Eagle Stream) processors and has a single-socket, 1OU (1 Open Unit) design intended as a base compute node. The Open Unit is specific to the Open Compute Project's Open Rack Standard. At 48 mm (about 1.89 inches, or 1⅞ inches) high, it differs from the traditional EIA 1U rack unit of 1.75 inches.

Grand Canyon is a 4OU-tall, single-drawer, dense chassis that holds up to 72 × 3.5-inch HDDs. It has three build options:

- **Dual storage server:** Two compute nodes and 72 HDDs, with all drives powered for higher performance and access speed. It is fully standalone.

- **Single storage server:** One compute node acting as a head node, which can connect to additional JBOD chassis.

- **Pure JBOD:** No compute. It is used as a disk shelf. In head-node-plus-JBOD setups, this can scale to 216 drives or more in multi-chassis configurations.

Like Yosemite and Crater Lake, Grand Canyon is a design Meta has contributed to the OCP.

**Checkpointing**

Prometheus uses a distributed checkpointing architecture that asynchronously writes model state, including weights, gradients, and optimizer data, to multi-tier storage. It employs delta checkpointing and in-memory staging to minimize the impact on training throughput. This fits the overall tiered model of fast local or in-memory staging, then flash, then durable HDD-backed storage, and supports high goodput (for example, above 85% on pre-training runs).

**Takeaway**

Meta developed this specialized custom storage scheme to minimize GPU waits in very large training clusters. Organizations running similar AI training workloads at hyperscaler or near-hyperscaler scale could write the software themselves. Smaller organizations, or those needing standalone object storage, could instead use software from Cloudian, MinIO, or Scality to provide the functionality they require. OCP-class NVMe SSDs should be available from OCP suppliers, and commodity disk drives are readily available as well.

It will be interesting to see whether Meta uses its Prometheus clusters for AI inferencing and agentic AI workloads. Those workloads, we suppose, won't have such large job ingestion times, and individual inference jobs won't require huge numbers of GPUs working in lockstep.

**Bootnotes**

In 2024, Meta said Hammerspace was its data orchestration software supplier. Hammerspace supported 49,152 Nvidia H100 GPUs split into two equal clusters, used in two Llama 3 AI training systems. Meta's Grand Teton OCP hardware chassis houses the GPUs, which rely on Meta's Tectonic distributed, flash-optimized, exabyte-scale storage system. This storage was accessed through a Meta-developed Linux Filesystem in Userspace (FUSE) API and used for AI model data needs and model checkpointing.

Both the Tectonic and Hammerspace-backed storage deployments use Meta's YV3 Sierra Point server, fitted with the highest-capacity E1.S format SSDs available.

A previous AI Research SuperCluster, with 16,000 Nvidia A100 GPUs, was used to build Meta's first-generation AI models. It "continues to play an important role in the development of Llama and Llama 2, as well as advanced AI models for applications ranging from computer vision, NLP, and speech recognition, to image generation, and even coding." That cluster used Pure Storage FlashArray and FlashBlade all-flash arrays.

The Meta BLOB-storage SDK is the client-side library that lets applications, such as AI training jobs, efficiently read and manage BLOBs from Meta's global storage fabric, with direct streaming.