OpenAI scales storage infrastructure to support over 1 billion ChatGPT users

Every OpenAI product depends on fast and reliable access to data, whether someone is logging in, checking Codex settings, or starting a new conversation in ChatGPT. Each of these actions may require many separate data lookups before the product can respond. If those requests slow down, the product feels slow. If those requests fail, the product stops working altogether.

Habitat is the online storage platform we built so OpenAI products can access needed information quickly and reliably. Habitat now handles more than 70 million requests per second and supports products used by over a billion people per week, across nearly 40 geographic regions. Habitat was first launched to support GPTs at DevDay 2023, as a simple Python library on the client side connected to a single database. Today it is a complex distributed system serving more than 500 petabytes of data.

Building and operating infrastructure at this scale is not easy, but it is not particularly difficult either. What made our situation unique was the unprecedented pace at which we had to scale to support enormous user growth and product demand, while building a mature platform. Often, systems engineers build for a scale 10 times larger and expect to serve for a few years while preparing for 10x growth. In our case, over the past three years we grew more than 10x year after year. Therefore, building and operating Habitat was a series of tactical decisions and sequencing: we understood each component down to the lowest level to get the most out of the existing stack, while avoiding storage and compute capacity shortfalls so we could buy time for fundamental investments.

As OpenAI grew, Habitat had to grow with it: first it became reliable enough for critical product traffic, then fast enough for global users, and finally capable of operating skillfully at enormous scale. This publication is the first in a two-part series about how we scaled online storage. Here we'll share how Habitat evolved, why we transformed it from a library into a service, and how we extended a service written in an uncommon language for the serving stack—Python—to become a reliable layer of storage platform.

In a future publication we'll explain in detail how we achieved multi-tenant reliability at large scale, our layered strategy for read optimization, and how we scaled our partnership with Azure Cosmos DB to reliably meet unprecedented demand.

Habitat started with a simple idea: product engineers shouldn't have to think about managing databases. Habitat was first launched to support GPTs at DevDay 2023 as a small Python library that communicated with ChatGPT's main server. It supported a small set of operations that internally mapped to database application operations on Azure Cosmos DB.

The library's job was to give product teams a simple way to store and retrieve data without having to learn the underlying details. Habitat handled the work required: determining what kind of data was involved, where it should come from or go, whether the request was permitted, and so on.

Product engineers didn't have to worry about schema lookups, routing, authorization, encryption, serialization, request signing, and connection pooling. They also didn't have to think about where the data came from: Azure Cosmos DB, caches, or other storage types.

By separating storage logic into an independent service, we created a single point of control for deployments, observability, and platform improvements.

This Python library worked well and Habitat was quickly adopted by product engineers at OpenAI, despite there being no centralized initiative to encourage self-service use of Postgres and Azure Cosmos DB.

As product needs evolved, it was also easy for developers to add support to the shared library for features like client-side caching, compression, or encryption.

By mid-2025, Habitat had reached the limits of its implementation as a client-side library. As the Habitat layer became more complex and the number of OpenAI services grew, backward-compatible changes to the protocol were no longer feasible.

In one case, we wanted to reduce the impact of a regional outage on the most critical datasets by migrating them to a group of regionally distributed Azure Cosmos DB accounts. This change required additional routing logic on the client, disabled behind a feature flag, its rollout to all clients, and then activation of the feature flag.

Coordinating deployments across dozens of services and working with each team to roll it out took entire weeks. Before we activated it, we realized we needed to introduce shadowing to ensure the sharding logic was correct. To do that we needed two more days. A solution for a bug in something we realized was broken? Two more days. Eventually we were ready to activate the flag, but one of their teams rolled back its service for reasons unrelated to client issues with a previous bug, which caused exactly the outage in the service we had worked so hard to prevent.

Changes to the client library required complex coordination across dozens of services, a process that became increasingly fragile, inefficient, and susceptible to operational failures. To reduce this operational burden in future deployments, we decided to separate Habitat into its own service.

By separating storage logic into an independent service, we created a single point of control for deployments, observability, and platform improvements. Instead of managing fragmented updates, we could implement improvements centrally and give immediate benefits to every OpenAI product.

A centralized service also gives us a single point of congestion where we provide the strongest elements for data security and privacy. In the Habitat service we can centrally enforce access control policies, audit logs, and limit access to underlying storage resources like Azure Cosmos DB. Habitat plays a critical role in protecting user data and preventing unauthorized access from external actors, internal threats, and agents.

We knew we needed a service, but for now we didn't want to leave Python, despite its additional costs as a service. Using Python for a high-throughput service increased network latency and the costs of CPU and memory scaling compared to running the library locally. Moreover, we understood that Python's inefficiencies would not be acceptable at 100x scale and so a rewrite seemed almost certain.

However, we viewed this as a strategically accepted technical debt. Our primary goal at that time was not to optimize costs or resources, but to let product developers move forward and achieve stability in the platform. By temporarily accepting the performance tradeoffs of a Python service, we could prioritize more immediate challenges, establish core APIs, and build robust infrastructure.

We also made a calculated bet that the rapid progress of our code models would simplify the technical path forward. We believed that when a full migration from Python became necessary, Codex and GPT would make it possible. In the end, that bet turned out to be correct.

Running Habitat as a Python service was not ideal for performance, but it was a necessary choice. Python let us move quickly, but that didn't mean we could ignore risks and accept much worse latency. When an average user request leads to hundreds of database calls, the slowest database call is the one the user will feel. We found that the main challenge in running a Python service at this scale is managing extreme latencies.

Asyncio helps Python execute I/O-bound workloads concurrently, but it doesn't overcome Python's GIL and doesn't provide CPU parallelism. Beyond proxying requests with many...