OpenAI is scaling its online storage infrastructure to serve over 1 billion ChatGPT users, reflecting massive growth in platform load and user-generated data.
OpenAI products depend on fast, reliable access to data—whether a user is logging in, adjusting their Codex settings, or starting a new ChatGPT conversation. Each action may trigger dozens of database lookups before the product responds. Slow requests make products feel slow; failed requests stop them entirely.
Habitat is the online storage platform built to provide OpenAI products with reliable, high-speed data access. The platform now handles over 70 million requests per second, supporting products used by more than 1 billion people weekly across nearly 40 geographic regions. What started two years ago as a simple Python client-side library connected to a single database has evolved into a complex distributed system serving over 500 petabytes of data.
Building infrastructure at this scale presents ordinary challenges, but OpenAI's situation is unique: the company has scaled more than tenfold year-over-year for the last three years while simultaneously building a mature platform. Most system engineers design for 10x growth and hope to hold for years. Operating Habitat required a series of tactical decisions and sequencing—extracting maximum efficiency from the existing stack while managing storage and compute constraints to buy time for foundational investments.
Habitat evolved through distinct phases: first becoming reliable enough for mission-critical product traffic, then fast enough for global users, and finally operating reliably at massive scale. This progression required moving Habitat from a library into a service.
Initially, in mid-2024, Habitat was a small Python library interfacing with ChatGPT's main server. It supported a limited set of operations mapped to the underlying database application, Azure Cosmos DB. The library insulated product engineers from database management details, handling schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling. Developers didn't need to know whether data came from Azure Cosmos DB, caches, or other storage types.
The library gained rapid adoption despite no formal organizational push away from self-serve Postgres and Azure Cosmos DB. As product needs evolved, the shared library easily accommodated new features like client-side caching, compression, and encryption.
By mid-2025, Habitat had outgrown the client-side model. With the layer growing more complex and OpenAI's service count increasing, backward-compatible protocol changes became infeasible. One example: OpenAI wanted to reduce blast radius from regional outages for critical datasets by migrating to regionally distributed Azure Cosmos DB accounts. Implementing this required introducing routing logic into the client behind a feature flag, coordinating rollout across dozens of services—a process taking days. Additional shadowing work, bug fixes, and re-coordinations extended the timeline further. Eventually, when one team rolled back their service for unrelated reasons to a previously buggy client version, it triggered the very outage the coordinated effort had tried to prevent.
The operational fragmentation made client library changes increasingly brittle and failure-prone. To reduce this distributed complexity, OpenAI converted Habitat into a centralized service. This established a single point of control for deployments, observability, and platform enhancements—replacing fragmented updates with centralized improvements benefiting every product immediately. The service also became a critical security chokepoint, centrally enforcing access control policies, audit logging, and limiting access to underlying storage resources while protecting user data from external, internal, and agent-based threats.
Running a Python service at scale presents inherent trade-offs. Python's overhead as a service increases network latency and adds substantial CPU and memory scaling costs compared to local library execution. OpenAI recognized that Python's inefficiencies would not scale to 100x demand, making an eventual rewrite inevitable. However, they accepted this as strategic technical debt: the priority was unblocking product developers and achieving platform stability over near-term cost optimization. Additionally, they wagered that rapid advancement in their own coding models would eventually simplify migration away from Python—a bet that proved correct.
Performance trade-offs were unavoidable, but meaningful latency degradation was not acceptable. When an average user request triggers hundreds of database calls, the slowest call determines perceived speed. The central challenge in running Python at this scale is managing tail latencies.
Python's asyncio handles I/O-bound concurrency but cannot work around the Global Interpreter Lock or provide CPU parallelism. Beyond I/O-heavy request proxying, Habitat handles CPU-intensive responsibilities: routing, compression, encryption, checksumming, downstream health checking, request shadowing, and hedging. With so many CPU-heavy workloads and background tasks, asyncio scheduling delay easily dominates tail request latency. Initial traces showed that for requests at p99 and higher latencies, downstream storage responded quickly but requests frequently stalled waiting for the responsible coroutine to be rescheduled to parse the response.