IBM has released llm-d 1.0, an open-source AI inference stack aimed at cost-efficient production-scale serving.

SAN JOSE, Calif., Oct. 20, 2026 – Today, llm-d project contributors announced the release of llm-d 1.0, a major milestone for the open source project designed to help organizations deploy and scale large language model inference more efficiently across modern infrastructure. Founded in 2025 by Red Hat, IBM (NYSE: IBM), Google Cloud, CoreWeave and NVIDIA, and now governed by the Cloud Native Computing Foundation (CNCF), llm-d plays a key role in the open foundation for enterprises deploying AI at scale in production.

As organizations race to put AI into production, many are discovering that running AI at scale can be as challenging and expensive as building the models themselves. Organizations need a way to deliver responsive experiences for users, support agentic workloads, and maximize utilization of increasingly scarce compute resources. llm-d 1.0 is built to address these challenges through intelligent distributed inference capabilities that improve performance, increase throughput and help organizations gain greater control over the economics of AI serving.

**Turning Innovation into Production Reality**

With the rise of agentic AI, new types of workloads are placing a different kind of stress on infrastructure systems. Agentic workloads continually access and reuse vast amounts of context while supporting parallel agents that can generate unpredictable bursts of activity. Managing that context often consumes more resources than generating the final output itself, making latency, throughput, memory efficiency and infrastructure costs critical factors in realizing AI ROI. llm-d already delivers hardware heterogeneity and advanced inference capabilities out of the box for complex infrastructure environments, including prefix-cache and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism, and batch and async inference, across multiple model servers, including vLLM and SGLang.

llm-d 1.0 is engineered to bring that performance to the realities of Day 2 operations and the ongoing work of running inference in production: serving multiple tenants, balancing competing priorities, providing zero-downtime features and supporting emerging agentic workloads, all within one suite of co-designed components that compose into a platform. New capabilities in 1.0 include:

**Multi-tenant quality of service** – Priority and fairness-aware flow control helps latency-sensitive requests, and bursty agent workloads share a fleet without starving one another.

**Zero-downtime operations** – Router high availability with hot standby and rolling updates keep traffic flowing through upgrades and planned maintenance. Management of prefill/decode replicas as one logical unit, with coordinated rollouts and lockstep upgrades, makes disaggregated serving practical in production.

**A stable foundation to build on** – A v1 configuration API and a hardened request path, validated across clouds and accelerators, give adopters a release they can standardize on.

**Agent-aware serving** – Session-aware routing keeps multi-turn conversations close to their cached context, while turn-priority scheduling and native Responses API state support help agents reuse context across turns and reduce recomputation.

The result is infrastructure that can support diverse AI workloads, ranging from latency-sensitive user interactions to large-scale autonomous agent workflows, without forcing organizations to choose between responsiveness and efficiency.

**Growing Ecosystem Momentum**

llm-d is already being used across a growing ecosystem of model developers, cloud providers, infrastructure vendors and enterprise users. Today, the project includes more than 500 contributors from over 25 organizations collaborating on the future of distributed AI serving and helping build common foundations across the AI infrastructure stack.

Core users, including organizations running large-scale distributed inference, frontier model developers optimizing inference stacks and cloud providers serving multi-tenant AI endpoints, have all seen measurable benefit from deploying llm-d.

In one recent use case, open frontier models and llm-d enabled IBM Research to explore quantum error correction hardware at 11.5 times lower inference cost than an all-proprietary configuration. [3] Researchers reported a prefix-cache hit rate of approximately 98% and a blended serving cost of about $0.11 per million tokens.

In a separate recent demonstration using benchmark workloads, researchers showed that llm-d could efficiently serve large mixture-of-experts models for agentic workloads. Researchers used llm-d to deploy GLM 5.2, an approximately 753-billion-parameter open-weight model, on widely deployed NVIDIA GPUs, demonstrating how enterprises can use existing GPU fleets to serve frontier-class AI at scale. On a workload with hundreds of agentic sessions running in parallel, the deployment delivered over 6.6 million output tokens per minute at peak on agentic benchmarks, serving up to 3,000 concurrent coding agents with zero preemptions. At current cloud rental rates, the deployment would have cost 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on the input-heavy traffic patterns typical of agentic workloads. This highlights how llm-d can save resources and costs for enterprises deploying AI in production at scale. [4]

Through its CNCF governance model and broad industry participation, llm-d aims to accelerate innovation while providing organizations with an open alternative for managing increasingly complex AI infrastructure. Looking ahead, the community will continue adapting to new hardware and models as they emerge, evolving the software stack to achieve peak performance across new architectures. To learn more, read the announcement blogs from IBM and Red Hat.

**Supporting Quotes**

Chris Aniszczyk, CTO, CNCF

"AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organizations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It's an important step toward making high performance open source inference more operationally mature and accessible across the cloud native ecosystem."

Brian Stevens, senior vice president and AI chief technology officer, Red Hat

"As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation."

Priya Nagpurkar, vice president, AI native systems, IBM Research

"At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control over the AI capabilities they depend on. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem."

Mark Lohmeyer, VP & GM, AI & Computing Infrastructure, Google Cloud

"Agentic AI is unlocking tremendous value for enterprises and consumers across a broad range of use cases. However, as these workloads scale rapidly in production, they place unique demands on the infrastructure and require new approaches to deliver the performance, reliability and cost-effectiveness our customers need. The release of llm-d 1.0 is an important milestone for the industry to address these needs. As a founding contributor to llm-d, Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine and enabling it across our Google TPU and NVIDIA GPU portfolio, enabling customers to scale multi-tenant environments, optimize their inference performance and dramatically lower their overall serving costs on a hardened, open-source foundation."

Shadi Saba, vice president of engineering, AI/ML, CoreWeave

"Every team running inference at scale hits the same problems: context that often costs more to manage than the output, tenants competing for the same fleet, and upgrades that can't interrupt traffic. llm-d 1.0 gives the community a hardened, open base to solve them once instead of separately. CoreWeave helped found the project, and customers can run llm-d on CoreWeave Kubernetes Service alongside N [source text truncated here]