Microsoft plans to expand its own AI inference infrastructure in 2027. The interesting question is not how fast one chip runs, but how much useful AI work a data centre can deliver with the power available.
When someone asks an AI assistant a question, the answer may arrive in seconds. Behind it, processors load model data, move information through memory and generate the response one token at a time. Multiply that work across millions of requests, and the ongoing cost of running AI becomes a major infrastructure problem.
The challenge grows with applications that do more than answer a short question. An AI agent may read a long document, use several tools and produce a response over multiple steps. It can consume computing capacity for longer, and its performance depends on how efficiently the underlying system handles memory and communication, not only how much arithmetic a processor can perform.
That is the context for Microsoft’s October 12, 2026 announcement. Microsoft says its Maia 200 accelerator is already serving production workloads, including Microsoft Copilot. The company plans to begin deploying Maia 300 in 2027 and expand the Maia infrastructure family to multi-gigawatt scale starting that year. These are different milestones: Maia 200 is operating today. Maia 300’s reported improvements and large-scale expansion are forward-looking.
Why inference needs a different kind of infrastructure
Training a model involves building or improving it using large amounts of computation. Inference is the work performed every time that trained model handles a request. Training may be a substantial one-time or periodic investment, while popular services must pay the inference cost continuously.
For inference, raw processing power is only part of the answer. Model parameters and temporary working data must reach the processors quickly. Long conversations also require systems to retain and access information from earlier parts of a request. If processors are waiting for data, adding arithmetic capacity alone may not deliver a matching increase in useful output.
This is why Microsoft’s Maia 300 engineering explanation places so much emphasis on moving less data and keeping it close to where computation happens. Maia 300 uses a multi-chiplet design that separates compute, memory-control and communication functions. Its software-directed approach aims to place data near the processor that needs it, while lower-precision number formats reduce how much data some inference operations must move and store.
There are trade-offs. Lower precision must preserve acceptable model behaviour for the relevant workload, and actual efficiency depends on software, model characteristics and system utilization. A hardware specification does not establish the experience a customer will see.
What Microsoft says Maia 300 will deliver
Microsoft reports that Maia 300 will use eight HBM4 memory stacks with 288 GB capacity and 18 TB/s peak bandwidth per accelerator. HBM is high-bandwidth memory placed close to computing hardware. Greater bandwidth helps feed processors working on large models. A planned rack will contain 96 accelerators within a 167 kW power envelope, with 27.6 TB of HBM and approximately 1.7 PB/s aggregate memory bandwidth.
Those numbers describe the full design, not a product customers can currently benchmark in production. Microsoft expects Maia 300 to deliver 40% more tokens per megawatt than what it calls leading GPU-based systems. It also reports up to seven times the rack-level token throughput of Maia 200 in a full 96-accelerator configuration.
The comparison needs care. A Maia 300 rack has twice the accelerator count of its predecessor, so seven times the rack throughput does not mean an individual Maia 300 accelerator is seven times faster. The 40% figure is an efficiency claim against an unspecified set of GPU-based systems and workloads in the announcement. It is not a universal speedup or a verified reduction in customer bills.
| Maia 300 design figure | Microsoft-reported value |
|---|---|
| HBM per accelerator | 288 GB (eight HBM4 stacks) |
| Peak HBM bandwidth per accelerator | 18 TB/s |
| Accelerators per rack | 96 |
| Rack power envelope | 167 kW |
| HBM per rack | 27.6 TB |
| Aggregate rack HBM bandwidth | 1.7 PB/s |
Source: Microsoft’s October 12, 2026 announcement and architecture deep dive. These are reported design specifications, not independent production benchmark results.
The network and the data centre are part of the design
Large AI workloads may need many accelerators to cooperate. When model data or intermediate results must move between them, communication can become a bottleneck. Maia 300 is designed with a shared communication approach across accelerators and racks. Microsoft’s proposed superpod reaches 12,288 accelerators across 128 racks, using Ethernet-based transport and optical links between racks.
That architecture also creates operational demands. A 167 kW rack must receive power, reject heat and remain serviceable. Microsoft describes liquid cooling for most of the rack’s thermal load, along with power telemetry that can inform workload placement. These are not supporting details: they help determine how much inference capacity can actually be operated within a data centre’s constraints.
The promised multi-gigawatt rollout makes the point especially clear. A gigawatt is a measure of power, not a measure of AI model quality or response speed. Microsoft is signalling the scale at which it intends to operate Maia systems, but the announcement does not establish that all planned capacity is already built or in use.
What this means for cloud and AI teams
Most organizations will not choose individual accelerator chiplets or design AI data-centre cooling systems. They will, however, make choices shaped by these developments: which model to run, how much context to send, what response latency is acceptable and what the resulting workload costs.
A model-serving service should ultimately be evaluated on its real workload. Tokens per second and tokens per watt are useful measurements, but users also care about response latency, quality, availability and cost per completed task. An agent that performs several tool calls may have a very different cost profile from a simple chat request, even when both use the same model.
Maia 300 is therefore worth following without treating Microsoft’s performance expectations as settled industry benchmarks. Its broader significance is the move toward designing the chip, memory, network, software and data centre as one inference system. As AI becomes part of everyday applications, efficiency at that level may matter as much as improvements in the models themselves.
References and further reading
Both sources were opened and checked on 14 October 2026. All figures are reported by Microsoft and have not been independently reproduced.
- Microsoft: Microsoft to expand Azure Maia system to multi-gigawatt deployments. Scott Guthrie, Microsoft, October 12, 2026. Release milestones and Microsoft-reported performance expectations.
- Microsoft Azure: Deep dive into Azure Maia 300 architecture. Rani Borkar and Saurabh Dighe, Microsoft Azure, October 12, 2026. Memory, chiplet, networking, rack and cooling design details.