700 Tokens Per Second: New TPU Engine Rivals Nvidia Speeds
A newly released open source software stack enables Meta to run advanced AI models on Google chips with performance metrics that challenge the current market leader.
By Priya Marino ·
The competitive landscape for artificial intelligence hardware is undergoing a significant shift as custom chips gain broader software support. On Wednesday, Google confirmed that its tensor processing units are now being used in production environments by Meta, a major move that marks the first time these chips have been sold to external customers for large scale AI workloads.
The announcement centers on the release of a new PyTorch native version of the SGLang inference engine. This software update allows Google hardware to function as a standard option for PyTorch, which is the dominant framework in the research and development community. Previously, running these engines on TPUs required the use of Google's proprietary JAX framework. The new versions eliminate this barrier, enabling developers to deploy models without extensive code rewrites. A matching open source version of vLLM, another popular inference engine, was also released to support this new hardware integration.
Meta has been an active partner in the development of this technology. Zongwei Zhou, the PyTorch TPU lead at Meta, stated that the SGLang TorchTPU system is already operating in production. He described the result as a remarkably mature serving stack that stems from a close three way collaboration between Meta, Google, and the SGLang team. This team is based at RadixArk, a startup founded by the creators of the SGLang engine. Jana van Greunen, director of PyTorch engineering at Meta, emphasized the strategic importance of the move. She noted that the company cannot be constrained by a single hardware architecture and that this new tool enables the deployment of PyTorch workloads directly onto Google chips.
This is not Meta's first interaction with Google hardware, but the approach has evolved. Google describes the older PyTorch XLA bridge as a predecessor to the new system. Reports from SemiAnalysis indicated that Meta research teams used TPUs via that older bridge between 2020 and 2023 before canceling those contracts. The current push for PyTorch on TPUs was largely aimed at Meta, which had previously resisted migrating its operations to the JAX framework. While the exact location of Meta's production units remains unclear, industry reports from last year suggested that Google Cloud TPU rentals were under discussion for 2026, with potential purchases planned for 2027. Meta reportedly signed a multi year rental agreement to facilitate this transition.
The timing of this release coincides with other major industry developments. One day prior to this announcement, Amazon revealed that it was open sourcing TorchNeuron, a native PyTorch backend for its own Trainium 2 and Trainium 3 chips. This disclosure occurred at the same PyTorch Conference where the Google and Meta news broke. Analysts view both moves as strategic efforts to loosen the software lock in that has historically favored Nvidia, the dominant provider of AI accelerators.
Performance benchmarks for the new stack show significant improvements. A tuned vLLM setup serving the Kimi K3 model, an open weight model from Moonshot AI with 2.8 trillion parameters, reportedly reaches speeds of over 700 tokens per second for a single user. This test was conducted on the seventh generation Ironwood TPU. The figure was provided by Inferact, a startup founded by the creators of vLLM that assisted Google in building the new engine. In comparison, vLLM's published setup on 16 Nvidia GB200 chips reached 452 tokens per second. The new kernels also deliver roughly 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes when speculative decoding is not used.
The software ecosystem is expected to expand rapidly. SGLang TorchTPU will continue adding support for new models, starting with DeepSeek V4.1 Flash and Reflection Beam. The development team is working toward day zero TPU support for new open models, which means compatibility on the day of release, with a target of the first quarter of 2027. This goal was outlined by RadixArk.
Stakeholders plan to provide further details on the technology and its implications for the industry. Google, Meta, and RadixArk will share more information at the SGLang Summit 2026. The event is scheduled for November 12 and 13 in San Francisco. As these custom chips gain native support for the industry's most popular frameworks, the pressure on the incumbent hardware provider is likely to intensify in the coming year.
