專為代理時代构建:最新 TPU 8tx 與 8i Max 登場
早在十多年前,我們為了滿足語音辨識與語言翻譯等 Google 內部需要,首度設計出張量處理器(TPU)。歷經八代 TPU 的演進,如今的應用場景已呈現爆發性成長,涵蓋範圍涵盖:第一,訓練並给出我們自家的 Gemini 模型;第二,處理 AI 代理工作負載;第三,透過我們的雲端基礎設施,為 Google Cloud 客戶给出服務;以及第四,近期更進一步讓客戶能將 TPU 直接部署到自己的資料中心。TPU 現已成為一項快速成長的業務,擁有近 100 家客戶,其中包含頂尖實驗室、交易機構、零售商與大學院校等。
今年四月在 Google Cloud Next 大會上,我們在同一代產品中推出兩款全新專用 TPU:專為大規模訓練构建的 8t,以及專為即時推論設計的 8i。相較於前一代 TPU,TPU 8t 及 8i 分別提高了高達 2.7 倍的訓練效能與 80% 的推論性價比。
今天,我們更進一步擴展第八代 TPU 產品陣容,新增兩款系統:專為龐大的脈絡視窗(context windows)與最高單機櫃(rack)運算密度設計的 TPU 8i Max,以及專為安全與機密推論构建的 TPU 8tx。這是我們首次在同一年內推出四款 TPU 系統。
我們擴展後的第八代 TPU 產品陣容:TPU 8t、TPU 8i、TPU 8tx 與 TPU 8i Max。
雖然我們一直與 Google DeepMind 攜手共同設計 TPU,但我們現在也越來越常收到外部機構針對特定需要的意見回饋。隨著運算規模從人類層級邁向代理規模(agent scale),企業紛紛尋求能夠跨越 AI 模型龐大脈絡視窗進行推論的加速器。另外,部分受高度監管的客戶,也針對獨特的工作負載提出了專屬加密能力的需要。
我們今天發表的全新 TPU 系統正是為因應這個新局勢。搭配我們近期發布的 PyTorch 相關更新,我們的目標是讓每位開發者與 Cloud 客戶都能更輕鬆地使用 TPU。透過將模組化(modular)的自研晶片與開放且易用的軟體堆疊相結合,我們能跟上 AI 產業的發展步伐,快速交付全新的硬體與系統规格。
圍繞晶片构建電腦系統
今天我們發表了兩款全新的 TPU 系統,而「新晶片」與「新系統」之間有個关键的差異。
設計一款新晶片意味著要在受物理定律限制、且製造週期複雜漫長的多年過程中,將數十億個微型電晶體(microscopic transistors)佈局在矽晶圓上。相對地,构建一套「系統」,則是關於我們如何將運算、網路與儲存資源整合在晶片周圍。這包含了如何將這些資源與晶片封裝在一起,以便迅速滿足新興的客戶需要與研究應用場景。
十多年前當我們构建第一代 TPU 時,我們便深知 AI 軟體演進速度將遠遠超越矽晶片製造的速度。因而,我們在系統平衡與模組化上做了一項基礎性的關鍵布局,將 TPU 架構設計為 AI 超級電腦(AI Hypercomputer)中的基礎建構模組,讓工程師能圍繞著晶片,快速調整封裝方式、高速記憶體、光學網路與安全防護機制。這種做法讓我們能在短短幾個月交付專用的超級電腦(supercomputers),而非幾年。
專為安全且具備龐大脈絡的 AI 量身构建
在我們的第八代 TPU 產品陣容中,每個系統都是針對 AI 生命週期中的不同階段量身构建。
TPU 8t 與 TPU 8i 奠定了這一代晶片的基礎:
- TPU 8t 旨在成為龐大基礎模型的訓練主力,具備最高的「有效產出率(goodput)」,即每消耗一瓦電力所能交付的實用智慧(useful intelligence)。
- TPU 8i 則是推論引擎,旨在突破記憶體高牆(memory wall),可將晶片間的通訊延遲大幅縮減高達一半。
如今,隨著企業將自主 AI 代理導入正式環境,我們針對這些基礎系統构建了兩款全新规格,以解決兩項迫切的新挑戰:硬體層級的資料安全,以及百萬詞元規模的脈絡處理能力。
TPU 8tx
TPU 8tx 作為第八代的第三款系統,針對安全與機密推論進行了最佳化,同時也格外適合小規模訓練。它是專為受高度監管的企業與政府機構所构建,讓他們能在處理高機敏資料的同時,安全地執行前沿或專屬模型(proprietary models)及自主 AI 代理。
一般而言,資料在儲存或透過網路傳輸時會進行加密,但在電腦處理資料的過程中,必須先在記憶體中解密。TPU 8tx 採用與 TPU 8t 相同的矽晶片,並搭配第五代 AMD EPYC CPU 主機,將安全機制直接內建於硬體層中,能確保敏感資料與模型權重即使在活躍使用狀態下,也能全程受到保護並維持隔離。另外,藉由採用我們專為即時推論所构建的相同低延遲 Boardfly 網路,TPU 8tx 能讓多步驟 AI 代理保持順暢運作,在幾乎零效能損耗的情況下,给出比 TPU 8t 低高達 2 倍的延遲表現。
將訓練等級的運算能力與高速推論網路結合,意味著企業可以在單一系統上同時執行並客製化自己的模型。TPU 8tx 讓團隊可以使用專屬資料微調模型、將大型模型蒸餾(distill)為較小模型,並透過強化學習訓練他們的 AI 代理,所有過程皆無需在不同叢集(cluster)間搬移敏感資料。
TPU 8tx 專為機敏的企業工作負載所設計,给出硬體級機密運算,支援專屬微調與安全的 AI 代理執行
TPU 8i Max
我們的第四款系統 TPU 8i Max ,則是為了解決另一個問題而生:記憶體高牆。
當數十個 AI 代理同時針對整個軟體程式碼庫或長達數小時的影片協同運作時,記憶體頻寬很快就會成為瓶頸。因而,我們將兩顆低延遲的 TPU 8i 晶片整合封裝在單一模組上。
TPU 8i Max 同步擴展了運算與記憶體資源,给出雙倍的運算能力、雙倍的 SRAM,以及雙倍於 TPU 8i 的 HBM 頻寬,確保處理器隨時備妥關鍵資料,避免大型工作負載發生停滯。另外,由於 TPU 8i Max 在現有的機櫃空間容納了兩倍的矽晶片,資料中心每平方英尺與每瓦電力能部署的運算能力也因而大幅提高。
採用雙晶片設計的 TPU 8i Max 系統,專為處理龐大的脈絡視窗與给出最高運算密度而构建。
第八代 TPU 的四款系統,不僅能靈活支援各種工作負載,更共同為客戶在代理式生命週期的每個階段,给出了專用的加速器:TPU 8t 用於大規模訓練,TPU 8i 用於低延遲推論,TPU 8tx 用於機密運算(confidential computing),而 TPU 8i Max 則用於處理龐大脈絡視窗與给出最高單機櫃運算密度。
將 TPU 開放給所有開發者
十多年前當我們剛推出第一代 TPU 時,僅有少數幾家公司執行的工作負載需要用到 TPU 等級的運算能力。當時的應用情境具備高度專業性,因而工程師會使用經過最佳化的框架並手動調校程式碼。
多年來,隨著 AI 技術不斷進步,越來越多企業開始採用加速器。如今,AI 的普及程度,已讓 TPU 成為廣泛企業與多元應用情境的絕佳選擇。因而,我們持續有系統地消除採用門檻,讓 TPU 變得和通用硬體一樣易於使用。
這項投入在今天透過以下關鍵領域全面展現:
- 向每位 PyTorch 開發者開放 TPU 開發環境,無需改寫程式碼:上個月我們開源了 TorchTPU,讓 PyTorch 程式碼與 Hugging Face 模型現在可原生執行在 TPU 上,同時為 vLLM 與 SGLang 等主流開源推論引擎给出開箱即用的 TPU 後端支援。為了讓硬體最佳化更加輕鬆,我們全新的開源 TPU agent suite(現已開放預覽)可直接串接至 Claude Code、Cursor 與 Google Antigravity 等程式編寫工具,在短短幾分鐘內自動完成 TPU 導入設定、診斷效能瓶頸,並使用自然語言自動調校客製化核心(custom kernels)。針對實際生產環境的部署,我們全新的開源 XLA Serving Engine 可讓使用者在 Python 中訓練模型,並直接在低延遲的 C++ 執行環境中,跨 TPU、GPU 與 CPU 给出模型服務。
- 在 InferenceX 與 AgentX 基準測試中展現業界領先的生產環境效能:在實際生產環境中,真正关键的是真實 AI 工作負載的執行成果,而非理論上的晶片峰值速度。在測試真實代理式程式編寫與長脈絡作業的業界標準 InferenceX 與 AgentX 基準測試中,我們在 TPU 上執行的原生 vLLM-TorchTPU 軟體堆疊創下了頂尖的性價比表現。最新的實測結果顯示,這些優勢能延伸至廣泛的前沿開放模型與開源服務堆疊,同時維持高度模型準確度。未來針對相同硬體推出的軟體最佳化更新,還將進一步推升這些表現。
- 為自主 AI 代理消除閒置運算浪費:由於 AI 代理具備狀態性(stateful),在等待資料庫查詢、外部軟體工具回應或人工回饋時,經常會暫停數秒甚至數小時。為了讓我們的加速器在這些暫停期間更具能源效率,我們全新的 Agent Compute Engine 軟體會在自動暫停並重啟閒置的代理工作階段,從而降低營運基礎設施成本。。
- 透過模型狀態層(TPU Sync)突破記憶體高牆:圍繞晶片构建電腦系統也意味著要在軟體層面統一記憶體、儲存空間與網路。當 AI 代理跨越百萬詞元的脈絡和持續的強化學習(Reinforcement Learning,RL)迴圈進行推理時,記憶體很快就會成為瓶頸。因而,我們開源了專為 TPU 叢集构建的模型狀態層 TPU Sync。TPU Sync 將晶片的 HBM、主機的 DRAM、共用儲存空間與資料中心網路,整合為單一且共同設計的記憶體階層(memory hierarchy),能在幾毫秒內自動將龐大的 KV 快取(KV cache)分層移出晶片,跨越分散的預填充(prefill)與解碼(decode)節點串流傳輸脈絡,並同步強化學習權重(RL weights),使得 KV 傳輸延遲與第一個詞元生成時間(TTFT)大幅縮減 90% 以上。
- 從單一控制器跨越 TPU 擴展分散式工作負載:現已於 Google Kubernetes Engine(GKE)上正式推出的 Cloud Pathways ,將驅動 Gemini 的 Google DeepMind 執行環境帶給每一位 Google Cloud 客戶。透過使用單一控制器(single-controller)架構與 pathwaysutils,使用者可以跨越多個切片(slices)調度大規模 TPU 系統,讓它們進行預訓練、強化學習與多主機推論時,宛如單一本地機器般運作。Cloud Pathways 結合了共置的 Python 附屬容器(sidecar)、一分鐘內的切片熱插拔(hot-swapping)以及彈性重新分片(elastic resharding)技術,讓長時間執行的工作在面對硬體發生中斷時仍可以持續運作。
- 透過 GKE 動態切片能力按需要拼接客製化 TPU 拓樸:現已全面推出的 GKE 動態切片(GKE dynamic slicing)能力,利用光路交換(optical circuit switching,OCS)技術,將實體硬體佈建與邏輯切片规格徹底解耦(decouple)。透過即時將多個 TPU 立方體(TPU cubes)動態組裝成更大的拓樸(超級切片,super-slicing),或是將其分割為較小、彼此獨立的工作負載(子切片,sub-slicing),這項更新能徹底消除過往僵化且需要手動重建節點集區(nodepool)的漫長週期。這種拓樸靈活性將作業啟動時間大幅縮短了高達 5 倍,並將故障復原速度提高達 4.5 倍,全面最大化所有 TPU 的整體機隊使用率與工作負載有效產出率。
結合十年經驗的共同設計
能夠在同一年內推出四款 TPU 系統以及一個統一且開放的軟體堆疊,歸功於晶片架構師、網路工程師、編譯器(compiler)團隊與 Google DeepMind 研究人員並肩合作的成果。
現在,我們也越來越常運用 AI 來構思並构建下一代的 AI 基礎設施。從使用 Google DeepMind 的 AlphaChip 強化學習系統,到運用 Gemini 在短短幾小時(而非幾週)內完成晶片佈局,我們正透過 TPU 驅動的 AI 服務,快速推進下一代 TPU 硬體的開發進程。這才是真正的共同設計,也是 Google 獨有的差異化優勢。
歡迎收看影片內容,了解更多有關我們全新 TPU 系統的資訊。
本文来源:https://blog.google/intl/zh-tw/products/cloud/-tpu-8i-max-and-8tx
若有版权问题,请来信告知:13951431913#139.com 或在右方提交评论。
Built for the era of agents: the latest TPU 8tx and 8i Max arrive
Google Cloud's eighth-generation TPU lineup is expanding with two new systems: TPU 8tx, designed for security and confidential inference, and TPU 8i Max, built for huge context windows and the highest per-rack compute density. This is the first time we have launched four TPU systems in the same year, paired with an open software stack to help every developer embrace the age of AI agents.
Amin Vahdat
Chief Technology Officer and Senior Vice President, AI Infrastructure, Google
More than a decade ago, we first designed Tensor Processing Units (TPUs) to meet Google's internal needs such as speech recognition and language translation. After eight generations of TPU evolution, application scenarios have grown explosively, spanning: first, training and running our own Gemini models; second, handling AI agent workloads; third, serving Google Cloud customers through our cloud infrastructure; and fourth, more recently, enabling customers to deploy TPUs directly in their own data centers. TPUs have become a fast-growing business, with nearly 100 customers including leading labs, trading firms, retailers, and universities.
In April at the Google Cloud Next conference, we introduced two new specialized TPUs within the same generation: 8t, built for large-scale training, and 8i, designed for real-time inference. Compared with the previous-generation TPU, TPU 8t and 8i deliver up to 2.7x higher training performance and 80% better inference price-performance, respectively.
Today, we are further expanding our eighth-generation TPU lineup with two additional systems: TPU 8i Max, designed for massive context windows and the highest per-rack compute density, and TPU 8tx, built for secure and confidential inference. This marks the first time we have introduced four TPU systems within one year.
Our expanded eighth-generation TPU lineup: TPU 8t, TPU 8i, TPU 8tx, and TPU 8i Max.
Although we have always co-designed TPUs with Google DeepMind, we are now receiving more and more feedback from external organizations about specific needs. As compute scales from human scale to agent scale, enterprises are seeking accelerators that can run inference across the huge context windows of AI models. Additionally, some highly regulated customers have asked for dedicated encryption capabilities for unique workloads.
The new TPU systems we are announcing today are built to meet this changing landscape. Together with our recent PyTorch updates, our goal is to make TPUs easier to use for every developer and Cloud customer. By combining modular, custom-developed chips with an open and easy-to-use software stack, we can keep pace with the AI industry and quickly deliver new hardware and system specifications.
Building computer systems around chips
Today we announced two new TPU systems, and there is a key difference between a “new chip” and a “new system.”
Designing a new chip means laying out billions of microscopic transistors on a silicon wafer over a multi-year process constrained by the laws of physics and complicated manufacturing cycles. In contrast, building a “system” is about how we integrate compute, networking, and storage resources around the chip. This includes how to package those resources with the chip to quickly meet emerging customer needs and research use cases.
Over a decade ago, when we built the first-generation TPU, we knew AI software would evolve far faster than silicon manufacturing. That led us to make a foundational bet on system balance and modularity, designing the TPU architecture as a base building block within an AI Hypercomputer so engineers could quickly adjust packaging, high-speed memory, optical networking, and security around the chip. This approach enables us to deliver specialized supercomputers in a matter of months rather than years.
Tailored for secure AI with massive context
Within our eighth-generation TPU lineup, each system is built to fit different stages of the AI lifecycle.
TPU 8t and TPU 8i anchor this generation of chips:
TPU 8t is designed to be the training workhorse for large foundation models, delivering the highest effective goodput—useful intelligence delivered per watt of power consumed.
TPU 8i is the inference engine, designed to break through the memory wall, reducing inter-chip communication latency by as much as half.
Now, as enterprises bring autonomous AI agents into production, we’ve built two new configurations for these foundation systems to tackle two urgent new challenges: hardware-level data security and million-token-scale context processing.
TPU 8tx
TPU 8tx, the third system in our eighth generation, is optimized for secure and confidential inference and is also especially well suited for small-scale training. It’s built for highly regulated enterprises and government agencies, allowing them to safely run frontier or proprietary models and autonomous AI agents while handling highly sensitive data.
Typically, data is encrypted at rest or in transit over the network, but while a computer is processing it, it must first be decrypted in memory. TPU 8tx uses the same silicon as TPU 8t, paired with a fifth-generation AMD EPYC CPU host, and bakes security directly into the hardware layer to keep sensitive data and model weights protected and isolated the entire time—even while actively in use. And by using the same low-latency Boardfly networking we built for real-time inference, TPU 8tx lets multi-step AI agents keep running smoothly with nearly zero performance overhead, delivering latency up to 2x lower than TPU 8t.
Combining training-grade compute with high-speed inference networking means businesses can run and customize their own models on a single system. TPU 8tx lets teams fine-tune models on private data, distill large models into smaller ones, and train their AI agents with reinforcement learning—all without moving sensitive data across different clusters.
TPU 8tx is designed for sensitive enterprise workloads, delivering hardware-grade confidential computing and supporting private fine-tuning and secure AI agent execution.
TPU 8i Max
Our fourth system, TPU 8i Max, is built to solve a different problem: the memory wall.
When dozens of AI agents collaborate at once across an entire software codebase or hours-long videos, memory bandwidth quickly becomes the bottleneck. So we integrated two low-latency TPU 8i chips into a single module.
TPU 8i Max scales compute and memory together, delivering double the compute, double the SRAM, and double the HBM bandwidth of TPU 8i—keeping critical data ready for the processor and preventing stalls on large workloads. And because TPU 8i Max fits twice the silicon into existing rack space, data centers can deploy substantially more compute per square foot and per watt.
The dual-chip TPU 8i Max system is built to handle massive context windows and deliver the highest compute density.
The four systems in the eighth generation of TPUs not only flexibly support diverse workloads, but together give customers dedicated accelerators at every stage of the agentic lifecycle: TPU 8t for large-scale training, TPU 8i for low-latency inference, TPU 8tx for confidential computing, and TPU 8i Max for massive context windows and the highest per-rack compute density.
Opening TPUs to all developers
When we launched the first generation of TPUs over a decade ago, only a handful of companies had workloads that needed TPU-class compute. Back then the use cases were highly specialized, so engineers used optimized frameworks and tuned code manually.
Over the years, as AI has advanced, more and more businesses have adopted accelerators. Today, the ubiquity of AI has made TPUs an excellent fit for a broad set of enterprises and diverse applications. So we’ve been systematically removing adoption barriers to make TPUs just as easy to use as general-purpose hardware.
That investment is now fully reflected across these key areas:
Opening the TPU development environment to every PyTorch developer with no code rewrites: Last month we open sourced TorchTPU, so PyTorch code and Hugging Face models can now run natively on TPUs, while providing out-of-the-box TPU backend support for mainstream open source inference engines like vLLM and SGLang. To make hardware optimization easier, our new open source TPU agent suite (now available in preview) plugs directly into coding tools such as Claude Code, Cursor, and Google Antigravity to automatically complete TPU onboarding setup in minutes, diagnose performance bottlenecks, and tune custom kernels using natural language. For real production deployments, our new open source XLA Serving Engine lets users train models in Python and serve them directly in a low-latency C++ execution environment across TPUs, GPUs, and CPUs.
Delivering industry-leading production performance on InferenceX and AgentX benchmarks: In real production environments, what matters is how real AI workloads perform, not theoretical peak chip speed. In the industry-standard InferenceX and AgentX benchmarks for real agentic coding and long-context operations, our native vLLM-TorchTPU software stack running on TPUs set top cost-performance results. The latest measured results show these advantages extend across a wide range of frontier open models and open source serving stacks while maintaining high model accuracy. Future software optimization updates on the same hardware will further boost these gains.
Eliminating idle compute waste for autonomous AI agents: Because AI agents are stateful, they often pause for seconds or even hours while waiting for database queries, responses from external software tools, or human feedback. To make our accelerators more energy-efficient during these pauses, our new Agent Compute Engine software automatically suspends and restarts idle agent sessions, reducing the cost of operating the infrastructure.
Breaking through the memory wall with a model state layer (TPU Sync): Building computer systems around the chip also means unifying memory, storage, and networking at the software layer. As AI agents reason across million-token contexts and continuous Reinforcement Learning (RL) loops, memory quickly becomes a bottleneck. That’s why we open-sourced TPU Sync, a model state layer built for TPU clusters. TPU Sync integrates the chip’s HBM, the host’s DRAM, shared storage, and data center networking into a single, co-designed memory hierarchy. It automatically tiers huge KV caches off the chip within milliseconds, streams context across distributed prefill and decode nodes, and syncs RL weights—reducing KV transfer latency and time-to-first-token (TTFT) by over 90%.
Scale distributed workloads across TPUs from a single controller: Now officially available on Google Kubernetes Engine (GKE), Cloud Pathways brings the Google DeepMind execution environment that powers Gemini to every Google Cloud customer. By using a single-controller architecture and pathwaysutils, users can orchestrate large-scale TPU systems across multiple slices so they operate like a single local machine for pretraining, reinforcement learning, and multi-host inference. Cloud Pathways combines a co-located Python sidecar, slice hot-swapping in under a minute, and elastic resharding so long-running jobs can keep running even when hardware interruptions occur.
Compose custom TPU topologies on demand with GKE dynamic slicing: Now fully available, GKE dynamic slicing uses optical circuit switching (OCS) to fully decouple physical hardware provisioning from logical slice specifications. By dynamically assembling multiple TPU cubes into larger topologies (super-slicing) on the fly, or splitting them into smaller, independent workloads (sub-slicing), this update eliminates the rigid, manual cycle of rebuilding node pools. This topology flexibility reduces job startup time by up to 5x and improves failure recovery speed by up to 4.5x, maximizing fleet utilization and effective workload throughput across all TPUs.
Co-design built on ten years of experience
The ability to deliver four TPU systems in the same year—along with a unified, open software stack—is the result of chip architects, network engineers, the compiler team, and Google DeepMind researchers working shoulder to shoulder.
Today, we are increasingly using AI to conceive and build the next generation of AI infrastructure as well. From using Google DeepMind’s AlphaChip reinforcement learning system to applying Gemini to complete chip layouts in hours rather than weeks, we are accelerating the development of next-generation TPU hardware through TPU-powered AI services. This is true co-design, and it is a differentiator unique to Google.
Watch the video to learn more about our new TPU systems.
Category: