Skip to content

An editorial team and a community building with AI

vibecoding.incJoin Discord ↗
← All articles

News · Google · 2 min read

Event date: · Published:

Google adds multi-million-token long-context training to MaxText on TPUs

Google added long-context training support to its open-source MaxText framework, letting training sequences scale up to 14 million tokens on Cloud TPU 5p and TPU 7 chips.

Google announced long-context training support in MaxText, its open-source JAX training framework, on November 9, 2026, letting training sequences scale up to 14 million tokens on Cloud TPU 5p and TPU 7 (Ironwood) chips, according to the Announcing Long-Context Training on TPUs with MaxText Context Parallelism post on the Google Developers Blog.

What's changing

  • MaxText, Google's open-source JAX training framework, now supports long-context training on Cloud TPUs.
  • Training sequences scale seamlessly up to 14 million tokens across TPU 5p and TPU 7 (Ironwood) chips.
  • Four context-parallel strategies are supported: All-gather, A2A (DeepSpeed Ulysses), P2P (Ring Attention), and a USP hybrid that combines A2A and P2P.
  • A Tokamax Splash Attention kernel manages attention compute and memory without materializing the full key-value sequence in HBM.
  • Llama 3.1 8B reaches 70% Model FLOPs Utilization (MFU) on TPU 5p for sequence lengths up to 4 million tokens, while TPU 7 maintains 50% MFU across all sequence scales below 6 million tokens.

What it means for AI builders

Context parallelism splits a long sequence across chips so no single chip has to hold activations for the full sequence, which is what makes training at these lengths possible. For teams pretraining or fine-tuning their own large language models, this gives an open-source, TPU-native path to multi-million-token context windows, well beyond the 1-2 million token windows common in production models today, without having to build proprietary long-context training infrastructure.

Availability

  • The functionality ships in MaxText v0.2.4, available on PyPI.
  • Google provides sample training recipes for both TPU 7 and TPU 5p configurations.

Sources

Written with AI assistance, strictly from the sources listed above. Before it went live, automated checks verified its dates, sources and every language edition.

Google adds multi-million-token long-context training to MaxText on TPUs | vibecoding.inc