Trainium training tests reveal long waits before the first step
Independent hands-on tests of Amazon’s Trainium chips found that faster compiled training still comes with a lengthy start. SemiAnalysis measured Qwen3-32B on Trainium2 and tested attention workloads on Trainium3. Compilation delays, compatibility work and profiling setup remain part of the engineering picture, while AWS’s proposed compiler improvements are future targets rather than the published measurements.
Artificial Intelligence··Midday
The first training step takes much longer
Amazon’s Trainium chips are being tested for research training through native PyTorch, the framework used to build and train machine-learning models. SemiAnalysis’s hands-on work covers Trainium2 and Trainium3. Its engineers found that getting to the first training step can take far longer than running later steps. In the Qwen3-32B model experiment on Trainium2, the first step took 184.6 seconds in eager mode, where operations run directly, and 255.7 seconds in compiled mode, which prepares code for execution.[1], [2]
Compiled Qwen training runs faster after startup
On 16 Trainium2 chips, the direct run averaged 11,099 tokens per second, compared with 14,156 for the compiled run. Tokens are the pieces of text a model processes. Those steady training rates came after a first step that took between 30 and 55 times as long as an average subsequent step. A separate gpt-oss 20B training recipe took 27 minutes and 17 seconds to compile. AWS is developing a redesigned compiler stack intended to shorten startup waits; that remains a company target.[1]
Trainium3 attention tests hit compilation time limits
Trainium3 passed 89 of 96 FlexAttention tests, which exercise attention operations used by models. The remaining seven exceeded the compilation time budget rather than producing numerical errors. They all used a batch size of four and a sequence length of 8,192. The engineers also reported numerical agreement with Nvidia hardware in custom attention tests. Neuron Explorer, Amazon’s profiling tool, exposes hardware timelines and memory transfers, although the team found its initial setup confusing.[1]
