The EUROPA consortium has described how it trained a language model with 951.9 billion parameters on Germany’s shared JUPITER supercomputer. Its engineering account covers the distribution of model experts, memory and communication across accelerators. Changes to TorchTitan and PyTorch helped arrange the training system around the public machine’s available memory and network connections.
The EUROPA research consortium has detailed its training setup for a language model with 951.9 billion parameters on JUPITER in Jülich, Germany. This single technical document, published through PyTorch, describes work involving Domyn and Fraunhofer institutes. The publicly funded supercomputer serves multiple research projects. The team used PyTorch and TorchTitan to distribute model training within that shared infrastructure.[1]
The model uses a mixture-of-experts architecture, in which an input activates selected specialist components rather than every parameter. Its configuration includes 384 experts, selecting eight for each token, with about 32.8 billion parameters active per token. One training replica spans 384 accelerators. The four-replica arrangement uses 1536 accelerators altogether, with both accelerator memory and Grace processor memory contributing to the setup.[1]
The team reorganized memory and parameter transfers to work around communication constraints between nodes. FlexShard assigns complete model experts to partitions, reducing the transfers needed for some optimizer operations. Other changes overlap transfers with computation and move saved intermediate values into processor memory.[1]
