MiTAC separates AI prompt processing and answer generation across servers
Taiwanese server company MiTAC Computing and Skymizer presented a design at the SC26 conference in Chicago that splits AI inference between two servers. One processes the incoming prompt, while the other generates the answer. The proposal is intended for organizations running large models on their own premises and connects the stages over a high-speed network.
Artificial Intelligence··Midday
A proposal shown in Chicago
Taiwanese server maker MiTAC Computing and Skymizer presented a two-server AI inference design at SC26 in Chicago, in the United States. Inference is the work a trained model does to answer a request. Their proposal separates processing the incoming prompt from generating the response, assigning the stages to different servers for organizations that run large models on their own premises.[1], [2]
The first server prepares the prompt
The first stage, called prefill, processes the prompt and is compute-intensive. MiTAC proposes using a server with AMD processors and accelerators for that work. It transfers the resulting key-value cache over a high-speed network. This cache holds information from prompt processing that the next stage uses as the model continues working on the same request.[1]
The second server generates the answer
The decode stage produces the response step by step and relies heavily on memory access. The proposed second server combines Intel processors with Skymizer accelerators to handle that stage. MiTAC presents the arrangement as a design for on-premises use, including large models such as Kimi K3. Its conference announcement also names Ubuntu as an operating-system layer and MiCoreView for system monitoring and management.[1]