Eigen RadarAI
Analysis

Microsoft splits expert-model execution between processors and GPUs

ONNX Runtime 1.32 adds planning and memory controls for running mixture-of-experts models across central and graphics processors. The release also changes how weights are loaded and how runtime plugins match the core. These are execution and compatibility changes for existing models, accompanied by a transition in Apple-platform package distribution.

Artificial Intelligence··Morning
Overhead view of an opened workstation on a bench: processor under a cooler, a graphics card, memory modules, and spare memory sticks beside the case.

Expert weights move across two processor types

Microsoft’s ONNX Runtime, software for executing trained artificial-intelligence models, now plans hybrid execution of mixture-of-experts models within each session. Version 1.32.0 tracks a memory budget while splitting work between a central processing unit and a graphics processing unit. Expert weights stored with reduced numerical precision can use shared-layout processor kernels. Memory-mapped storage provides another route for accessing those weights. The newly shipped capability changes the runtime handling of existing models.[1]

Model packages gain separate weight handling

New interfaces expose a device’s memory capacity. Compilation modes can separate model dependencies from weight dependencies, and Python bindings can create or load portable packages and choose their variants. Several workers can stage external weights in pinned memory for transfer to a graphics processor. The release incorporates ONNX 1.23.2 and operator set 28, alongside fixes covering graph validation, model loading and buffer lifetimes.[1]

Plugins must match the runtime core

The old TensorRT RTX bridge gives way to NVIDIA’s standalone plugin, while the Arm Compute Library provider is removed. Core and plugin operator schemas must match: Microsoft warns that disagreements can yield wrong execution or crashes without being detected at registration. Apple-platform distribution also reaches a transition point, with this the final CocoaPods package before a move to Swift Package Manager.[1]

References

  1. News sourceMicrosoftONNX Runtime 1.32 adds hybrid CPU and GPU expert offloading↩1↩2↩3