A model that fits on one core
Deepgram’s Nova-3 speech-recognition model fits on one physical Trainium2 core. In the port described by Deepgram and AWS, that detail determined how to partition the device. The team replaced the default four-logical-core arrangement with LNC1, exposing eight physical cores separately. That allowed eight independent model replicas on the same device. For the operator, usable capacity starts with how the model occupies memory. Combining cores in this workload gives up opportunities to run additional copies.[1]
The measurement boundary is offline batch transcription. Processing a backlog of contact-centre recordings is a different job from serving someone waiting for a response during a conversation. Larger batches can fill the device, but require work to be accumulated. The amount of work extracted from this device cannot directly establish conversational latency or the electricity efficiency of a whole facility. Those conclusions require measurements at a common service target and a wider system boundary.[1]
Reducing repeated memory work
The team reduced repeated memory work. Its custom feed-forward kernel loads a weight tile and reuses it across several token tiles; the reference implementation repeatedly fetched those weights. Depthwise convolution also moved from an implicit matrix multiplication that used the Tensor Engine poorly to multiplication and accumulation on the Vector Engine. Assigning the same model’s operations to different hardware units changed which part of the chip could be kept busy.[1]
Compilation was another constraint: batch sizes above 32 required a custom two-dimensional convolution kernel. The encoder and decoder were compiled into separate static graphs. These details show the engineering between a portable model definition and an efficient serving arrangement. Software can change the useful work obtained from existing silicon. Capacity in this case depends on fitting memory access and execution to the job, alongside the option of adding more devices.[1]
The CPU belongs in the same calculation
The bottleneck also appeared outside the chip. Small decoder calls made CPU dispatch overhead prominent. Python’s multithreaded arrangement faced the interpreter lock, while a multiprocess arrangement incurred dispatch costs. Deepgram’s production server uses Rust, and the team built a lightweight runtime with one worker thread per replica. It invoked ahead-of-time compiled NEFF files through libnrt, removing Python and PyTorch from the serving path. The host feeding the accelerator belongs inside the capacity calculation.[1]
Eight replicas therefore do not imply eight times the service capacity for every application. Deepgram’s measurements compare specific serving arrangements at particular batch sizes. The narrower engineering lesson is useful: fitting a model on a core, reusing weights and dispatching work promptly from the host operate together. Another model can encounter different limits. These layers determine the distance between the number of devices an operator buys and the results users receive on time.[1]