llama.cpp’s November 7 prerelease lets users compute some uncached model experts on the host processor instead of transferring their weights to the computing device. A tuning tool tests the machine and recommends a cache size and processor share. The cache first fills its slots before sharing begins, addressing slow initial filling at high processor shares.
The local model-running engine llama.cpp added a way to divide uncached expert computation in its November 7 prerelease b12191. Mixture-of-experts models activate selected components, called experts, for each step. Users can now assign some missing experts to the host processor rather than move their weights to the computing device. The setting changes the balance between data transfers and computation on that particular machine.[1]
Device-cache computation and work on host-held expert weights run separately, with their outputs subsequently combined. An empty cache continues loading missing experts until its group of slots is occupied. Only then does the selected processor share apply. This filling rule addresses the slow accumulation of cached experts that could otherwise occur when users allocate a large share to the processor.[1]
A companion tool runs the server with 2 cache sizes and 3 processor shares. It estimates transfer and processor speeds, then simulates other settings using the observed expert routing. Its output recommends cache-size and processor-share arguments. The recommendations depend on that machine's measurements; the release supplies no fixed speed gain applicable to all hardware.[1]
