Ayush Pande compared several ways to run a language model on his Intel Arc graphics card and settled on Vulkan-based llama.cpp. He reported faster output than with Ollama on that machine, while two other setups failed or ran slowly. His account also describes memory-related tuning and image input, with unresolved configuration problems limiting the comparison.
On his own Intel Arc A750 graphics card, hardware writer Ayush Pande settled on the Vulkan graphics interface for running llama.cpp, software that executes language models locally. His November 7 personal trial compared it with Ollama and two Intel-oriented alternatives. With the same Gemma model, he reported around 33 output tokens per second under Vulkan, against 19–22 under Ollama. Tokens are the units of text a model processes.[1]
The comparison ran in separate software containers on an older home server, with access to its graphics card. OpenVINO compiled but crashed when Pande tried running models. His SYCL version ran slowly, and rebuilding it did not resolve the problem. He did not identify the underlying problem, leaving those particular setup failures as a constraint on the comparison.[1]
Pande adjusted attention processing and the amount of work handled in a small batch. He reserved the larger batch setting for lightweight models because bigger ones could exhaust available memory. A separate image component let the model accept visual input. His local workloads included home automation and document services, which he described using with this graphics-card setup.[1]
