A small model with selectively active components
OVHcloud, a cloud services company, announced Olala mini 7A1B on 17 November as the first member of its language-model family. The model has 7 billion parameters, the numerical values learned during training, with 1 billion active for a request. Its planned Apache 2.0 distribution includes downloadable weights. The official launch is scheduled for 19 November.[1], [2]
Each layer contains 256 specialist components, with only part of the model activated for an individual request. The initial context window, which sets the amount of text handled at once, is thirty-two thousand tokens, or text units. OVHcloud combines Differential-TPA and Mamba-3 in the model’s hybrid attention architecture.[1]
