Together Compiler retunes AI inference through engines that stay running
Together AI has introduced Together Compiler, an agent-driven system for continually tuning inference as models, hardware and traffic change. It keeps inference engines running while testing replacement computations, rather than restarting them for every experiment. The company reports production use on its own workloads and is offering customers early access through a limited beta.
Artificial Intelligence··Midday
Together announces a compiler with limited customer access
Together AI, an artificial intelligence infrastructure provider, introduced Together Compiler on 4 November as an implementation of its Polymorphic Inference approach. The company says it already uses the system on production inference workloads and some serverless endpoints. Customers can request early access through a limited beta. The system uses software agents to seek optimizations matched to the model, hardware and changing pattern of requests.[1], [2]
Engines remain running while agents test candidates
Instead of restarting an inference engine for every experiment, Together Compiler keeps an engine loaded and inserts candidate computations while it runs. Multiple agents generate ideas in parallel and coordinate through a shared message board. This supplies experiments while other agents are still reasoning or using tools. The inputs include the traffic shape as well as the model and the available hardware.[1]
Measurement moves from a few layers to realistic workloads
The compiler uses faster engines that load only a few model layers to try candidates quickly. Slower engines load more layers to check whether an optimization holds up on realistic data distributions. A separate cost model combines hardware characteristics, small tests and measured computations to narrow the choices for distributing work. Together says changing prompt lengths, routing behavior and request bursts make continual retuning relevant to production serving.[1]
