IBM Research’s ALTK-Evolve extracts short behavioral rules from earlier agent attempts and adds them to later instructions without changing model weights. In controlled customer-service tests, those guidelines improved some measures of consistent task completion. The useful size of the rule set varied between retail and airline tasks, and customers were simulated by another language model.
IBM Research, the company’s research division, has evaluated ALTK-Evolve on customer-service conversations. The method extracts short rules from successful and unsuccessful agent attempts, covering actions such as authentication and transaction confirmation. It adds those rules to later instructions. The model’s weights remain unchanged, and rules learned from training tasks are carried into separate test tasks.[1]
Every task was attempted four times. The evaluation checks whether the final database state matches the expected result and whether the task succeeds consistently across those attempts. The researchers report improvements in retail tasks when learned guidelines are added. Another language model plays the customer, so these are simulated conversations rather than observations of a live customer-service operation.[1]
The amount of guidance mattered differently in the two domains. On airline tasks, a shorter list achieved more consistent success than the full set of learned guidelines, even though the full set gave higher average accuracy. Longer instructions increased the input-token expense. In matched conversations completed successfully under the two setups, the number of assistant turns stayed unchanged. The team reports held-out results separately from leaderboard comparisons that include training tasks.[1]
