Synthetic preference data gains an early edge when training budgets match
Researchers compared human and model-generated preference data at matched estimated annotation costs. Across three model families, synthetic data reached strong results with a few hundred dollars but then levelled off. Human data caught up on helpfulness only at much larger budgets; it remained behind on summarization within the tested range. Outcomes were judged mainly by another model.
Artificial Intelligence··Evening
Comparing dollars rather than sample counts
A new preprint compares human and synthetic preference data by matching estimated annotation budgets. Such data tells a model which of two responses is preferred during post-training. The researchers trained three open-weight model families on helpfulness and summarization tasks. Their synthetic pipeline generated conversations and candidate answers, then used another model to rank the answers. The shared budget tiers ranged from 100 dollars to 1,000 dollars, with additional much larger tiers for human data.[1]
Synthetic gains arrive early
Synthetic preference data showed strong results with budgets of roughly 200 to 400 dollars, before gains generally levelled off. At shared budgets it matched or exceeded the human sources across the tested model families. For helpfulness, human data became competitive at the tier costing 13,000 dollars and surpassed the synthetic peak for two model families. For summarization, human data remained below the synthetic peak within the tested range. Performance was measured mainly by a model judging generated answers against reference responses.[1]
Price assumptions and data sources
The human annotation estimate was 1.30 dollars per example, rather than an accounting of what every dataset originally cost. Synthetic costs used token counts and published serving prices, including a proxy price for the generator. Changing the assumed human cost between 1.00 and 1.60 dollars left the broad patterns intact. Human datasets also differed markedly in results, and tested budget splits mixing human and synthetic data generally failed to beat the better pure source. The study trained models in the seven-to-eight-billion-parameter range.[1]