🔗 Not all model upgrades are upgrades

A surprising evaluation of model upgrades reveals that newer does not always mean better. Testing Claude Sonnet 4.6 against Sonnet 5 showed that despite Sonnet 5 boasting a 33% lower per-token cost, it burned through tokens at a staggering 12x rate compared to its predecessor for architecture tasks, resulting in higher overall costs. In fact, while Sonnet 4.6 outperformed Sonnet 5 in output quality on most architecture tasks, the latter excelled in process-driven tasks like code upgrades. Notably, Sonnet 5 made fewer mistakes in following specific instructions. However, neither model addressed inherent content gaps, highlighting that sometimes the content, rather than the model, dictates success. This evaluation underscores the importance of measuring workload specifics before making a switch and emphasizes that well-documented information is crucial for performance.