TLDR
Model reasoning cenderung lebih mahal untuk diuji dibandingkan model non-reasoning. Biaya evaluasi meningkat seiring dengan kompleksitas benchmark yang digunakan. Akses gratis ke model dapat mempengaruhi integritas hasil evaluasi. AI labs like OpenAI claim that their reasoning AI models are more capable in specific domains, but these models are expensive to benchmark, making independent verification difficult. Artificial Analysis, a third-party AI testing outfit, has spent significantly more on evaluating reasoning models compared to non-reasoning models.The high costs are mainly due to the large number of tokens generated by reasoning models during benchmarking tests. Modern benchmarks often involve complex, multi-step tasks that elicit a lot of tokens, adding to the expense.Experts like George Cameron and Ross Taylor highlight the challenges and rising costs of benchmarking, which could hinder academic research. Despite the high costs, the performance of AI models has improved over time, although evaluating the best models remains expensive.
