The benchmark covers three types of graphs, four task families and 84 executable tools. Authors' conclusion: the quality of tool calls counts more than their number.
A team led by Jiarui Tan and Zhongjian Zhang proposed GABench, a testbed dedicated to the analysis of graphs by agents based on language models. The construction is based on an automatic task generation chain: 10,400 tasks in total, each with a verifiable ground truth, spread over three types of graphs and four categories of analysis, with 84 executable tools allowing the agent to access the data and perform its operations.
The results are harsh for the state of the art: current agents stumble over complex tasks, and performance depends heavily on the execution infrastructure chosen, not just the underlying model. The most directly actionable lesson concerns the use of tools — the quality of calls takes precedence over their quantity, with a proliferation of approximate calls degrading the result rather than improving it.
Read the full article on the original source
Go to the source →