A team at ETH Zurich evaluated the real usefulness of repository-level context files: the instructions are followed well, but the general repository descriptions are useless.
Files AGENTS.md, placed at the root of a repository to give context to coding agents, have become a practice recommended by model providers. Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev and Martin Vechev sought to know if they actually work, in a work entitled "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? "
The evaluation combines two approaches: SWE-bench tasks with context files generated by a language model, and a new collection including files written and validated by developers. The main result is against the established practice: these files generally do not improve success rates, while increasing the inference cost by an average of more than 20%.
The detail is more instructive than the overall verdict. The authors note that the instructions contained in these files are well followed by the agents — so the problem does not come from inattention of the model. It is the general descriptions of the deposit, yet the most widespread and explicitly recommended part, that prove to be useless: the agent pays the context without profiting measurable.
Read the full article on the original source
Go to the source →