Astro2Method: LLMs know how to choose a learning method for astronomy in simple cases, much less in difficult cases

Out of 500 questions from 346 astronomy articles, five models reach nearly 90% of the right answers in easy versions, but cap around 60% as soon as the authors' explanation is only partial.

Machine learning has become a common tool in astronomy, but choosing the right method for a given problem requires dual competence, astrophysics and computer science. The great language models could advise the researchers on this point: Xingzhong Fan, Hangfeng He and Fatemeh Nargesian wanted to check it with Astro2Method, a set of 500 binary questions validated by experts, built from 346 articles published between January 2024 and June 2026.

90% to 60%

When interviewed directly, five LLMs receive between 87.5 and 92.6% of the correct answers on "easy" questions, where the authors of the source article fully justify their choice, but only 58.9 to 64.5% on "hard", where this justification is partial. Neither the generation augmented by research on previous articles, nor most of the agentic methods (ReAct, Self-Ask, CRITIC) bring regular gain.

Agents allowed to search on the Web progressed especially when they found the article from which the question was drawn. Providing the description of the experimental pipeline, names of masked methods, on the other hand significantly improves difficult cases. The authors conclude that the models benefit from relevant experimental information, but are still struggling to deduce them themselves from neighbouring work and knowledge of the field.