Up to 63% of the tools presented at CFSI 2026 beaten by a single prompt on a newer model

One team replayed 35 LLM-based software engineering works: in 37 to 63 percent of cases, a single prompt on a next-generation model did better than the tooling released a year earlier.

The study, by Nahian Salsabil, Joy Saha, Simantika Bhattacharjee Dristi, Nicholas Phair, Nusrat Jahan Mozumder, Matthew B. Dwyer and Sebastian Elbaum, is based on an intuition that many practitioners whisper that some of the engineering built around language models only compensates for their current weaknesses. The authors put it to the test by taking 35 contributions presented at ICSE 2026, all built on LLMs, and confronting them with the poorest alternative imaginable: a single, automatically generated prompt executed on a next-generation model, without any iterative refinement.

Depending on the measure chosen, between 37% and 63% of this work is overtaken by this single prompt. Substitution works particularly well on so-called constructive tasks — code generation, program repair — where the raw capacity of the model carries the bulk of the result. The scaffolding developed a year earlier then becomes an expensive detour to a result that the model now achieves on its own.

The authors' conclusion is not about the quality of the work examined but about the choice of subjects: investing in techniques designed as circumvention of temporary deficits produces research with a short lifespan. They argue for focusing on the difficulties that survive the change in model — evaluation, guarantees, integration into production chains — rather than on those that the next generation will erase on its own.