BioEVAL: 22 teams are developing a PhD-level bioengineering exam; the models’ accuracy peaks at 90 per cent

608 tasks across eleven sub-areas, including literature reviews and experimental images to be interpreted. There remain considerable differences between the sub-areas.

On 24 September, a group of 22 research groups from around the world published BioEVAL, a test bed designed to measure what large language and multimodal models can really do in bioengineering. The collection includes 608 tests calibrated at the doctoral level and spread over eleven sub-fields of the discipline.

The protocol is not limited to the multiple-choice questionnaire, even if it forms the bulk of it: 359 questions selected after a quality audit, to which are added 218 literature synthesis tasks and 10 multimodal problems requiring the interpretation of experimental images. This structure aims precisely to separate the restitution of knowledge from the work of critical reading and analysis of raw data.

The results give up to 90% correct answers on the MCQs, a similarity score of 0.72 on the literature synthesis and 80% accuracy on the small multimodal sample. The most useful observation, however, concerns the dispersion: performance varies greatly from one sub-field to another, a sign that the overall competence covers very unequal abilities depending on the specialty.