Breast damage: RAG assistant finds good evidence in his first three results for 17 questions out of 18

Brazilian researchers also show that enriching the reference annotations changes the scores displayed without changing the ranking of the document search systems.

A research-augmented generation (RAG) system for the management of breast lesions must first find the correct passage in the medical literature. A Brazilian team has published on arXiv an evaluation protocol that takes into account how the reference "correct answers" are annotated. Their development bench has 18 documents and 18 questions, closed or open, one per document. The research combines dense embeddings, BM25 and cross-encoder reclassification, with segments from 128 to 1,024 tokens and different recovery rates.

The hybrid configuration chosen, with segments of 384 tokens and 20% overlap, places a valid passage among the top three results for 17 out of 18 questions, with scores of 0.94 on the main metrics (Recall@3, R-Precision, MRR@3, nDCG@3). By refining the annotations, the passages recognized as evidence have increased from 25 to 37 without the ranking of the systems moving, which shows how much the scores depend on the annotation. The authors acknowledge the small size of the test set and call for independent validation, especially by radiologists.