MedVTok: a unique tokenizer for medical AIs to talk about all imaging modalities, both in 2D and 3D

Driven across 28 million images and 70,000 volumes from nine modes, it reaches the state of art on 24 test benches, from segmentation to reporting.

For a multimodal model to reason about a patient's condition, it must convert very different examinations — x-rays, scanner cuts, MRI — into a common representation. Current medical visual tokenizers are generally linked to a single modality or dimension, which requires the multiplication of poorly reusable representations.

MedVTok projects 2D images and 3D volumes in the same space of tokens. Two mechanisms contribute to this: a regularization that requires anatomical consistency from one cut to the next, absent pixel-by-pixel supervision, and alignment with several expert encoders that retains the knowledge specific to each modality.

Driven across 28 million images and 70,000 volumes covering nine modalities, the same tokenizer is used for classification, image retrieval, segmentation, synthesis, visual questions and answers and reporting generation, with state-of-the-art performance on 24 test benches.