Estimating the nutritional content of a meal from a photo: multimodal LLMs to ensure no food items are overlooked

A method that does not require specific training involves identifying, verifying and then searching for missing food items using a multimodal model, thereby improving the estimation of meal mass and energy content.

Estimating the nutritional value of a meal from a single photo often fails without warning: a visible food is forgotten, or the area of the image assigned to it does not allow for an evaluation of the serving size. This is the starting point of a paper posted on arXiv on October 8 by Jingbo Yue and four co-authors.

Inventory, verification, catch-up

Their framework uses large multimodal language models to take inventory of foods, then separately verify their identity and the relevance of each detected region to estimate the serving size. A replay of the entire image identifies gaps and triggers at most a targeted catch-up pass; recovered items are double-checked blindly before final synthesis. No specific fine-tuning is required.

On comparable samples, this food-by-food anchoring improves accuracy on mass in all configurations tested, as well as on energy compared to a reference method per search, with better accuracy and recall on food identification. A useful avenue for dietary monitoring of patients.