AI Act: VERA, an open source dashboard tested in a large European bank to evaluate LLM before deployment

The tool translates into scores readable by non-engineers twelve technical requirements of the AI Act measurable by benchmarks, and also measures the energy consumed by each assessment.

Before authorizing a language model in a regulated sector, institutions must gather technical evidence of responsible practices. The COMPL-AI framework proposes a technical reading of the European AI Act into six sets of principles and eighteen requirements, twelve of which can now be measured by benchmarks. Problem: these test batteries fly on the command line, while people who validate deployments are usually not engineers.

Required scores, and energy footprint

A team of researchers presents VERA, a self-hosted and open source dashboard deployed in a large European bank. The manager selects a model, launches public benchmarks and scores for each of the twelve measurable requirements, together with the energy consumed by each assessment, which also documents the environmental impact. Four open-weight models were screened.

In an internal study, reading results via the dashboard took an average of 30 seconds, compared to 56 seconds from raw files, and participants noted its usefulness 4.36 out of 5. The study code and material are published so that other organizations can replicate the approach.