A new measure, the response coverage gap, shows that six LLMs leave behind entire profiles of true respondents, to the point of doing less than a random response draw.
"Silicon sampling" consists of having a language model answer a questionnaire by assigning it a demographic profile, to constitute a synthetic population. These samples are increasingly used to replace human panels. But the usual evaluations compare the accuracy of individual answers, their distribution question by question or the correlations between questions. None of them compares the respondents themselves: a generator can therefore obtain good scores while producing combinations of answers that no human would give.
Measuring distance to real respondents
The authors propose the respondent coverage gap (RCG): each respondent becomes a point defined by all of his or her responses, and we measure how far each human is from the nearest synthetic respondent, compared to the usual distance between two human samples.
In two major American social surveys, the six LLMs evaluated leave unrivalled human respondents. In the main settings, they do even worse than a naïve reference that draws each answer at random according to the proportions observed in humans, and regardless of the mode of prompt or inference tested, a simple statistical model (Gaussian copula) fitted to human data is ahead of them all.
Read the full article on the original source
Go to the source →