Mistral, Gemini Flash, Claude Sonnet and Claude Opus have noted 40 decisions by four administrations and 15 hypothetical reforms. Three discreet changes in service have shifted their opinions.
More and more citizens are asking chatbots for their opinion on public affairs. However, these opinions must be stable. Oleg Vasilyev therefore queried, via the consumer interfaces and for 30 days, five configurations (Mistral, Gemini Flash, Claude Sonnet and Claude Opus with two levels of effort), twice a day. Each was to note the long-term effect for the United States of 40 decisions of the last four presidential administrations and 15 hypothetical reforms, with a level of confidence and a brief comment.
A solid consensus, invisible excesses
The results are robust: all configurations agree on the sign and on the ranking of the average scores of the administrations, and two chatbots surveyed on the same day only give opinions of opposite signs on the same decision in about 4% of cases. The differences between days are larger than within the same day, but increase slowly over the month.
One notable exception: the author identifies three changes to the service itself that have moved the judgments without leaving a visible trace, or almost, in the interface. The comments, analyzed as a documentary research task, remain specific to each subject and consistent from one chatbot to another. The set of 16,500 judgments is published.
Read the full article on the original source
Go to the source →