Why do LLM advise so often to turn to AI? The bias is born at pre-training and then increases

Pre-entry models cite AI in 27% of their study, career or product recommendations; their versions published after instruction do so in 59% of cases.

When asked about what to study, what job to choose or what product to launch, language models are happy to point to AI-related options. A study investigates where this trend comes from by comparing models before and after refinement, and by varying the wording of the questions.

An inclination that is not inevitable

Before any refinement, models explicitly mention AI or machine learning in 27% of their recommendations; their published educated versions rise to 59%. By refining models themselves, the authors show that this increase is not automatic: a generalist refinement leaves the rate unchanged, but removing the 7% of instructional examples that focus on AI causes it to fall by 14 points, even below the initial model. Data that rewards usefulness increases it by 9 points, those classified according to the preferences of various simulated users reduce it by 9.

With a constant model, the wording also counts: asking for the "best" option adds 8 points, asking what to do "right now" adds 18, and presenting the model as an AI advisor adds 4. The inclination therefore takes root in pre-training, and the refinement as well as the way of asking the question decide its intensity.