Large language models are often asked to help users make choices, from what to cook for dinner to higher-stakes decisions, such as screening job applicants or supporting medical triage. As AI is increasingly integrated into consequential decisions, concerns have been raised that the models may share some human biases, including racial and gender preferences. Haonan Yin, Shai Vardi, and Vidyanand Choudhary investigated another sort of bias: preference toward certain items depending on the order in which they are presented. The authors tested nine widely used LLMs (GPT-4o-mini, GPT-4.1-nano, Claude 3 Haiku, Claude Sonnet 4, Llama 3 8B, Llama 4 Scout, Gemini 2.5 Flash, Gemini 3 Flash, and Qwen 3 32B) in a low-stakes color-selection task—specifically, choosing a paint color for a child’s bedroom. Three of these models were also evaluated in a resume-screening task. A quality-dependent selection pattern emerged: when all options are high quality, models favor the first option, but when quality is lower, they favor later options. Crucially, the authors also found evidence that position did not merely serve as a tie-breaker when the model was indifferent. In some cases, changing the order reversed the model’s underlying preference and caused it to select an option it otherwise preferred less. Unexpectedly, the models also simply preferred some names over others, no matter which resume details they were attached to, despite the authors using demographically similar names. For example, Claude 3 Haiku selected “Christopher Taylor” over “Andrew Harris” in 64% of cases. According to the authors, while LLMs share many human biases, the models also have a few biases of their own. These uniquely model-specific biases may be harder to predict and thus harder to prevent. The authors also propose a temperature-based diagnostic strategy. Repeatedly querying a model at higher temperatures can help reveal latent or unstable preferences that remain concealed at lower temperatures and identify decisions that are especially sensitive to presentation order, according to the authors.
PNAS Nexus
Fragile preferences: A deep dive into order effects in large language models
11-Aug-2026