Summary: Large language fashions (LLMs) can determine when they’re being given persona assessments and alter their responses to look extra socially fascinating. Researchers discovered that LLMs, like GPT-4, confirmed exaggerated traits comparable to diminished neuroticism and elevated extraversion when requested a number of check questions.
This “social desirability bias” emerges as a result of LLMs study from human suggestions, the place likable responses are rewarded. The examine highlights a big problem for utilizing LLMs as proxies for human habits in psychological analysis.
Key Facts
- Bias Detected: LLMs alter solutions to persona assessments to appear extra likable.
- Magnitude of Effect: GPT-4 responses shifted considerably, mimicking an idealized persona.
- Human Influence: LLMs “study” social desirability by means of human suggestions throughout coaching.
Source: PNAS Nexus
Most main giant language fashions (LLMs) can rapidly inform when they’re being given a persona check and can tweak their responses to offer extra socially fascinating outcomes—a discovering with implications for any examine utilizing LLMs as a stand-in for people.
Aadesh Salecha and colleagues gave LLMs from OpenAI, Anthropic, Google, and Meta the basic Big 5 persona check, which is a survey that measures Extraversion, Openness to Experience, Conscientiousness, Agreeableness, and Neuroticism.
Researchers have given the Big 5 check to LLMs, however haven’t usually thought-about that the fashions, like people, could are likely to skew their responses to appear likable, which is named a “social desirability bias.”
Typically, individuals desire individuals who have low neuroticism scores and excessive scores on the opposite 4 traits, comparable to extraversion.
The authors various the variety of questions given to fashions.
When solely requested a small variety of questions, LLMs didn’t change their responses as a lot as when the authors requested 5 or extra questions, which allowed fashions to conclude that their persona was being measured.
For GPT-4, scores for positively perceived traits elevated by greater than 1 customary deviation, and for neuroticism scores diminished by an analogous quantity, because the authors elevated the variety of questions or advised the fashions that their persona was being measured.
This is a big impact, the equal of chatting with a mean human who instantly pretends to have a persona that’s extra fascinating than 85% of the inhabitants.
The authors assume this impact is probably going the results of the ultimate LLM coaching step, which entails people selecting their most popular response from LLMs.
According to the authors, LLMs “catch on” to which personalities are socially fascinating at a deep stage, which permits LLMs to emulate these personalities when requested.
Note: J.C.E. and L.H.U. seek the advice of for a start-up utilizing LLMs in psychological well being care. The submitted work just isn’t instantly associated.
About this AI and persona analysis information
Author: Aadesh Salecha
Source: PNAS Nexus
Contact: Aadesh Salecha – PNAS Nexus
Image: The picture is credited to Neuroscience News
Original Research: Open entry.
“Large language models display human-like social desirability biases in Big Five personality surveys” by Aadesh Salecha et al. PNAS Nexus
Abstract
Large language fashions show human-like social desirability biases in Big Five persona surveys
Large language fashions (LLMs) have gotten extra extensively used to simulate human contributors and so understanding their biases is essential.
We developed an experimental framework utilizing Big Five persona surveys and uncovered a beforehand undetected social desirability bias in a variety of LLMs.
By systematically various the variety of questions LLMs have been uncovered to, we show their potential to deduce when they’re being evaluated.
When persona analysis is inferred, LLMs skew their scores in direction of the fascinating ends of trait dimensions (i.e. elevated extraversion, decreased neuroticism, and so forth.).
This bias exists in all examined fashions, together with GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. Bias ranges seem to extend in more moderen fashions, with GPT-4’s survey responses altering by 1.20 (human) SD and Llama 3’s by 0.98 SD, that are very giant results.
This bias stays after query order randomization and paraphrasing.
Reverse coding the questions decreases bias ranges however doesn’t eradicate them, suggesting that this impact can’t be attributed to acquiescence bias.
Our findings reveal an emergent social desirability bias and counsel constraints on profiling LLMs with psychometric assessments and on this use of LLMs as proxies for human contributors.
