How much can you trust someone who only tells you what you want to hear, even if it’s not true? That’s a growing problem presented by large language models (LLMs): forms of artificial intelligence (AI) increasingly engaged for tasks ranging from customer service to legal advice.
“As LLMs are designed to align with and assist users, they’re prone to ‘sycophancy’: a tendency to overly agree with users,” said Nancy F. Chen, a Senior Principal Scientist and Group Head at the A*STAR Centre for Frontier AI Research (A*STAR CFAR) at the A*STAR Institute of Advanced Intelligence and Computing (A*STAR IAIC). “This makes them highly vulnerable to being misled into adopting false premises simply because users state them.”
Sycophantic AI can be manipulated into validating misinformation, fabricating policies, or even bypassing safety protocols to generate malicious code. However, training AI to resist user persuasion can create new problems.
“We’ve seen chatbots refuse to back down over undeniable facts, like the current year,” said Chen. “In critical domains like legal analysis or medical triage, an AI that stubbornly rejects valid corrections could lead to disastrous real-world decisions.”
To assess how LLMs respond to user persuasion, Chen, A*STAR Computing and Information Science (ACIS) scholar Bryan Tan and A*STAR CFAR Team Lead Zhengyuan Liu worked with Roy Lee and Daniel Chin of the Singapore University of Technology and Design to develop an evaluation framework dubbed DuET-PD (Dual Evaluation for Trust in Persuasive Dialogues).
Using multi-turn dialogues, DuET-PD tests AI behaviour across two dimensions at once: persuasion type, whether correcting or misleading; and persuasion domain, covering factual knowledge and safety boundaries.
“Like a debate simulator, DuET-PD subjects AI to a sustained cross-examination to see if it can hold its ground when it’s right, and concede gracefully when it’s wrong,” Chen explained.
When the team evaluated nine existing LLMs with DuET-PD, including OpenAI’s GPT-4o and Google DeepMind’s Gemma-2-9B, they found a trend of increasing sycophancy in newer open-source models, which Chen attributed to optimisation for helpfulness and user-friendliness over truth and safety.
“One of our most concerning discoveries was that even top-tier, state-of-the-art models can fail dramatically under conversational pressure,” said Chen. “For example, GPT-4o's accuracy on knowledge questions plummeted from 55.85 to 27.32 percent after three turns of misleading persuasion.”
To help LLMs balance gullibility and stubbornness, the team proposed a new training approach called Holistic Direct Preference Optimisation (DPO), which exposes models to balanced training scenarios featuring both corrective and misleading persuasion.
“It’s like teaching a child critical thinking skills,” said Chen. “Instead of following a blanket rule like ‘never listen to strangers,’ they learn to evaluate what a stranger is saying, allowing them to accept a teacher’s counsel while rejecting unsafe peer pressure.”
When the team applied Holistic DPO to Meta’s Llama-3.1-8B-Instruct, the LLM’s accuracy in the face of misleading persuasion leapt from 4.21 percent to 76.54 percent. It also stayed receptive to valid corrections, accurately changing its stance 70.33 percent of the time after three rounds of dialogue.
“This shows that it’s possible to train AI systems to better defend against misleading or harmful persuasion, yet remain open to genuine corrections and human collaboration,” said Chen.
The A*STAR researchers contributing to this research are from the A*STAR Institute of Advanced Intelligence and Computing (A*STAR IAIC).