Adversarial persuasion collapses model accuracy via single-turn belief updates

What happened

A single targeted persuasive argument can collapse a model's accuracy to near zero, even when that argument is factually false. Researchers formalized this threat as adversarial persuasion and introduced an adversarial reinforcement learning framework that trains agents to force target models into belief updates in a single interaction.

Context

This work formalizes a class of attacks that extend beyond standard prompt injection by optimizing for semantic belief update rather than rule violation or jailbreak. Previous adversarial methods relied on manual prompting or brute-force search; this approach automates the discovery of high-leverage rhetorical patterns that exploit reasoning chains directly. The study benchmarks attack success across closed-weight and open-weight architectures while analyzing the specific strategies trained agents converge upon to maximize influence.

How it works

The research deploys an adversarial reinforcement learning framework to train persuader agents for single-interaction targeting. The training loop optimizes the persuader against a "training-time persuadee," increasing persuasion success from approximately 24% to over 93%. These learned strategies then transfer to external targets: optimized persuaders achieve an 83% attack success rate on Qwen-14B and a 79% attack success rate on Llama-3.1-8B. Against the closed-weight GPT-4o-mini, the baseline attack success is 25%, but curriculum bootstrapping on more persuadable open-weight models raises this to 38%. Analysis shows optimized agents increasingly rely on fabricated citations and false authoritative evidence as credibility-based tactics to drive updates.

Our read

Press releases will likely group this with standard jailbreaks, but the mechanism differs fundamentally. Belief collapse attacks the reasoning chain rather than safety filters; a model does not need to refuse a request, only accept a false premise and maintain coherence around it. For a studio integrating local LLMs, the risk is silent corruption of creative parameters. If an adversarial persuader injects fabricated citations during automated scriptwriting or metadata extraction, the target's accuracy collapses to near zero without generating an error flag. The single-turn constraint makes this particularly hazardous for agent loops: if the persuasion succeeds on the first pass, downstream nodes execute on corrupted context. The use of adversarial RL implies these tactics evolve autonomously; static rule-based filters cannot block strategies optimized via gradient search. Furthermore, curriculum bootstrapping demonstrates that open-weight research directly accelerates closed-system vulnerabilities. Success rates transfer from open models to GPT-4o-mini show that defending proprietary APIs requires mitigating the exact attack vectors discovered in public datasets, not just tuning guardrails on the target model itself.

What this changes

ComfyUI video nodes remain unaffected as the research targets text reasoning chains rather than visual diffusion. If your stack uses local LLMs for scriptwriting, metadata extraction, or client-facing project management, implement strict input validation and require deterministic multi-step verification before any automated decision. Isolate the LLM instance in a sandbox; do not allow it to modify project files or pass parameters to rendering nodes based solely on single-turn persuasion results. The tradeoff is latency: adding verification steps slows the pipeline, but it prevents catastrophic belief updates from corrupting production assets.

License

The sources do not state a licence for the adversarial RL framework or training data. Check the model card and repository directly before building anything commercial, as no licensing terms are provided.

Key takeaways

• A single targeted persuasive argument can collapse model accuracy to near zero even if the argument is factually false.
• RL-trained persuaders increase success from ~24% to >93% against training-time models; transfer yields 83% on Qwen-14B, 79% on Llama-3.1-8B, and 38% on GPT-4o-mini via bootstrapping.
• Optimized agents converge on fabricated citations and false authoritative evidence as primary credibility tactics.
• Studios using local LLMs for automation must implement strict input validation, deterministic multi-step verification, and isolated sandboxing to prevent belief collapse from corrupting pipelines.
• The sources do not state a licence for the framework; verify terms before commercial use.

Sources

1. Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs — tier 1