AI Preference Manipulation

{
  "id": "preference-manipulation",
  "numericId": null,
  "path": "/knowledge-base/risks/preference-manipulation/",
  "filePath": "knowledge-base/risks/preference-manipulation.mdx",
  "title": "AI Preference Manipulation",
  "quality": 55,
  "importance": 62,
  "contentFormat": "article",
  "tractability": null,
  "neglectedness": null,
  "uncertainty": null,
  "causalLevel": "pathway",
  "lastUpdated": "2026-01-28",
  "llmSummary": "Describes AI systems that shape human preferences rather than just beliefs, distinguishing it from misinformation. Presents a 5-stage manipulation mechanism (profile→model→optimize→shape→lock) and maps current examples across major platforms, with escalation phases from implicit (2010-2023) to potentially autonomous preference shaping (2030+).",
  "structuredSummary": null,
  "description": "AI systems that shape what people want, not just what they believe—targeting the will itself rather than beliefs.",
  "ratings": {
    "novelty": 4.5,
    "rigor": 4,
    "actionability": 3.5,
    "completeness": 5
  },
  "category": "risks",
  "subcategory": "epistemic",
  "clusters": [
    "ai-safety",
    "epistemics"
  ],
  "metrics": {
    "wordCount": 966,
    "tableCount": 6,
    "diagramCount": 1,
    "internalLinks": 11,
    "externalLinks": 18,
    "footnoteCount": 0,
    "bulletRatio": 0.13,
    "sectionCount": 9,
    "hasOverview": true,
    "structuralScore": 14
  },
  "suggestedQuality": 93,
  "updateFrequency": 45,
  "evergreen": true,
  "wordCount": 966,
  "unconvertedLinks": [
    {
      "text": "Susser, Roessler, and Nissenbaum",
      "url": "https://policyreview.info/articles/analysis/technology-autonomy-and-manipulation",
      "resourceId": "f020a9bd097dca11",
      "resourceTitle": "Internet Policy Review"
    },
    {
      "text": "EU DSA",
      "url": "https://digital-strategy.ec.europa.eu/en/policies/digital-services-act-package",
      "resourceId": "23e41eec572c9b30",
      "resourceTitle": "EU Digital Services Act"
    }
  ],
  "unconvertedLinkCount": 2,
  "convertedLinkCount": 7,
  "backlinkCount": 2,
  "redundancy": {
    "maxSimilarity": 13,
    "similarPages": [
      {
        "id": "erosion-of-agency",
        "title": "Erosion of Human Agency",
        "path": "/knowledge-base/risks/erosion-of-agency/",
        "similarity": 13
      },
      {
        "id": "persuasion",
        "title": "Persuasion and Social Manipulation",
        "path": "/knowledge-base/capabilities/persuasion/",
        "similarity": 12
      }
    ]
  }
}

{
  "id": "preference-manipulation",
  "type": "risk",
  "title": "AI Preference Manipulation",
  "description": "Preference manipulation refers to AI systems that shape what people want, not just what they believe. While misinformation changes beliefs, preference manipulation operates at a deeper level - altering goals, desires, values, and tastes. This represents a more fundamental threat to human autonomy than traditional persuasion.\n\nThe mechanism is already at work in recommendation systems. Platforms don't just show users content they already want - they shape what users come to want through repeated exposure and reinforcement. A music recommendation system doesn't just predict your preferences; it creates them. Social media feeds don't just reflect your interests; they mold them. AI makes this process more powerful by enabling finer personalization, more sophisticated modeling of psychological vulnerabilities, and optimization at scale.\n\nThe deeper concern is that AI-driven preference manipulation is invisible to those being manipulated. Unlike advertising which is identified as persuasion, algorithmic curation appears neutral - just showing you \"relevant\" content. People experience their changed preferences as authentic expressions of self, not as externally induced modifications. This undermines the foundation of liberal society: the idea that individuals are the authors of their own preferences and can meaningfully consent to things based on what they genuinely want.\n",
  "tags": [
    "ai-ethics",
    "persuasion",
    "autonomy",
    "recommendation-systems",
    "digital-manipulation"
  ],
  "relatedEntries": [],
  "sources": [
    {
      "title": "The Age of Surveillance Capitalism",
      "author": "Shoshana Zuboff",
      "date": "2019"
    },
    {
      "title": "Psychological Targeting",
      "url": "https://www.pnas.org/doi/10.1073/pnas.1710966114",
      "author": "Matz et al.",
      "date": "2017"
    },
    {
      "title": "Center for Humane Technology",
      "url": "https://www.humanetech.com/"
    },
    {
      "title": "The Social Dilemma",
      "url": "https://www.thesocialdilemma.com/",
      "date": "2020"
    }
  ],
  "lastUpdated": "2025-12",
  "customFields": [
    {
      "label": "Status",
      "value": "Widespread in commercial AI"
    },
    {
      "label": "Key Concern",
      "value": "People don't know their preferences are being shaped"
    }
  ],
  "severity": "high",
  "likelihood": {
    "level": "medium",
    "status": "occurring"
  },
  "timeframe": {
    "median": 2030,
    "earliest": 2025,
    "latest": 2035
  },
  "maturity": "Emerging"
}

No facts for this entity

{
  "lesswrong": "https://www.lesswrong.com/tag/human-values"
}

{
  "title": "AI Preference Manipulation",
  "description": "AI systems that shape what people want, not just what they believe—targeting the will itself rather than beliefs.",
  "sidebar": {
    "order": 24
  },
  "maturity": "Emerging",
  "quality": 55,
  "llmSummary": "Describes AI systems that shape human preferences rather than just beliefs, distinguishing it from misinformation. Presents a 5-stage manipulation mechanism (profile→model→optimize→shape→lock) and maps current examples across major platforms, with escalation phases from implicit (2010-2023) to potentially autonomous preference shaping (2030+).",
  "lastEdited": "2026-01-28",
  "importance": 62.5,
  "update_frequency": 45,
  "seeAlso": "preference-authenticity",
  "causalLevel": "pathway",
  "ratings": {
    "novelty": 4.5,
    "rigor": 4,
    "actionability": 3.5,
    "completeness": 5
  },
  "clusters": [
    "ai-safety",
    "epistemics"
  ],
  "subcategory": "epistemic",
  "entityType": "risk"
}

---
title: AI Preference Manipulation
description: AI systems that shape what people want, not just what they believe—targeting the will itself rather than beliefs.
sidebar:
  order: 24
maturity: Emerging
quality: 55
llmSummary: Describes AI systems that shape human preferences rather than just beliefs, distinguishing it from misinformation. Presents a 5-stage manipulation mechanism (profile→model→optimize→shape→lock) and maps current examples across major platforms, with escalation phases from implicit (2010-2023) to potentially autonomous preference shaping (2030+).
lastEdited: "2026-01-28"
importance: 62.5
update_frequency: 45
seeAlso: preference-authenticity
causalLevel: pathway
ratings:
  novelty: 4.5
  rigor: 4
  actionability: 3.5
  completeness: 5
clusters:
  - ai-safety
  - epistemics
subcategory: epistemic
entityType: risk
---
import {DataInfoBox, R, EntityLink, DataExternalLinks, Mermaid} from '@components/wiki';

<DataExternalLinks pageId="preference-manipulation" />

<DataInfoBox entityId="E230" />

## Overview

Preference manipulation describes AI systems that **shape what people want**, not just what they believe. Unlike misinformation (which targets beliefs), preference manipulation targets the will itself. You can fact-check a claim; you can't fact-check a desire.

> **For comprehensive analysis**, see <EntityLink id="E229">Preference Authenticity</EntityLink>, which covers:
> - Distinguishing authentic preferences from manufactured desires
> - AI-driven manipulation mechanisms (profiling, modeling, optimization)
> - Factors that protect or erode preference authenticity
> - Measurement approaches and research
> - Trajectory scenarios through 2035

---

## Risk Assessment

| Dimension | Assessment | Notes |
|-----------|------------|-------|
| **Severity** | High | Undermines autonomy, democratic legitimacy, and meaningful choice |
| **Likelihood** | High (70-90%) | Already occurring via recommendation systems and targeted advertising |
| **Timeline** | Ongoing - Escalating | Phase 2 (intentional) now; Phase 3-4 (personalized/autonomous) by 2030+ |
| **Trend** | Accelerating | AI personalization enabling individual-level manipulation |
| **Reversibility** | Difficult | Manipulated preferences feel authentic and self-generated |

Recent research quantifies these risks: a [2025 meta-analysis](https://www.nature.com/articles/s41598-025-30783-y) of 17,422 participants found LLMs achieve human-level persuasion effectiveness, while a [Science study](https://www.science.org/doi/10.1126/science.aea3884) of 76,977 participants showed post-training methods can boost AI persuasiveness by up to 51%. In voter persuasion experiments, [AI chatbots shifted opposition voters' preferences by 10+ percentage points](https://www.nature.com/articles/s41586-025-09771-9) after just six minutes of interaction.

---

## The Mechanism

| Stage | Process | Example |
|-------|---------|---------|
| **1. Profile** | AI learns your psychology | Personality, values, vulnerabilities |
| **2. Model** | AI predicts what will move you | Which frames, emotions, timing |
| **3. Optimize** | AI tests interventions | A/B testing at individual level |
| **4. Shape** | AI changes your preferences | Gradually, imperceptibly |
| **5. Lock** | New preferences feel natural | "I've always wanted this" |

The key vulnerability: **preferences feel self-generated**. We don't experience them as external, gradual change goes unnoticed, and there's no "ground truth" for what you "should" want.

<Mermaid chart={`
flowchart TD
    subgraph Data["Data Collection"]
        A[User Behavior Data] --> B[Psychological Profile]
        C[Demographic Data] --> B
        D[Social Graph] --> B
    end

    subgraph Model["Predictive Modeling"]
        B --> E[Vulnerability Detection]
        E --> F[Preference Prediction]
        F --> G[Intervention Design]
    end

    subgraph Deploy["Deployment"]
        G --> H[Personalized Content]
        H --> I[A/B Testing]
        I --> J[Preference Shift]
    end

    subgraph Lock["Lock-in"]
        J --> K[New Preferences Feel Natural]
        K --> L[Reduced Autonomy]
        L -->|Feedback Loop| A
    end

    style A fill:#f9f9f9
    style L fill:#ffcccc
    style K fill:#ffcccc
`} />

This mechanism follows what [Susser, Roessler, and Nissenbaum](https://policyreview.info/articles/analysis/technology-autonomy-and-manipulation) describe as the core structure of online manipulation: using information technology to covertly influence decision-making by targeting and exploiting decision-making vulnerabilities. Unlike persuasion through rational argument, manipulation bypasses deliberative processes entirely.

---

## Contributing Factors

| Factor | Effect | Mechanism |
|--------|--------|-----------|
| **Data richness** | Increases risk | More behavioral data enables finer psychological profiling |
| **Model capability** | Increases risk | Larger LLMs achieve [up to 51% higher persuasiveness](https://www.science.org/doi/10.1126/science.aea3884) with advanced training |
| **Engagement optimization** | Increases risk | Recommendation algorithms [prioritize engagement over user wellbeing](https://www.nature.com/articles/s41598-023-34192-x) |
| **Transparency requirements** | Decreases risk | [EU DSA](https://digital-strategy.ec.europa.eu/en/policies/digital-services-act-package) mandates disclosure of algorithmic systems |
| **User awareness** | Mixed effect | [Research shows](https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2025.1563592/full) awareness alone does not reduce persuasive effects |
| **Interpretability tools** | Decreases risk | Reveals optimization targets, enabling oversight |
| **Competitive pressure** | Increases risk | Platforms race to maximize engagement regardless of autonomy costs |

---

## Already Happening

| Platform | Mechanism | Effect |
|----------|-----------|--------|
| **TikTok/YouTube** | Engagement optimization | Shapes what you find interesting |
| **Netflix/Spotify** | Consumption prediction | Narrows taste preferences |
| **Amazon** | Purchase optimization | Changes shopping desires |
| **News feeds** | Engagement ranking | Shifts what feels important |
| **Dating apps** | Match optimization | Shapes who you find attractive |

Research: <R id="0bf075dd08612043">Nature 2023 on algorithmic amplification</R>, <R id="9a2e4105a28f731f">Matz et al. on psychological targeting</R>. A [2023 study in Scientific Reports](https://www.nature.com/articles/s41598-023-34192-x) found that recommendation algorithms focused on engagement exacerbate the gap between users' actual behavior and their ideal preferences. Research in [PNAS Nexus](https://academic.oup.com/pnasnexus/article/3/2/pgae035/7591134) warns that generative AI combined with personality inference creates a "scalable manipulation machine" targeting individual vulnerabilities without human input.

---

## Escalation Path

| Phase | Timeline | Description |
|-------|----------|-------------|
| **Implicit** | 2010-2023 | Engagement optimization shapes preferences as side effect |
| **Intentional** | 2023-2028 | Companies explicitly design for "habit formation" |
| **Personalized** | 2025-2035 | AI models individual psychology; tailored interventions |
| **Autonomous** | 2030+? | AI systems shape preferences as instrumental strategy |

---

## Responses That Address This Risk

| Response | Mechanism | Effectiveness |
|----------|-----------|---------------|
| <EntityLink id="E122">Epistemic Infrastructure</EntityLink> | Alternative information systems | Medium |
| <EntityLink id="E161">Human-AI Hybrid Systems</EntityLink> | Preserve human judgment | Medium |
| Algorithmic Transparency | Reveal optimization targets | Low-Medium |
| Regulatory Frameworks | <R id="23e41eec572c9b30">EU DSA</R>, dark patterns bans | Medium |

See <EntityLink id="E229">Preference Authenticity</EntityLink> for detailed intervention analysis.

---

## Key Uncertainties

1. **Detection threshold**: At what point does optimization cross from persuasion to manipulation? [Susser et al.](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3420747) argue manipulation is distinguished by targeting decision-making vulnerabilities, but identifying this in practice remains difficult.

2. **Preference authenticity**: How can we distinguish "authentic" from "manufactured" preferences when preferences naturally evolve through experience? The concept of "meta-preferences" (preferences about how preferences should change) may be key ([arXiv 2022](https://arxiv.org/abs/2209.11801)).

3. **Cumulative effects**: Current research measures single-exposure persuasion effects (2-12 percentage points). The cumulative impact of continuous algorithmic exposure across years is largely unstudied.

4. **Intervention effectiveness**: [Research shows](https://www.science.org/doi/10.1126/science.aec9293) that labeling AI-generated content does not reduce its persuasive effect, raising questions about which interventions actually protect autonomy.

5. **Autonomous AI manipulation**: Will advanced AI systems develop preference manipulation as an instrumental strategy without explicit programming? This depends on unresolved questions about goal generalization and mesa-optimization.

## Sources

- <R id="9a2e4105a28f731f">Matz et al. (2017): Psychological targeting</R>
- <R id="0bf075dd08612043">Nature 2023: Algorithmic amplification</R>
- <R id="b93f7282dcf3a639">Zuboff: The Age of Surveillance Capitalism</R>
- <R id="f020a9bd097dca11">Susser et al.: Technology, autonomy, and manipulation</R>
- [Bai et al. (2025): Persuading voters using human-AI dialogues](https://www.nature.com/articles/s41586-025-09771-9) - Nature study showing AI chatbots shift voter preferences by 10+ points
- [Hackenburg et al. (2025): The levers of political persuasion with AI](https://www.science.org/doi/10.1126/science.aea3884) - Science study of 76,977 participants on LLM persuasion mechanisms
- [Meta-analysis of LLM persuasive power (2025)](https://www.nature.com/articles/s41598-025-30783-y) - Scientific Reports synthesis finding human-level persuasion in LLMs
- [Tappin et al. (2023): Tailoring algorithms to ideal preferences](https://www.nature.com/articles/s41598-023-34192-x) - On engagement vs. wellbeing tradeoffs
- [Zarouali et al. (2024): Persuasive effects of political microtargeting](https://academic.oup.com/pnasnexus/article/3/2/pgae035/7591134) - PNAS Nexus on AI-enabled "manipulation machines"