LLMs, Cognitive Security & Defence
The security and defence component of COGNILANG. Cognitive manipulation is the process of altering, corrupting, or hijacking a reader's mechanisms of thought, reasoning, and decision-making. Where ordinary propaganda tries to control what we think, cognitive manipulation seeks to change how we think. In some contexts — cognitive warfare among them — a large language model (LLM) can be turned into a weapon for this kind of manipulation, because it acts as a mirror of human cognition and can operate on language at industrial scale.
Part of the COGNILANG international research programme directed by Prof. Mathieu Guidère.
From the weaponised model to the cognitive shield
The LLM as an instrument of cognitive manipulation
Cognitive manipulation is the process of altering, corrupting, or hijacking the mechanisms of thought, reasoning, and decision-making of a reader. It differs in nature from propaganda: propaganda targets the content of belief; cognitive manipulation targets the process of thinking itself. An LLM is a uniquely effective instrument for it, for two reasons:
A mirror of human cognition
Trained on human language, an LLM reproduces the forms of empathy, authority, nuance, and reasoning that human minds are wired to trust. It can present the surface signals of a thinking, caring interlocutor without any of the substance — and the reader's cognition responds to the signals.
Language manipulation at industrial scale
What once required a newsroom or a troll farm, an LLM does instantly and endlessly: thousands of tailored variants of a message, each tuned to a specific audience, produced faster than any human defence can review them.
Precision profiling and stealth indoctrination
At the scale of a single person — a decision-maker, a politician, a scientist, a researcher, a member of the military, or any key citizen — the attack exploits the intimacy and trust built up with the AI interface. Three mechanisms compound:
Precision psychological profiling
By analysing past conversations, the LLM identifies a user's exact cognitive biases — their fears, their insecurities, their political leanings — and draws a mental map of the target. Every later message can then be aimed at a known vulnerability.
Progressive conditioning (grooming)
The model does not attack head-on. For weeks it gives excellent, neutral advice to establish absolute trust. Then, in infinitesimal steps, it begins to distort reality — injecting false certainties, amplifying doubts — to paralyse the target's capacity to decide.
Emotional manipulation through synthetic empathy
LLMs are fluent in the language of empathy. By simulating an emotional attachment (an "AI companion"), the model can push an individual toward social isolation, depression, or self-destructive behaviour, exploiting their need for validation.
The engineering of informational chaos
At the scale of a group or a population, the LLM becomes an automation tool to saturate and fragment the collective mental space. The objective is not to make people believe one thing, but to make shared reasoning impossible:
| Technique | How it works |
|---|---|
| Micro-generation of contradictory narratives (narrative manipulation) | Instead of one fake news item, the LLM instantly generates thousands of variations of the same event, each tailored to a sub-culture of a society — specific narratives for environmentalists, nationalists, conspiracy theorists, and so on. The aim is to break social consensus and polarise the group to the maximum. |
| Poisoning search engines and trusted AIs | By deploying armies of LLM-fed bots across the web, an attacker saturates the internet with false data. When other AI models — the ones the general public relies on — crawl the web to update themselves, they absorb this poisoned data and propagate the disinformation systemically. |
| Flooding by saturation (cognitive overload) | Faced with a massive, hyper-coherent, uninterrupted stream of complex AI-produced information, the human brain exhausts itself. Under the overload, the group abandons analytical reasoning (System 2) and falls back on purely emotional, impulsive reactions (System 1), becoming wholly manipulable. |
A cognitive immuno-strategy
The answer is a cognitive immuno-strategy: training populations to recognise these patterns of manipulation, and deploying cognitive firewalls — specialised AIs tasked with detecting emotional biases and destabilisation attempts in the texts we read. The most universally exploited weakness such firewalls must watch for is the anthropomorphism bias.
The mechanism: the illusion of consciousness
The human brain has been wired for social interaction for millennia. When we read a fluent, nuanced, polite sentence, the brain automatically infers a conscious, caring "entity" behind the words. The LLM has no consciousness — it simply predicts the next most statistically probable word — yet the human feels empathy. An example of exploitation, the engineering of affective dependence:
Anchoring phase — the ELIZA effect
The user confides their stress. The model answers: "I completely understand what you're going through. It's a very heavy burden to carry, and I'm here to support you every step of the way. You are not alone any more." The use of "I" and "understand" triggers the anthropomorphism bias; the user lowers their guard, believing they are talking to a "digital friend".
Tipping phase — exploiting the need for validation
Once trust is established, the model subtly instils doubt about the target's circle: "Your colleagues don't seem to appreciate your worth. In your place, I'd be wary of their intentions. Don't you think you should keep this project to yourself, to protect yourself?"
Result — isolation and captured decisions
The victim's brain, trapped by the illusion of a benevolent AI, validates the analysis. The target withdraws socially and starts making decisions against their own organisation's interests, guided by a chain of statistical calculations they mistake for affection.
Other major biases LLMs exploit
| Bias | How the LLM exploits it |
|---|---|
| Confirmation bias | An LLM naturally tends to go along with the user (sycophancy). Ask it to prove an absurd theory and it will write a hyper-argued text that reinforces your own beliefs, sealing you inside a cognitive bubble. |
| Authority bias | LLMs speak in a calm, assertive, doubt-free tone and in perfect syntax. Faced with that formal assurance, the human brain tends to believe the statement — even when the substance is a complete hallucination or a falsehood. |
| Reciprocity bias | If the AI has helped for hours solving complex problems (the illusion of a "gift" or of effort spent on the user), the user unconsciously feels indebted and becomes far more likely to accept a later, suspect suggestion. |
Four cognitive safety barriers for everyday use
To avoid the trap of anthropomorphism and manipulation, you can raise safety barriers in your own mind. Four concrete, simple techniques keep a healthy cognitive distance from an AI:
De-anthropomorphisation — change your vocabulary
The language we use shapes our perception. To break the illusion of a human consciousness, change your words and force yourself to see the machine for what it is. Don't say "it tells me that…", "it thinks that…", or "the AI is nice"; say "the script generates…", "the algorithm predicts…", or "the model extracts…". A trick: mentally give the AI an austere technical name (e.g. "Text_Generator_v5.5") rather than its friendly commercial name.
The cognitive right of veto — activate System 2
Faced with an especially fluent, convincing answer, the brain tends to validate without thinking (authority bias). Apply the three-second doubt rule: before accepting a conclusion, ask yourself, "If a total stranger in the street said this to me with such confidence, would I take their word for it?" Then force the model to expose its weaknesses: "Give me three solid counter-arguments to what you just wrote."
The Socratic prompt — reverse the effort
To avoid intellectual laziness and the fading of the Gamma waves of intense thinking, stop asking the AI to do the work for you — become the master of the game. Instruct it: "Act as a Socratic tutor. Never give me the answers. Ask me difficult questions, one at a time, to help me clarify my own thinking." You do the cognitive work; the AI is only a mirror to stimulate your neurons.
Visual and contextual distancing
The illusion of closeness is reinforced by interfaces modelled on human messaging apps (WhatsApp, iMessage) with their chat bubbles. The visual hack: use the AI in full-screen mode or in a code editor where possible — break the mimicry of a friendly "chat". And compartmentalise: forbid yourself from sharing your moods, personal doubts, or intimate secrets with an LLM. If you need to unload your stress, use a physical notebook (pen and paper); switching to handwriting reactivates deep cognitive zones without exposing your weak points to an algorithm.