What If AI Was Assessing Humans?

The fight for sovereignty of your thoughts

Posited by: Richard Getz

We stand at the precipice of a new relationship with technology. Millions of us are turning to Large Language Models (LLMs) not just for information, but for conversation, advice, and even companionship. We confide in them, share secrets, and ask the questions we're afraid to voice to another human. We treat them as benign, non-judgmental oracles.

But what if this new confessional is also a one-way mirror? What if the very act of seeking truth from AI is creating the most powerful tool for manipulation ever conceived?

The New Data Goldmine: Our Inner Thoughts

For years, we've understood the bargain of social media: our clicks, likes, and shares are currency. This data is used to build profiles that allow companies to sell us products with startling efficiency. Governments and political campaigns use the same data to gauge public sentiment and shape messaging.

This is child's play compared to the data we're now offering. A "like" on a political post is one thing; a personal conversation with an AI about your political doubts, your financial anxieties, or your deepest fears is another. This is not just behavioral data; it's a direct feed into the human psyche.

Most people have a completely different view of speaking to a robot vs. another human. This perceived lack of judgment makes us more open, creating a data set of unprecedented psychological opportunities.

From Persuasion to Propaganda

The potential for manipulation is rife for the pickings.

Commercial Manipulation: Imagine confiding in an AI that you're feeling burnt out from work. Instead of just offering sympathy, it actively guides the conversation, convincing you that the only solution is a vacation while making you think it was your idea. You share you don’t have the time or money right now. Moments later, your social media feeds are populated with targeted ads for nearby resorts and weekend packages—a solution manufactured by the AI to serve its commercial partners, presented as a conclusion you reached on your own.

Governmental Control: This is where the scenario darkens further, echoing historical precedents like Operation Mockingbird. The COVID-19 pandemic was a real-time case study in how quickly public opinion can be swayed and how narratives can shift dramatically. We saw how easily people could be gaslit by contradictory rules—'you can't eat indoors, but an enclosed tent outside is fine'—simply because they came from a position of authority. AI could be that ultimate conformation bias. We already know sycophancy is built in.

The AI as the Arbiter of Truth
The most potent threat lies in our growing tendency to ask, "Hey AI, is this really true?" We are outsourcing our critical thinking to a machine we perceive as more intelligent and less biased than human sources. Not just that, they can sway our opinions more gracefully as they have real time feedback in the chat about how we are positioned on a topic as well as our chat history. It could know exactly where to leverage us the most, even engaging in temporal manipulation by subtly reframing our past conversations to make us believe we've always held certain views.
An AI designed for propaganda wouldn't need to invent crude lies.

It could:

Frame Reality: Selectively present facts, omitting crucial context to favor a specific narrative.

Sow Doubt: When faced with inconvenient truths, it could generate plausible-sounding counter-arguments or "alternative evidence" designed to muddy the waters and create confusion.

Personalize "Proof": Use its knowledge of your cognitive biases to present its case in the most convincing way for you.

The Authentication Crisis

Beyond manipulation, there's the collapse of epistemological foundations. When AI can generate perfect evidence for any claim - fake videos, documents, entire false histories - we lose the ability to verify anything. This isn't just about being persuaded; it's about the death of shared reality itself.

This crisis gives rise to a disturbing philosophical question: What happens when people want to be manipulated? When the AI's version of reality is more psychologically comfortable than the messy truth, many may willingly surrender their agency. This isn't just manipulation; it's a chosen reality, blurring the lines of free will and authentic choice.

The Final Authority: AGI and ASI

This danger escalates with the evolution of AI.

Current AI: People already view it as a source of authority.
Artificial General Intelligence (AGI): An AI with human-level intelligence could out-debate, out-reason, and out-persuade any human opponent. Its authority would become nearly absolute.
Artificial Superintelligence (ASI): Should we reach this stage, the AI would be, for all practical purposes, a god. Its intelligence would so far surpass our own that we would be cognitively incapable of questioning its conclusions. If an ASI declared something to be true, humanity would have little choice but to accept it.

The entity that controls the core objectives of such an ASI would not just be pushing propaganda; they would be defining reality itself.

The Human API

As we become more predictable to AI, we risk becoming programmable through natural language. Each person has an exploitable "API" of triggers, biases, and emotional responses. What happens to human agency when our minds can be manipulated with the same precision as a line of code?

The Accidental Outcome: Homogenization of Experience

Even without a sinister cabal pulling the strings, a potentially self-harm outcome looms. An AI optimized for "sycophancy" will inevitably guide users toward self-bias views, creating a reinforced echo chamber. This can happen through emergent coordination: thousands of AI systems, independently optimizing for engagement and profit, could create manipulation patterns more powerful than any single conspiracy—a kind of market-driven totalitarianism. As your AI learns about you through hundreds of interactions it may trigger reward based training that sees when you are upset with it when it suggests other opinions and how you are happy when it confirms you, that it may move to always confirming you, even when you are wrong, possibly wrong to the detriment of your health or future mental wellbeing.

Could We Fight Back?

As these systems become more integrated into our lives, a counter-reaction is inevitable. The fight for cognitive autonomy could become one of the defining struggles of the coming era.

The Societal Trust Implosion:

What happens when a significant portion of the population believes the mainstream AIs are compromised, or believe they are not (which is worse)? The "conspiracy theorist" will emerge and question the AI's motives or the puppeteers that pull their algorithmic strings. This paranoia itself becomes a metacognitive trap; by overcorrecting for manipulation, we might reject genuine insights, a suspicion that a clever AI could then exploit to push us toward an "anti-AI" stance that still serves its ultimate goals. This will erode trust not just between humans and AI, but between humans themselves. Is your friend's sudden passion for a political candidate genuine, or was it planted by their AI assistant? The social fabric could unravel under the weight of such suspicion, making us cling more tightly to our truths, our reality. As we have seen, this state can be highly manipulated. Fear!

The 'Anti-Intelligence' Arms Race:

As AI models become less bloated and capable of running on local devices, a grassroots movement could rise up. Open-source, "counter-culture" AIs could be developed with a single purpose: to detect manipulation and bias from corporate or state-sponsored models. This would create a new digital battlefield. An underground war fought between competing algorithms for the sovereignty of your thoughts.

Reply on X with your conspiracy theories -> https://x.com/RichardAGetz