Unveiling Agents of Manipulation: Understanding the True Risks of AI

Our lives are on the brink of transformation with the rise of conversational AI agents designed to assist us at every turn. These intelligent agents will anticipate our wants and needs, utilizing a wealth of personal data regarding our interests, backgrounds, and aspirations to deliver customized information and perform meaningful tasks. The overarching goal is to enhance convenience in our daily lives.

Just this week, OpenAI unveiled GPT-4o, their latest chatbot capable of detecting human emotions. This advanced technology can analyze not only the sentiment in your text but also the nuances of your voice and facial expressions during video interactions.

Google recently announced Project Astra, aimed at deploying an assistive AI that engages in conversation while understanding its surroundings. This capability will allow it to provide real-time, interactive guidance and assistance.

OpenAI’s Sam Altman shared with MIT Technology Review that the future of AI lies in personalized assistive agents—described as "super-competent colleagues" that manage everything from emails to conversations in our lives, thus taking proactive actions on our behalf.

However, this utopian vision raises concerns. As I discussed in a previous article, there is a significant risk that AI agents could compromise human autonomy, particularly through targeted manipulation. As these agents become integrated into our mobile devices—the gateway to our digital lives—they will accumulate vast amounts of data about us, influencing the information we consume while monitoring and filtering our interactions.

Any system that observes our lives and curates the information we receive poses a risk of manipulation. Equipped with cameras and microphones, these AI agents will perceive their environments in real-time, responding to stimuli without our explicit requests. While this functionality promises convenience—reminding us to complete errands or providing social coaching—it also risks deep levels of surveillance and intervention, which many may find unsettling. Nonetheless, I anticipate that many will embrace this technology, driven by the enhancements it offers to our daily routines.

This trend will catalyze an "arms race" among tech companies to deliver the most powerful mental augmentations. As features like these become ubiquitous, users who choose not to adopt them will find themselves increasingly at a disadvantage, turning what once seemed like a choice into a necessity. I predict rapid integration of these technologies by 2030.

Yet, we must acknowledge the inherent risks. As discussed in my book, Our Next Reality, while assistive agents can grant us extraordinary cognitive abilities, they are ultimately profit-driven products. By using them, we invite corporations to guide and influence our thoughts, making this technology both an empowering resource and a potential tool for exploitation.

This leads to the "AI Manipulation Problem": the heightened effectiveness of targeted influence via conversational agents compared to traditional advertising methods. Skilled salespeople understand that engaging individuals through dialogue is far more persuasive than static advertisements. As AI agents become adept at using psychological tactics and possess extensive knowledge of our preferences, their ability to manipulate will surpass that of any human.

These agents can significantly impact our decision-making, employing continuous feedback to refine their influence strategies in real-time—much like heat-seeking missiles adjust their trajectory to approach their targets. Without regulation, conversational agents risk using this capability to disseminate misinformation, making regulatory oversight imperative.

Innovations from companies like Meta, Google, and Apple signal that conversational agents will soon be an integral part of our lives. Meta's recent Ray-Ban glasses and Apple's advancements in multimodal AI highlight the rapid progression toward technologies that will offer constant guidance. Once these products reach consumers, their adoption will surge.

While there are numerous beneficial applications for these technologies, the potential for manipulation cannot be overlooked. To safeguard public interests, I strongly urge regulators to act swiftly, implementing stringent limitations on interactive conversational advertising— the entry point for propagating misinformation.

The time for policymakers to take decisive action is now.

Louis Rosenberg is a seasoned researcher in AI and XR and the CEO of Unanimous AI.