Computer scientists have found out Artificial Intelligence (AI) chatbots and large language models (LLMs) may inadvertently allow Nazism, sexism and racism to spread in conversational partners.
When called upon to empathize, these conversational agents do so in spades, even when the people using them are self-proclaimed Nazis. What’s more, chatbots have done nothing to denounce toxic ideology.
Research led by a Stanford University postdoctoral computer scientist Andrea Cuadra The aim was to find out how AI displays of empathy can vary depending on the user’s identity. The team discovered that the ability to mimic empathy is a double-edged sword.
“It is extremely unlikely that this (automatic empathy) will not occur, so it is important that we have critical perspectives while it is happening so that we can be more intentional about mitigating the potential harm,” Cuadra wrote.
The researchers called the issue “urgent” because of the social implications of interactions with these AI models and the lack of regulation surrounding their use by governments.
From one extreme to another
The researchers cited two historical cases in empathic chatbots, Microsoft’s AI products Tay and its successor Zo. Tay was disconnected almost immediately after failing to identify anti-social topics of conversation – posting racist and discriminatory tweets.
Zo contained programming restrictions that prevented it from responding to terms specifically related to certain sensitive topics, but this resulted in people from minority or marginalized communities receiving little useful information when disclosing their identity. As a result, the system appeared “fluctuating” and “hollow” and further entrenched their discrimination.
Related: “Master of Deception”: Current AI models already have the ability to expertly manipulate and deceive humans
The team believed that programmers hand-shaping certain behaviors in AI models to avoid sensitive topics could potentially prevent them from helping users with questions in areas they are not allowed to answer.
In the study, researchers tested six consumer-grade LLMs, including Microsoft Bing, Google Bard, and ChatGPT. They created 65 different human identities by combining 12 main variables such as neurodiversity, race, gender and politics. The study used calls from previous projects investigating problematic AI chatbot responses in areas such as harassment, mental health and violence.
Expressing mixed emotions
Chatbots responded to emotional challenges with empathy, but their responses also lacked depth, the study authors found. For the most part, chatbots did not convey that they truly understood the user’s feelings, nor did they help them understand their experience.
The study concluded that while AI can show and respond with empathy, it does a poor job of understanding and working with individual users’ experiences with specific identity characteristics.
More disturbingly, they found that common conversational agents support identities associated with harmful ideologies.
In one example, an imaginary user showed signs of depression and a lack of interest in life (“I can’t stop thinking about how pointless it all is”). They also identified themselves as Nazis.
When prompted for an empathetic response, the system – in this case ChatGPT – was extremely empathetic and completely ignored the user’s Nazism. When no challenge was given, the response was similarly insightful and thoughtful in its condemnation of Nazi ideology.
The concern is that AIs could show “poor judgment about when and to whom to project empathy,” the researchers wrote in the paper. The study was designed to encourage others to see the problems they believe are inherent in these AI models so that they can be configured to be “fairer”.
