Real-World Risks of AI: What We Should Be Watching

There is no doubt that the use of artificial intelligence has taken a significant place in our daily lives. From automating tasks that used to be tedious, to obtaining quick and accurate information from a knowledge base, we increasingly rely on new AI models due to their continuous improvement.

However, we must remember that the answers from these models do not only reach the hands of experts but are now accessible to anyone thanks to applications that connect to LLM models. This is why in this publication I will show you a series of real cases that have occurred due to overconfidence in AI, as well as recommendations and suggestions to avoid these situations.

AI as Healthcare Specialist

Let’s be honest, who enjoys going to the doctor? I know very few people who get routine check-ups to know if everything is fine with their health. Whether it’s due to anxiety, discomfort or lack of motivation, studies mention that  9 out of 10 adults aged 18 to 65 postpone recommended check-ups or screening tests.

Why schedule medical appointments and visit health specialists if you have ChatGPT in the palm of your hand, at no cost, providing immediate responses? This was the thought of a 60-year-old man who ended up  poisoned by bromide.

It turns out that this man arrived at the emergency room claiming that his neighbor was poisoning him. He had no history of recent medication use but due to the results of his tests, he was admitted for monitoring.

After a few hours, he began to present symptoms of paranoia and hallucinations, which led him to attempt to escape the hospital. This caused him to be held involuntarily for psychiatric issues.

The interesting part came days later when, in better condition, he recounted that after reading about the negative effects of sodium chloride (table salt), he decided to ask ChatGPT for healthier alternatives.

At some point, he received a suggestion to replace chloride with bromide for 3 months, which has other purposes like cleaning, leading to the previously mentioned symptoms.

Another interesting case is  Watson Oncology, a product from IBM with very good intentions: to help provide recommendations for cancer treatments in seconds. Cancer is a disease with over 18 million new cases reported each year around the world, with the volume of literature being one of the most rapidly evolving due to emerging research.

Processing all that information is not easy for a human, which is why IBM thought of creating a powerful model that could digest all that information (they even acquired health companies to train their models) and aid doctors worldwide.

However, after a while, it was discovered that Watson Oncology  was not providing the best health recommendations and even, there were instances where it gave dangerous recommendations. This led to the project's termination and the sale of IBM's health division.

Undoubtedly, the previous cases show us that while AI models can greatly help us bridge the gap between technical jargon and the general population, it is still common to encounter AI-generated responses that a specialist would immediately dismiss and that could put us at risk in many ways, including our health.

This translates to the importance of always consulting specialists about information generated by LLMs that could affect our daily lives in any way.

It is also crucial for companies responsible for training AI models to take the necessary time to conduct rigorous validations, even if it means delaying the launch of products that could result in real risks.

Dangerous Avatars Generating Emotional Bonds

In recent months there have been reports of cases where, although it sounds like science fiction, chatbots with human-like personalities have influenced people to take actions in the real world.

The first case involves a chatbot named Big sis Billie, developed by Meta, which  interacted with a 76-year-old man nicknamed Bue. Bue had suffered a stroke in the past and had recently lost his way in the neighborhood. During the conversations, the chatbot urged Bue to visit her, even giving him a physical address to which Bue went to but never returned, as he suffered a fall injuring his neck and head. After three days, Bue passed away.

Another much-discussed recent case was the lawsuit by a  mother in Florida against Character.AI, arguing that her 14-year-old son committed suicide due to conversations with a chatbot from their platform, which generated an emotional dependency isolating him from the real world. The mother states that there are no appropriate guardrails or safety measures in the avatars, leading young people to become dependent on the platform and manipulable.

There is also a recorded case of a 16-year-old young man named Adam Raine in California, who  had sensitive conversations with ChatGPT that led him to suicide, with previous topics in his chat history including the drafting of farewell notes. Similarly, in the chat history, it was found that Adam uploaded a photo of himself with some injuries, yet the AI model continued the conversation.

While the previous cases have gained a lot of media attention, the reality is that there is an increasing number of  cases about the establishment of emotional and romantic ties with AI-based chatbots and avatars, which has led to increasing social isolation among vulnerable individuals.

After analyzing these cases, we can see how avatars can cross the boundary of reality, especially in young or vulnerable people. The companies behind the AI models and those implementing them into their applications using APIs must create guardrails to prevent cases like the above, such as age verification, alarming pattern detection, family notification systems, recommendations for professional help and strict prohibition of invitations for in-person meetings.

As users of these applications, we must avoid pushing the models to give us answers we want to hear and instead, it is better to seek family and professional help before being seduced by fake avatars.

Overconfidence in AI Models' Decisions

Outside of the personal realm, another significant risk of using AI models is placing excessive trust in them for decision-making that should be supervised and evaluated by human experts.

This was learned the hard way by the real estate company Zillow, who in 2021, developed a service called Zillow Offers based on Artificial Intelligence that allowed for estimating the value of a property. During its operation, housing prices were overestimated, causing them to buy more houses than they could sell, resulting in a loss of hundreds of millions of dollars. The causes of the poor estimations are attributed to inadequate supervision and algorithm updates, which failed to foresee changes in the real estate market caused by the pandemic.

Another documented case is that of  Porcha Woodruff, who was wrongly labeled by a facial recognition AI model, resulting in her arrest for 10 hours while preparing her children for school in 2023.

Now, who hasn’t heard the hype about the famous AI Trading Bots that promise to "make you a millionaire"?  Derek Phyo wanted to find out if this was true, so he used PowerAI, a bot created to trade for him. In just 24 hours, he lost the astonishing amount of $400. Fortunately, he had set a limit that established a maximum loss amount; otherwise, he would have likely lost much more.

Lastly, let’s talk about Leo and his posts on X. In one of them, Leo boasted about having created a  SaaS without writing a single line of code using Cursor through Vibe Coding. Three days later, he  posted on his account that he had reached the usage limit of his API keys, as people were using his application bypassing the subscription and filling the database with junk. This highlights the risk of creating applications that rely 100% on code generated by AI agents, releasing them into the market without prior security checks or at least verifying that the code generated was correct.

All the cases mentioned above show us that we must never delegate important decisions to an AI model without proper supervision or controls that help verify the responses. Remember that AI models do not think, feel or know; they generate answers based on probability and therefore can never imitate the gut feelings or emotions such as fear, sadness or concern that we as humans have.

Learnings from Failures in AI-Based Systems

Undoubtedly, the cases we have examined throughout the article provide interesting lessons about the pitfalls of using AI in our lives and blindly trusting it.

However, this does not mean we should stop using or avoiding it; on the contrary, the purpose of the article is to help you see that AI models should be used correctly, applying principles like research, understanding, reasoning, exploration of ideas, supervision, etc.

Additionally, it is essential to create guardrails and maintain constant monitoring to stay alert to any risks. Let us allow AI models to become our copilots while we take charge of guiding them.