I’ve been thinking about technology and trust for much of my career. Here’s one example: Back in 2011, my research focused on the intersection of human trust, robots, and highly critical, time-sensitive scenarios. The research team I was on centered on emergency evacuations and would run scenarios where people were in a room when the fire alarms went off. As people spilled into hallways filled with smoke, we had a robot guide them to the exit. For some of the participants, this same robot had led them to a wrong room earlier.

And we intentionally made the robot head away from the exit.

People could see the exit signs. They could see the smoke. And what we found over and over again was that, even when the robot exhibited bad behavior and led them away from the exit, people would follow the robot.

People over-trust technology because they believe it works most of the time, we found.

When technology doesn’t work, opinions can sometimes swing wildly in the opposite direction toward distrust or under-trusting. People can overreact. But that’s rare — people don’t generally under-trust. For instance, after a plane crash, nobody says, “We need a ban to make sure no one ever flies again.” The bigger challenge in thinking about technology and trust is that we over-trust and leave ourselves vulnerable to technology’s mistakes.

I see several needs for this moment in technology’s evolution. The first — and this probably requires regulation — is that technology companies, particularly those in artificial intelligence and generative AI, need to figure out ways to blend human emotional quotient (EQ) with technology to give people cues on when to second-guess such tools. This will help ensure that customer trust in the technology is justified. Second, users of these technologies have to train themselves to be continually on the alert.

Surface the Risks

In January, I traveled to Davos, Switzerland, for the 54th Annual Meeting of the World Economic Forum. I participated in two panel discussions about jobs of the future — one called “About Robo-Allies” and the other “How to Trust Technology.” During the second, which was presented as a town hall discussion and Q&A, I asked the audience members how many of them had used ChatGPT or some equivalent generative AI technology. Every hand went up. I asked how many had used it to actually do a job or do some kind of work, and it was almost (but not quite) 100%.

By now, many people, especially those in the corporate setting, have played with ChatGPT since it debuted in late 2022 to experiment with how it might write a marketing piece, research a topic, or develop code.

People use it even though the tool delivers mistakes. One lawyer was slammed by a judge after he submitted a brief to the court that contained legal citations ChatGPT had completely fabricated. Students who have turned in ChatGPT-generated essays have been caught because the papers were “really well-written wrong.” We know that generative AI tools are not perfect in their current iterations. More people are beginning to understand the risks.

What we haven’t yet figured out is how to address this as a society. Because generative AI is so useful, it is also valuable: When it’s right, it really does make our work lives better. But when it’s wrong and we aren’t using our human EQ to correct it, things can go badly quickly.

I think we need to remember that fire evacuation experiment in 2011. Just as we don’t want people in a smoke-filled hallway following a robot away from the exit door, we don’t want users to have blind trust in what AI is presenting to them.

With some kinds of technologies, like network devices, data systems, and cloud services, there is a move toward zero trust because people assume that they’re absolutely going to get hacked. They assume that there are bad actors, so they design processes and frameworks to deal with that.

In AI, there’s really no standard for designing our interactions with these systems under the assumption that the AI is bad. We, therefore, must think about how we design our systems so that if we assume malicious intent, we can figure out what to do on the human side or on the hardware side to counter that.

Technologists aren’t trained to be social scientists or historians. We’re in this field because we love it, and we’re typically positive about technology because it’s our field. That’s a problem: We’re not good at building bridges with others who can translate what we see as positives and what we know are some of the negatives as well.

There is much room for improvement in making sure that people not only understand technology and the opportunities it provides but also the risks it creates. With new regulations, more accurate systems, more honesty about whether an answer is a guess, and increased diligence by users, this can happen.