I’ve spent the last five years as a lead design researcher or principal design manager on a number of automation and intelligent systems projects. In my research into artificial intelligence (AI) design, it’s clear that using traditional user research methods, such as interviewing, to gain confidence in AI product design direction has limiting factors. These factors include the almost universal shallowness of AI literacy among product makers and consumers, the focus on a uniagency (single-agency) framework during interaction design research, and a lack of integration of design research and data science during the product research phase.

Let’s first take a look at the low AI literacy as a limiting research factor. The shallowness of AI literacy has no respect of race, gender, economic status, or cognitive ability. Most people in the world just aren’t well versed in how AI products operate. Asking them questions about how they’d like their self-driving car to behave is akin to asking them questions about how they’d like their interstellar Stargate traveling device to be designed. We can’t assess their desirability for AI-products by asking them what they want, like we have so easily done in the interaction design age.

The next limiting factor is the agency framework we ascribe to the current user experience research methodology. Currently, most user experience research deals solely with a one-sided agency framework—we assume that a person will decide when and how to use a device, platform, or service. But user experience in an autonomous-product era shifts the agency paradigm to one that is multi-agency. When you have a human interacting with a system that has its own decision-making ability, you now have two sentient parties, each with their own agenda operating in the same context, cooperating to achieve a common goal. This two-party agency dynamic presents unique challenges to traditional user-research methods.

Failure to anticipate human needs in this two-party agency world has led to a spectrum of maladies from the mildly uncomfortable to the downright catastrophic; one only has to look at recent tragic deaths of pedestrians involving self-driving cars.

Or look at artificial intelligence use in the criminal justice system, where innocent citizens are flagged by predictive policing algorithms or violent criminals are released on the “word” of an algorithm-enabled “threat assessment score,” even though judges and prosecutors disagreed. These misfires have sent every sector of our society—from ordinary citizens to celebrities, corporate CEOs, academics, government officials, and even the Pope—searching for a more ethical and better way to design AI products and services. But instead of usability, they speak of abstract concepts such as trust, privacy, freedom, and identity. The answers to these questions are not obvious. Fundamental values are being negotiated when humans interact with sentient machines.

And the last limiting factor of current user research in an AI-fueled world is a pragmatic one—our separation of design research and data science. Our inability to integrate research methods into data science methodology is hampering our ability to make ethical and more humane AI products.

So now that we have established the limits of current user research methods when it comes to AI product development, let’s present an unlikely but uniquely suited field of research as a useful addition—design anthropology.

As a research lead on AI product development, I discovered that the most important factors that gave confidence to our design research were not concrete measurable traits like usability, flow, or even user experience, but understanding the intangible human needs locked inside culture, values, and human rituals.

For example, during a project where we were trying to design the self-driving car people wanted to own in the future, the team found that understanding why humans find joy in driving was much more valuable than asking if people wanted a steering wheel in their new self-driving car. I achieved this confidence in design direction by incorporating some new design methods, including games, simulation, and provocations to help dig deeper into the psychological need humans have for ownership and autonomy. This, unbeknownst to me, was the application of design anthropology.

Design anthropology is a field that focuses not on the use of a product but on the intrinsic value the user brings to the product experience. This is a paradigm shift that is needed for more humane AI product development.

In the book, Design Anthropology: Theory and Practice, educator and design anthropologist Elizabeth (Dori) Tunstall (2013) writes that design anthropology focuses on how design translates human values into tangible experiences. Its focus is to ask the unanswered questions in a way that yields better definitive direction for future-oriented design. Design anthropology uses design as a way to understand culture, values, rituals, and other abstract notions and translates them into design.

As a design anthropologist, I use an ethnographic approach that is iterative, reflexive, and action-oriented. Rather than being reliant on observing the current state, design anthropology employs simulations, mock-ups, props, and tangible interactions to explore futuristic and speculative design. Design anthropology isn’t about passive observance but is more proactive, using games, scenarios, and performances to understand participants’ feelings and reactions to future states. Here are some methods that differ from traditional design research:

• Biased toward building: using provotypes (prototypes designed to provoke discussion), artifacts, and simulations designed to induce reactions not observed during current states of use

• Perpetual synthesis model: the research phase does not end or begin; it is iterative and present throughout the product development cycle

• Transdisciplinary product development: requiring teams of various disciplines working together to create new conceptual, theoretical innovations that integrate their methods to build provotypes and do research to gain confidence in design direction

It is clear that developing human-centered AI products will require companies to go beyond normal user research methods. Anthropology offers several methods we can integrate to get at the most important factor in determining confidence in AI-design direction—determining what it fundamentally means to be human and how not to design that out of the equation.