When pet peeves discredit the whole field of UX
And supersonic robopilots in the January '22 Edition.
Happy New Year!
And hello from snowy Denver!
Hope you all are well, warm, and staying satisfactorily busy in these dark days of winter.
A thought on… hot takes and newborn onesies
You wouldn’t know it from the state of discourse in our field, but user experience research long ago left its awkward teenage phase. You could date the origin to 1993 when Don Norman coined the term, or a lot further back to Bell Labs and early human factors or HCI research. I’ll go with Joe Dumas, who argued that our kind of work really began to crystallize in the late 80s.
And about 18 years later, as the field reached maturity, Arnie Lund began pushing for a more multidisciplinary field, incorporating a wider range of methods to capture a more holistic view of the user experience. To an extent, that’s exactly what’s happened. We have a lot of methods. Some of them are classics, unlikely to go anywhere. Others pop up every few years, riding the wave of some business trend.
Researchers tend to be more curious and more skeptical than the general population. They love pursuing understanding, which we understand is always an asymptotic endeavor. That it’s easy to get blinded by bias. And so it’s natural for researchers to think twice about any method that hasn’t been time-tested or thoroughly validated.
But what this leads to is a popular genre of writing in UX research: a methodological criticism that proves too much. When you take a close look at the arguments and apply them generally beyond the topic at hand, it ends up invalidating a lot of what we know to work.
Take the much-maligned Net Promoter Score. It’s harmful—indeed, the worst (and in case people don’t know how you really feel, you can dress your baby in the message). Why? Well, because people are so dumb that they can’t differentiate a 6 from a 10. Plus, perverse incentives might lead some who ask it to game the system. And honestly, that weird calculation stresses me out.
But every one of these criticisms could, in principle, apply to all the scales that we use on a regular basis, like the SUS or SEQ. Personally, I have found a lot more participants to get confused answering the SUS than the NPS, but we still use it—and rightfully so. A skilled facilitator can help mitigate any confusion. And you can assume that the noise affects every respondent more-or-less equally, making comparisons totally legitimate.
Or you may have heard that surveys are dangerous. After all, you could ask a bad question! Really, you should be trained so that you ask more good questions and analyze the data properly, checking it for anomalies before you report it out. But… wouldn’t all these “dangers” apply to good ol’ user interviews as well?
Or, I wonder: can making a small change to the wording of a question or the way it’s presented without changing its meaning have a significant impact on the response?I sure hope not, otherwise researchers shouldn’t ask anything except validated psychometric scales. Even if a colleague and I are moderating a single study using the same script, we’re never going to conduct it exactly the same.
We largely got the multidisciplinary field that Lund wanted. We’re lucky to have so many methods and approaches as tools in our toolkits. By all means, we should look at them critically. But there’s no universal tool that applies equally well in every situation. And even the best tool for the job could be used poorly.
In many of these cases, we should be holding the mirror up to ourselves. When a researcher doesn’t choose the right method, or perhaps chooses the right one but fails to use it properly, that’s not a judgment on the tool. It’s a reflection of the researcher.
Thanks for reading The ¼" Hole! Subscribe for free to receive new posts and support my work.
What I’m up to: helping others ease from academia to UXR.
Just one article between last month and now, continuing the series for transitioning grads. More to come here:
3 reasons grad school won’t count as UXR work experience (and what you can do)
Folks get a lot of valuable experiences in grad school—in many cases, doing the same kinds of things they might later do at a company. Unfortunately, this often isn’t recognized, but grads can still find upside if they know where to look.
The Distractor: Maybe, soon, flying will suck less.
Beyond talking about date selectors and the choice architecture of seats and upgrades, air travel doesn’t come up much in user experience, despite its importance to the foundation of the field, and the multifaceted ways in which flying becomes a more miserable experience each year.
But we may see some major innovation in flight this decade, starting with autonomous commercial jets:
We can't make flying cheaper or increase the traffic density without making it less safe. We can't make it safer without making it more expensive or blocking off more airspace. There is a shortage of pilots, people do want to do more economic activity in the air, nobody wants to compromise the safety, so how do we move the Pareto front forward instead of being constrained to that surface of trade-offs?
Trust is going to be a major hurdle, and a big opportunity for researchers.
Elsewhere, a Denver startup is progressing steadily towards scaleable supersonic passenger flights. Think about it: Seattle to Tokyo in 4 and a half hours. New York to London in 3 and a half hours. United has already ordered 15 planes!
Tell me what you think.
As always, I’d love to hear your feedback—just hit reply. If you’ve found this issue interesting or useful, please forward it along to someone else.
Last thought: I’m curious how folks are feeling about all the web3 hubbub. How should we think about it, and approach it, as UX researchers?
Until next time,
In fairness, I don’t think Jeff and co. are arguing this, and none of their research has borne that hypothesis out. But the fact that they’ve run a bajillion studies on variations of this question suggests that a significantly large population of researchers must actually believe this, and needs substantial data to be convinced otherwise.
Maybe flying should suck - it's a huge contributor to climate change!
Not to be contrarian here, but my first thought upon reading that paragraph about supersonic flights was: 'will this just accelerate climate change?'
The short answer is maybe. The article you linked talks about the possibility of suing sustainable aviation fuels to reduce the footprint of flying, but with further research into this it seems to be a bit of a catch-22 - we need more electricity to produce the fuel, but the majority of the world's energy supply is fairly unsustainable. So we would just shift the emissions around. (Not to mention the demand on electricity from other innovations like EVs)
"Electricity, water, and CO2 are used to create jet fuel that can be used in existing planes. But Ian Mason, a renewable energy engineer at the University of Canterbury, found that it would need at least an extra 28,000 GWh per year of electricity, two-thirds of the entire current supply, and at a time when many other sectors of the economy are seeking to electrify."
So this is an interesting systems view of UX to me. By making the flight experience more pleasant or enjoyable, are we encouraging behaviour that is eventually going to destroy the planet (and therefore... the experience)?
Or conversely, by introducing friction to the flying experience, are we taking a more responsible, long-term stance that ensures the experience can continue? (Hydrogen technology might even be able to catch up...)
Price is another way to create friction, but I'd argue that making flying a generally unpleasant experience (and making alternatives viable and desirable) could make people thinking twice about taking a flight. Though, depressingly, evidence seems to conclude that we need to discontinue commercial flying to even have a chance of meeting our reduction targets.