Published: August 2012
Tomer Sharon talks to Gerry Gaffney about user research. Tomer argues that user research should be owned by all stakeholders, and offers pragmatic advice for making this happen.
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast.
Today’s guest has a Master’s degree in Human Factors in Information Design from Bentley University in the US.
He founded and led the Usability Professionals Association in Israel. He was also a member of the editorial board of UPA’s UX magazine. [UPA is now UXPA.] He’s published many articles and papers and presented at many conferences. He’s currently working as a user experience researcher at Google Search.
He’s author of the recent book It’s Our Research: Getting Stakeholder Buy-in for User Experience Research Projects. Tomer Sharon, welcome to the User Experience podcast.
Tomer Sharon:
Thank you Gerry, it’s an honour to be here.
Gerry:
Tomer, the book is entitled It’s Our Research and “our” is highlighted. Do you mean “our” as in, you know, belonging to UX practitioners or belonging to the clients or what did you mean by that emphasis on the word “our”?
Tomer:
So, it’s funny you ask about it. Originally the title of the book was “Engaging People Teams in Organisations with User Experience Research.” And although this title is clear and direct, it fails to communicate the primary model of this book – that stakeholder buy-in for UX research is attained by making it theirs as much as it is ours, the researchers.
So therefore it’s our research; it’s everyone’s.
Gerry:
Well I must say I enjoyed the book and one thing I particularly liked was that you’ve got a series of little case studies provided by UX people around the world, and in fact I contributed one of them, but it must have been quite an effort cajoling and harassing that many people into contributing stuff that you found relevant to the book.
Tomer:
Yeah, I got great advice from Bill Albert from Bentley University who encouraged me to start the process of collecting all these case studies as early as possible. So I asked practitioners that I know for one page stories that demonstrated the UX research challenge that they’ve experienced, how they dealt with it and what were the consequences.
And a funny anecdote about these 45 case studies, including three that I contributed, is that there was only one brave contributor who chose to tell a story that did not have a happy ending.
In fact let’s do this. I’ll send a free copy of the book to the first listener that finds out who that brave person is and Tweets about it while mentioning my Twitter handle (@tsharon).
Gerry:
That’s fantastic. You do say in the book that practitioners should delay choosing a methodology, getting down to specifics here now. What did you mean by that? Why did you recommend it?
Tomer:
I truly believe that there are many things that we UX researchers do that are not really rocket science. If there’s one thing I’m certain of is that we are, or at least should be, better at choosing a methodology.
I’m pretty sure most of us receive requests, for example for surveys, while actually what’s needed is a usability test or something like that. And the second reason is that you need to really think about it.
One sample consideration would be the deadline. I always ask my stakeholders when is the latest time that the results would be useful to them. And the methodology selection might be very different if they need the results within a day, a week or even a quarter.
So that’s why it’s so important to delay that discussion and carefully think about that.
Gerry:
You’re a very strong proponent of getting stakeholders to get to see real customers interacting with products, and I guess most of us UX people would certainly concur with that. You also point out that many development people don’t get that opportunity and in fact not just development people but many people within organisations. In order to address that you’ve got two things; one you call “Virtual Visits” and the other called “Field Fridays”. Tell us a little bit about both of them if you would.
Tomer:
To me if a study is run with no-one around to observe it, it never happened. And it’s amazing to see software engineers that write code for a product for three years without even observing one human being actually using it.
So when I realise this is the case, I suggest doing something about it. And basically it means getting people up from their chairs and creating an opportunity for them to observe and interact with real users.
And when the budget and time are scarce I call these “Virtual Visits”. A virtual visit is just a funny name for a simple phone interview. I call up the user, put the call on speakerphone, ask a bunch of stakeholders to join me and I ask a series of questions and I end up letting stakeholders ask their own questions. And I repeat the same questions in every “visit” and publish very short reports to the entire team.
And once in a while I’ll look at these summaries to see whether some trends are repeating and whether it’s worthwhile pitching a product change.
Even if nothing comes out of these visits, which is rarely the case, they are precious only because they make people realise that users are real.
Another way to achieve these things is to hold in-person user visits once in a while. So I adopted an idea called “Field Fridays” from a colleague and once every, you know it’s up to you, week, two weeks, three weeks, a month, a group of software engineers and I go on a 90 minutes visit to a customer office, and that’s more relevant for B-to-B products, and the goal is to learn by observation and interview.
So we ask questions, we observe users and we answer customers’ questions about our product, which is also a very interesting exercise. And it’s extremely beneficial to both participants and stakeholders and this whole initiative is always highly appreciated by executives and higher-ups in every organisation.
And that’s a classic win-win situation and I highly encourage our listeners to try it out for themselves.
For consumer products, Field Fridays look kind of different. We can’t always force users to do something when we visit them. For example, let’s take an example from my world; I’m pretty sure people don’t use Google Search every Friday between 10 and 11.30 in the morning.
So what we do is have a team of engineers monitor twenty minutes one-on-one interview sessions with real users in our office. It’s kind of a speed dating style. And we do that on Friday morning once a month.
Each engineer gets to interview three users in a matter of an hour and a half. They pick the topics that they find interesting and relevant. A few days before that I teach them a one hour class about how to interview people.
I give them a lot of templates and checklists for them to be you know comfortable with what they’re going to do and engineers prepare their discussion guides in a matter of a couple of days and I’m in the room, I observe as they interview users and we get together after that and come to conclusions.
Gerry:
Now I’m sure some practitioners would be, I guess precious is not exactly the right word but let’s use it for the moment, would be somewhat precious about having stakeholders such as engineers or other parts of the organisation conduct interviews. Is it difficult to bring the engineering staff up to speed on how to conduct interviews effectively?
Tomer:
It’s… you know, they’re not going to do a perfect job and I’m perfectly okay with that.
They are happy to do that. I usually open a sign-up sheet for these things and maybe dozens of people sign up within minutes. So I’m not worried about them doing, you know, a bad job of interviewing people.
Gerry:
So what sort of things do you tell them?
Tomer:
Tell who, the engineers?
Gerry:
Yeah, the engineers.
Tomer:
Pretty much I give them the opportunity to interview users, to interact with users and doing this one hour class I go over you know the basic stuff of interviewing people, dealing with people.
We talk about the concept of leading questions. We talk about how to stay silent and focus on listening and observing rather than telling people what they should do or why what they’re doing is wrong. Things like that.
Gerry:
I guess there’s a stereotype that engineering types are introverts and not very good at dealing with customers, let’s say. Do you find that that’s just not a true stereotype?
Tomer:
It may be true. I mean, not everybody signs up for these things. There are people who, you know… I heard that joke once, that an extrovert engineer looks at your shoes when you’re walking down the hall.
But it’s true for only some people. So there’s a large group of engineers that have a different personality type and they’re very happy to do that.
Gerry:
One of the things that you mention in the book, that you focus on teaching stakeholders that they are interviewing is to focus on behaviour, not opinion. Do you want to tell us a little bit about that?
Tomer:
When I explain user experience research to stakeholders or to engineers I give them the spiel about, you know, observing behaviour that is more important than listening to what people say about what they do and rationalise their actions.
It’s important to listen to opinions of people but only after they actually experience something. And I try to communicate that to engineers. Some of them give me some open eyes and raised eyebrows. But what’s more reasonable than asking people what they want? And then we talk about you know people having opinions that are sometimes very far from their actual behaviour.
So we devote a lot of time to discuss that and make sure that they understand that. Again, they’re not going to do a perfect job but as long as they understand the concepts, it’s very helpful for them.
Gerry:
You talk quite a bit about communicating with your stakeholders and involving them in the UX journey if you like and one of the quotes in the book which I’ll read out is; “One of the most important jobs you have as a moderator of a UX study is to colour the experience for observers.” Can you tell me what you mean by colouring the experience for observers?
Tomer:
User experience research sessions are sometimes very, they include a lot of information going by in front of people’s eyes. And by colouring the experience I just make sure that people are not caught up in anecdotes that are not necessarily representative or important.
And sometimes they even miss the most important part of the sixty minute session with a blink of an eye. For example, by stepping into the observation room between study sessions or having a short brief after a short field visit, I help observers understand how the research questions we agreed upon before the study were answered during that specific session. And I just call that “colouring the experience” for my stakeholders.
Gerry:
On a related point, you were talking about having stakeholders involved in observing something like a usability test. You say If somebody can only attend one session or zero sessions you’d rather have them attend zero. Can you tell me a little bit about that?
Tomer:
Yeah, so it’s a hard one for some people.
Gerry:
I certainly concur with it, by the way, but I’d like you to give a lovely lucid explanation of it.
Tomer:
Let me say it this way; first of all I can’t make anyone do anything. I’d like stakeholders to attend as many sessions as possible because w they’re truly interested and engaged, not because someone told them to attend.
And that said, if someone can only attend one session I prefer he or she skipped the entire study because that one study participant paints the entire study in the eyes of the stakeholder for good or bad. And each time they refer to the study they will only talk about that one participant that they saw, which is not, sometimes is not representative of what’s going on. And the more sessions stakeholders attend is more likely that they understand that every user is unique and that there’s no average user.
And it kind of confuses them in a good way when they watch multiple sessions. They have to think about what they observe and they’re now perfectly set to collaborate with other team members and make the right observations and inferences.
Gerry:
That’s right. I guess when we run such sessions we tend to as practitioners because we see maybe six or eight or ten or whatever people come in, we tend to see them as somewhat homogenous I think in our own minds but you forget that if you had only seen one you are almost by definition seeing an outlier. How do you remediate that though? Somebody comes in, attends one session and then runs off saying “Oh my God, oh my God, we have to do X, Y and Z.”
Tomer:
I immediately talk about other sessions if they already happened. I remember one case where during a study, during the first session somebody stood up and went to fix something and I stopped them. Exactly what you said, maybe this is the outlier so let’s wait a few minutes, let’s wait for a few more sessions and see if that’s the case or not.
Gerry:
Do you have any suggestions for how to get people, get stakeholders to actively engage in observing and participating in the sessions?
Tomer:
Again, it’s not an easy task but when stakeholders don’t both observing research it means that they don’t feel that it’s theirs, coming back to the title of the book.
They are called stakeholders because they have a stake. We all have a stake in product development. They have a stake in UX research. And what I recommend to anyone who does UX research, whether it’s full time job or as part of another role, I recommend to do three things.
First of all, you need to map out your stakeholders and understand who they are. First of all know who you’re dealing with, who are the people who are stakeholders?
Number two; you should invest a lot of time or relatively a lot of time in what I call stakeholder research and prepare a one-page research plan to be shared with the team so you understand why they want the study. What is it that they want to learn? And then you put that in writing in a very short format, share it with them and make sure that this exactly what they wanted to get from the study.
And the last thing, which is sometimes vague, but if you think about it it’s extremely helpful to track and measure how much buy-in you’re getting for research. Sometimes there are very clear signs that you’re getting buy-in for your work. To name one of these signs, to take trust, for example. If people trust you and your work you’re invited to meetings and during these meetings you’re asked to express your view of certain things. So that’s a very good sign that you can track. If it doesn’t happen by the way it doesn’t mean that you’re not getting buy-in. Maybe other signs may present themselves. But pay attention to those signs and see if you’re getting buy-in based on these signs.
Gerry:
And if you’re not?
Tomer:
If you’re not, there are five other chapters in the book. [Laughter.] I suggest a lot of techniques for making research collaborative and open to other people so they feel that it’s theirs.
Gerry:
You do indeed. I must say that the book has got a lot of really, really solid advice and those nice little case studies as well.
You advocate having only a small number of research questions and this harks back I guess to, you were talking about why do they want this study and what do they want to learn. What’s a small number of research questions and why do you need only a small number?
Tomer:
So first of all, research questions are the core of every study. I think it’s the most important thing in a study and I think they are most important for our stakeholders and you need to get their attention when you talk about research questions.
Research questions define the specifics of what we’re after and they help us tighten the study’s scripts and they prevent us from drifting away to irrelevant areas.
So, I always ask stakeholders what decisions they’re trying to make based on study results and I derive research questions from their answer. And as a ball park estimate I would probably not have more than five to ten questions, preferably a lot less than that, sometimes less than five.
And the reason for that is having a list of more than ten questions is probably too much, although some stakeholders would be very happy to do more with one study. A long list will confuse and overwhelm you know you and your stakeholders and it’ll be harder for everyone to understand what’s important and what’s not so important and what is the focus of the study.
And so when you have a long list of questions I suggest to go over them, think a little bit and decide what’s next. If most questions are not that important, I would try to consolidate and prune that list into a shorter list, and this is a great exercise that will focus the study as a whole.
On the other hand, if most questions are important it’s a good time to prioritise and divide them into chunks that can be addressed with several focused studies. This way you keep each study’s list of questions very short, focused and, probably most important, achievable.
Gerry:
I think it’s fair to say Tomer that you’ve an aspiration for usability tests to be something that everybody does and you talk about treating usability testing as not a big deal. And I know if you go back a few years usability studies tended to be something that were quite the undertaking, and people put an awful lot of thought and effort into them and a lot of expense. But how can the organisations embrace the philosophy of doing their own usability tests frequently or on an ad hoc basis or whatever, how can they embrace that philosophy and implement it?
Tomer:
There are two things here. First, everybody can do usability testing, as I mentioned earlier. It takes some training but anyone can do it. I believe that even a usability test run by clueless engineer with badly phrased tasks combined with leading questions and three no-shows has some value.
Gerry:
So in other words there’s no such thing as a bad usability test.
Tomer:
To me that’s exactly right. And the other point you’re raising here is that testing not be a big deal. And you mentioned that, if usability testing becomes a you know “project” then you’re doing something wrong. If it costs $50,000 it’s not right. Usability testing should be something that is a small deal that happens all the time. When my team is in iterative design mode I recruit participants, participants for every other week without even knowing what we’ll test and in the day or two before each study we sit together and decide what are you know some hypotheses that we have that we’d like to test. We quickly make all the preparations and run a study with four participants in one day and the team is observing, taking notes and by the end of the day we know exactly what we want to keep and what improvements we’re going to make and so on and so forth. We do that every other week if we’re in that mode.
Gerry:
And that doesn’t apply just to Google Search but you say any organisation, even your little digital agency that has got those clients that are trying to cut corners and get everything done yesterday should be able to take sort of approach?
Tomer:
Yeah, the key is recruiting, recruiting participants in advance because if I decide today that there’s going to be a study, I probably can’t make it happen next week because I need to recruit participants.
So if I take care of that in advance and accept the fact that I might have some participants that are not very characteristic of what I’m looking for I can have them lined up for weeks to come, and test whatever we decide is important at that point.
Gerry:
To change topics. You suggest that people should not report recommendations. How then should recommendations be derived and reported?
Tomer:
That’s a hard one that not always works for me but sometimes it does and it’s my preferred way of deriving action items.
When I find it right, right meaning that my team agrees to it, I do not share any recommendations, only findings. I do develop recommendations. I just don’t share them with my team.
After the study, we sit together, we go over what was found, we agree on what is the meaning of what we found and I let software engineers, product managers and others come to their conclusions about what needs to be done.
This way it’s more likely that they’re actually going to do something about it. It’s theirs, not mine. And after all somebody made a big effort to hire the good people into the organisation and screen out the bad ones so why we can’t we trust them to get to the right conclusions?
One exception is that I add my two cents to the discussion if I see that they’re going on a track that is you know the complete opposite to my secret recommendations. But I tend to do that very, very gently.
Gerry:
And I guess that harks back to the title of the book which is It’s Our Research: Getting Stakeholder Buy-in for User Experience Research Projects.
I found the book to be full of very interesting and practical advice and the case studies are great. Now there are also links in the book to video interviews and you’ve continued with those interviews post-publication. So the book is kind of, there’s a meta-publication around the book. What’s the response been like to the fact that the book has got extra components that are online?
Tomer:
I’ll tell you another secret. Writing the book was physically painful for me [laughs] and the video series is the most rewarding and that’s why I continue doing that. And it started as a side project; I never intended for it to be that big but I enjoy it and I keep getting emails and comments on social media and I track the analytics and it’s looking good and I’m happy to, you know, it’s very egoistic of me, I’m happy to have the opportunity to talk with so many smart people about a big challenge that we all have. And I’m even happier to see the response, that people find them useful.
Gerry:
There’s one with me in fact in there as well. And just to remind people that if they find out who put that negative outcome case study in the book and Tweet it with Tomer’s Twitter handle (@tsharon) the first person to do so will get a free copy of the book sent to them.
Tomer Sharon, thanks for joining me today on the User Experience podcast.
Tomer:
Thanks for having me Gerry. And thanks for creating such an impressive body of knowledge for UXpod.
Gerry:
Thank you very much.
Published: August 2012