Published: 6 June 2016
How do you know your idea is any good? Through validation or invalidation
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast.
Validating Product Ideas Through Lean User Research is the latest book from Tomer Sharon. He was on UXpod about four years back talking about his book [It’s Our Research: ]Getting Stakeholder Buy-In for UX Projects]. At that time he was working with Google but he’s now head of user experience at WeWork in New York City designing living spaces, communities and services around the world. Today in fact he’s speaking to me from somewhere in Japan.
He also has a Masters degree in Human Factors in Information Design from Bentley University.
Tomer Sharon, thanks for joining me again on the User Experience podcast.
Tomer Sharon
Thank you Gerry. It’s great to be here again
Gerry:
I had a nice hard copy of the book but Julian Huxham stole it in Sydney so I’ve been working off a, I’m missing my mark-ups but one of the first things I marked up in the hard copy was “bullshit personas”. Tell us about bullshit personas because I just love that term.
Tomer:
Well first I didn’t coin it, I’m just using it. But I am not a big fan of personas. A lot of people treat personas as a research technique. They’re not. They are a communication tool that is good for certain situations. A lot of people confuse personas with user experience; we’re doing user experience, we do personas therefore we’re doing user experience and that’s partially true. Nicer names for these are “assumptive personas” and “proto personas”. I prefer to call them what they are, they’re bullshit personas. Bullshit personas are personas that are not based on research, that are only based on intuition, gut feeling, we know these things. So until proven otherwise these are bullshit personas and I prefer to call them, you know that’s what they are.
Gerry:
Yeah and I think they really run the risk of losing one’s credibility as well don’t they?
Tomer:
Yes, yes. They’re designed really nicely so people, you know the big posters, so people believe in them but they’re based on nothing.
Gerry:
Now the book title contains the term “lean user research”, when we think lean, I guess we think Agile. How does lean user research relate to the Agile development process?
Tomer:
Honestly, I have no idea. [Laughter.] That’s the truth. I have no idea. I never worked in a full Agile environment so I, and you know, I read about it but I don’t want to guess. I can tell you about the user research and people who know what Agile is can maybe make the comparison.
So to me it’s just research that is being done by what we used to call stakeholders, so by these start-up founders and enterprise product managers and maybe designers and maybe others, engineers sometimes. They do the research, and they do it on their own quickly and cheaply and there are some, you know quickly I’m going to go over several principles that I identified as defining the lean user research approach.
So first of all, and I know these are relevant for all kinds of research, but this is truly important here: working on research based on very precisely stated questions. I see a lot of research that is being done with a goal in mind and then you find so many interesting things along the way and you forget what you started with. With lean user research you clearly define with your team what questions need answers by the end of this research effort and this is the only thing that you’re after because these will move you forward. So that’s one.
Second in lean user research, we fall in love with the problem first rather than a solution so we’re not going to test the solution first. First we’re going to learn as much as we can about a problem. We can talk about it later. Lean user research focuses on behaviour, human behaviour and not on necessarily on human opinion or attitude. Research is never done. I never met a company that develops a product and after version one or two they say okay, we’re done. They always continue to develop, therefore there’s always research. Research is not a final thing that you do to prove something or to find something. You can always find more and learn more.
Lean user research techniques are nimble and fast. They involve rich reporting. The results are not necessarily coming in the form of a report or a presentation. They can come in different, richer forms and, building on my first book, lean user research is everyone’s. It’s not just the person who’s doing that that’s interested in it. So these are the principles of lean user research, how they compare or relate to Agile, I can only guess.
Gerry:
And when you say it’s everyone’s, and referring back to your previous book, I guess it’s very much the case that this book is not aimed at UX people, not that it’s not of interest to UX people, but you’ve got a much, much broader audience. I presume that was deliberate?
Tomer:
Yes, yes, very much so. I’ll tell you a behind-the-scenes story. When I wrote the book proposal and Lou Rosenfeld and I talked about it, this was for start-up founders. I have a lot of experience mentoring start-ups and working with start-up founders and I wanted to write a book for them. He persuaded me to broaden the audience to product managers and enterprises and engineers as well because when you think of it they have the same challenges as start-up founders in many cases. I know that user experience people might find this useful and interesting be it this is not written, you know, when I think about the language that I use and the content that I use, that I present there, it’s not written for them in mind although I know it is of interest.
To side-track us totally, I mean one could argue that, and I know there’s been a whole movement of do-it-yourself usability and, well, lean user research is an example, but do you think that there is a future for individual UX people or should UX skills be just something that every member of the team has and develops?
I’d love for every member of the team to develop that. I think it’ll take some time but in any case I see user experience people and specifically researchers, as facilitators and mentors. Yes there is the type of research that involves, you know, high level research, that involves a lot of things that a product manager or an engineer wouldn’t know about and wouldn’t care about and that’s fine. But between you and I and the listeners, I think 80% of what researchers do is not rocket science and others can do that pretty well.
Gerry:
Yeah, in fact Steve Krug's book is called "It’s Not Rocket Surgery". He’s specifically talking about testing of course but I think most of these techniques are very much open.
Now the book is based, as you say, well not based but it does enable people or suggest people pose a series of important questions. I’d like to talk about the first one you’ve got in the book and that’s “What do people need?” Why should somebody ask this and how can they answer it?
Tomer:
Well to me this is the most important question to ask. You know nowadays many people understand the importance of design and they interpret that as developing beautiful and usable products. It’s not that every product is usable and beautiful but many product developers now understand that this is important.
The thing is that they ignore the problem and the problem is that sometimes beautiful and usable products are products that nobody needs and if there’s no need for this product, if it doesn’t solve any problems, then I don’t see the point of developing it.
Here’s a nice number, I interviewed 200 start-up founders and product managers for the book and 86% of them decided that their product will solve their own personal problem and only 2% of them, that’s very few people, relied on research. So that’s why I think this question; “What do people need?” is so much more important than developing a usable product. We need to develop the right product not just a product that works right.
So to answer that question, I would say that the best predictor of future behaviour is current behaviour and therefore there are several techniques that can help us track and understand or reveal that behaviour. For example, experience sampling or interviewing or observing people. A lot of people, when they think of research they think of surveys. This is not what I mean by research to identify needs. And there are three questions, especially when we’re talking start-up founders, three questions that they ask in surveys that I just, I can’t say enough how bad they are to identify needs and they trust in the answers to those questions and the questions are, if they describe the product or show it and then they ask “Would you use it?” and then they ask “Would you pay for it?” and finally they ask “How much you’d pay for it?” And based on the answers they conclude that there’s a need. And there’s always going to be a need because they always ask their friends and family and everybody wants them to be successful so they say, “Yes, I’ll use it. I’ll pay for it. A lot.”
Gerry:
It’s a bullshit questionnaire, a bullshit survey.
Tomer:
Exactly, exactly.
Gerry:
Now you said one of the things, and you talk about experience sampling in the book in some depth and it’s a really interesting technique but you do say, you said a minute ago that looking at current usage is important but a lot of people have said to me over the years, “Oh, the product we’ve got is completely new and there’s nothing like it on the market and therefore we can’t do research into current behaviour.” What do you think of that?
Tomer:
So I don’t buy that. The product is supposed to solve something; even if it’s new it’s not solving a new problem. So if people have that problem already, this is falling in love with the problem, identify what is the problem, what do they do today to solve it? Why do they do that? Are they happy with their solution? Why are they happy? If they’re not happy, why are they not happy? How much do they care about the problem?
A lot of people confuse, get confused with the definition of a problem. A problem is not just the gap between the current state and the future state, the ideal state; it’s also how much people care about it. If they don’t care enough about a problem that is valid then they’re not going to do anything to solve it or pay for anything to solve it. I think we can find several hundreds of thousands of apps on both stores to prove that. There are many, many apps, for example, that solve problems, real problems but problems that people don’t care about or enough about.
So, first falling in love with the problem, not the solution. If it’s a new product, it does a new thing it’s falling in love with the solution, not with the problem.
Gerry:
I think it’s often hard for people to realise just how unlike their users they are.
Tomer:
Yes.
Gerry:
I mean I often use the term ‘real people’ or ‘real humans’, go and talk to real humans. You pose the question, ‘Who are the users?’ How can someone with an idea for a new product answer this question?
Tomer:
So first of all, assumptions. Bullshit personas, I don’t say don’t create them, create them, but then go after the truth. So create your assumptions, they could be your bullshit personas, that’s fine, but then interview those people, observe these people. If you find that they are indeed the audience, meaning they do have the problem that you’re trying to solve with your idea, then move on.
If they’re not, then correct your course and identify other or go and develop other or more bullshit personas or other audiences and, again, try to prove, change the bullshit personas to real personas based on real human behaviour.
Gerry:
I think, this is a generalisation, but a lot of people who are, perhaps particularly in start-ups, but say a lot of technical people are people who have got a background in problem solving, they’re not always entirely comfortable interacting on a very personal level with people they don’t know so going out and doing user research, I think it would be quite a challenge. Do you have any thoughts on that?
Tomer:
Yes, and your and my friend Caroline Jarrett and I had a little argument about what I’m about to say. So most of the techniques that I describe in the book are indirect, exactly because of what you just said. These people are, in many cases, not very good in moderating research sessions. So if it was up to me to not write the book, not guide them at all, have them not do research at all compared to have them do indirect research such as, I don’t know, I’ll take an example… usertesting.com. That’s an indirect method. They don’t interact with the research participant. They’re not synched so they can somehow mask the fact that they’re not good moderators. So I try to kind of accommodate to get to that type of personality.
Gerry:
And what did Caroline say? Caroline likes a good argument.
Tomer:
[Laughs.] Caroline says, “Oh My God, you’re not encouraging people to interact in person with their users. That’s awful.” I agree but again I agree that in person interaction is best, is the best thing to do but if it’s none of that or indirect methods, I prefer indirect methods.
Gerry:
Okay, so let’s move on to, you talk about MVP and of course MVP or minimum viable product is one of the buzz words over the last four or five years I guess or maybe a little less, I’m not sure, but you talk about what an MVP is and is not. You’ve got very clear views which I thought the people listening would be interested in. Tell us about that.
Tomer:
Probably the most misused buzz word in the past five years. So an MVP is a few things. It’s a process that allows its creators to validate or invalidate assumptions, quickly with a subset of potential users. I’d like to say a few words about invalidate in another behind the scene details, I wanted to call the book “Invalidate Product Ideas” because that’s what mostly happens but Lou though thought it was too negative so we decided to go with validate. So...
Gerry:
You could have called it “Bullshit Product Ideas.”
Tomer:
Yeah, yeah we had many options. [Laughter.] I should publish that, just a list of the options. So the key word there was it’s a process, okay? It’s not necessarily a product. The second thing it could be is a prototype, with minimum functionality that facilitates learning. Learning is the key point there. And lastly, and this is how I like to call it, it’s an experiment. We’re trying to learn something, we’re trying to make the minimum effort to learn as much as possible. And that is not necessarily something that looks like a product.
So here is what an MVP is not. It’s not a cheaper product, nobody said that an MVP is cheap. It’s not necessarily a minimal version of a product with the smallest possible feature sets. That’s not…
Gerry:
Which I guess is the typical definition, isn’t it?
Tomer:
Yes, yes. But that’s but that’s not the definition. The definition is what we can learn, the maximum learning based on the minimum effort. It’s not necessarily a smaller set of features. It’s not designed to scale across the entire customer base, not having any presumptions of that and it’s not necessarily the first version of the product. And also it’s not, and this, I’ve seen that a lot as well, doing the same thing that you did before; you heard of MVPs, and then calling it an MVP. So if you used to do, used to be doing prototypes, now you call your product by its MVP, that’s not necessarily the case.
I know it’s kind of general but I want to give some examples. I have, I don’t know, twelve of them, I’m not going to, we’ll see. An interview or a phone call could be an MVP based on what you talk.
Gerry:
Can you flesh that out a little bit? How could that be an MVP?
Tomer:
You can test, first of all you’re not developing anything so it is a minimum effort. You are trying to learn, okay? So sometimes you learn by a conversation. I know I’m doing what I just said, calling things differently but you can learn with a minimum effort. We all know how we can learn in interviews so I just wanted to demonstrate that an MVP could be a conversation. It doesn’t have to be a product.
It could be paper prototype. It could be a pre-order page, so put a page online for a product that doesn’t exist yet and see how people respond to it; do they want to buy it? If they are willing to pay for it. That’s a good MVP. A blog post, an online ad campaign, a video even, a lot of start-ups like that. They produce a video about a product that doesn’t exist yet and see how viral it goes. If a lot of people are interested, maybe there’s something there. It doesn’t mean that this is it, that everything is validated based on the response to the video but that could be an MVP. A contract… I know of a company that, a start-up company, that decided that, I can’t remember the product it was maybe a system for ordering lunch for enterprises, for employees in enterprises, so he went to different enterprises with a one page contract that explained the product and how much it cost and decided that if he had 500 signatures to him, the idea is validated and he will you know make it happen.
So a contract could be an MVP. And you know other things like The Wizard of Oz that you pretend to have a product and see how people respond to it and so on and so forth but just wanted to demonstrate that it’s not necessarily a feature or a small version of a product or a prototype.
Gerry:
And I guess you’ve alluded to it there but also when you mentioned Wizard of Oz and fake doors, similar sort of things, right?
Tomer:
Kind of yeah. Fake door, it’s a small thing. It’s an experiment, again, and it’s a minimal viable product. You pretend to provide a product feature or service and what I’ve seen is that it mostly happens on the Web or in apps. So without developing anything, it’s similar to what I described earlier with a page, you communicate to visitors that the thing exists and you ask them to act on it. If they do you know they want it and it’s time for you to start working on developing it. Of course you should do more research but the example I have, and usually it’s very small things, so it could be a button on a website that says “download our shopping app” and as soon as you click it the label changes from “download shopping app” to “coming soon” or “in progress” or something like that.
Sometimes you see that in websites in navigation bars. You pick a certain option and then it changes to “coming soon”. So they’re just testing to see if you’re interested in something. Of course that’s not you know the entire research that lets you go into deciding whether to develop a product or not but that’s a small way, a fake door, that can tell you if there’s interest.
Gerry:
Of course it wants to be guided because there’s nothing more frustrating than “coming soon” links that have been there for ten years and you know they’re never going to come
Tomer:
Yes, it should be you know a short timed experiment, not something that’s there for ten years.
Gerry:
And if you had to pick out, you describe in great detail many techniques in the book and I must say the descriptions are very thorough and well supported and this is the sort of book one could give to a product manager or a developer and say, look here, do these things and you could see them running with it. But if you yourself had to pick say two techniques over any others, what would you pick?
Tomer:
So I’ll go with Caroline here. The first one, the first one is observation. There’s nothing like being there, observing what people do. The thing is that you learn so much that sometimes you don’t know what’s important. So it’s really important to watch what people do, not just stand there, but watch what people do, listen to the language and jargon that they use, that can be extremely helpful. Notice things, pay attention to things. And in the book I think I have a list of ten things to pay attention to. So you’re pretty much like an investigator, you’re trying to find evidence that things are happening. And then gathering different artefacts, you don’t gather anything, you just take photos of things, but gathering these artefacts can be extremely helpful in what these product managers want. They want features. Tell us what features we need? So observe what people do and gather artefacts. I used to work in a company where we had a very complicated product, networking and firewall security product, and I went and visited people in their workplace and I almost even before I started talking to them I looked at their walls and saw what charts and maps and things they created on their own to help them work with our product. These were, this is the exact list of missing features for the product. This is by the way how they saw the problem that we didn’t. So that would be the first technique, observation.
Gerry:
You reminded me there, I think it’s Paco Underhill “Why We Buy” or “Why We Shop” talks about one of the key observations that they made in their research was that people have two hands, thinking about the need for putting, adding baskets into appropriate places in a bookshop, increasing the quantity of books being bought. It’s such a simple, obvious thing but until they’d observed it they didn’t know that this was a problem for people.
Tomer:
That’s a great observation. [Laughs.] And the second one, a kind of a personal favourite of mine, is experience sampling. Experience sampling is a technique for, it’s kind of a combination of a qualitative and quantitative technique in which research participants are interrupted several times a day to note their experience in real time. This used to be called pager studies in the 1950s. So back then the researchers asked people if they paged people several times a day asking them “how do you feel?” or “where are you?”
Today we have Google so we don’t need to ask that question but you can add the key thing is asking the same question over and over again. And speaking of Google I can give an example from Google from my time at Google because this is public knowledge so we can look for it. We ran a study, an experience sampling study every year starting I think 2011 that we called DIN - daily information news. We wanted to know what people want to know, with or without connection to technology or Google. So we just asked them, we ran an experience sampling study with many, many participants and asked them for three days, I think eight times a day, “what did you want to know recently why?” And their answers were tremendously helpful in understanding what we don’t do at all. So I’ll give you two examples, answers that were not really interesting for us were, for example, “How tall is Tom Cruise?” So this is something that Google has known how to answer for a long time now. But questions such as “Am I a good writer?” or “What colour should I paint the big wall in my living room?” These are questions, try Google, I mean you’re not going to get any answer, any reasonable answer to that. So I’m not saying Google is going to answer these questions but when we looked at the thousands and sometimes dozens of thousands of answers to that same question, we didn’t change the question over the years, you see some patterns there and some of the patterns are pretty almost mundane and predictable. So we saw, for example, that people and we tracked the time that we asked the question. So we saw, for example, in the morning there are four things that many, many, many people want to know and I bet you can guess these.
Gerry:
Well, I’m guessing one would be the weather.
Tomer:
Right.
Gerry:
[Laughs.] I’m not going to make an attempt to get the three because they’re gonna make me seem weird!
Tomer:
No, so weather is one, traffic is two, news is some cultures, in many cultures, news is three. And “What’s the first thing on my calendar?” is four. So it’s pretty boring and then we asked ourselves why if we know that this is what people want to know and they wake up in the morning why should we you know force them to ask? Why don’t we give them the answers to those questions right when they wake up? So that’s how, one of the reasons why Google Now started for example.
So experience sampling is a really nice technique that anyone can run. It’s very simple to run.
Gerry:
In my household we’re totally addicted to Google Now and my younger son can fake my voice and my wife’s voice to ask our phones questions.
Yeah it’s interesting. Experienced sampling sounds awfully intrusive and simultaneously reading your book, I was reading David Eggers The Circle, have you read that?
Tomer:
No I did not.
Gerry:
It’s kind of based on a fictitious, well you know it’s clearly based on a mashup of Google and Facebook where this large organisation begins to know everything and probe into everything and of course it ends up in a sort of dystopian mess as you can imagine but it’s worth a read.
Any case I’m side-tracking us. One thing I really liked about the book was the fact that it’s really specific in its instructions and lots of support materials from people who are actually carrying out the work and if Julian hadn’t stolen my copy I’d have that list in front of me. But do you want to tell us a little bit about the support materials that you provided in the book?
Tomer:
Yes, that’s another thing of writing to non-researchers. So you know after I wrote the Lean Startup book, I thought to myself, yeah it makes sense, it’s great but you didn’t really have instructions on okay, what do I do next? So if people really want to act on what I write in the book they need templates, they need checklists, they need demonstrations. So I included all of that. So I had templates for taking notes and templates for discussion guides and screeners and one-page research plans and something I call a MVP board and others.
Every chapter ends with a short checklist and lists or just short descriptions of other methods that can help answer the questions that we pose there, and I also have an accompanying YouTube channel with exercises. So one that I really like is an observation exercise, so to practice paying attention to things. So I recorded a woman who is doing her grocery shopping and I asked her to think out loud as she was doing that and it’s a 20-minute video so I have that version. People can watch it and try to pay attention to different things. And I also have created the same video with annotation of what to pay you know here are the things that you should pay attention to as they happen. So this demonstrates how to identify these moments of learning.
Gerry:
Other than reading the book, what key advice would you give to listeners who are contemplating jumping into new product development?
Tomer:
So when I think about it, people have to know about the book, then they need to decide to buy it and then they should…
Gerry:
Or steal it.
Tomer:
Yeah, or steal it. And then… get it somehow… and then they might read it and then the very few are actually going to do something about it are the important ones.
So my recommendation is just do it. Don’t just you know hear about it but buy it, read it. Actually do it. I say in the introduction that it’s not a reading book. It’s a doing book. I don’t see why people would read it cover to cover. I see them look in the table contents, identify a question, the book is organised by questions that people have about their products or users. Identify a question they currently have and then dive into the chapter and just do it.
So I would just recommend not waste, and this is especially for start-up founders, not waste their savings and time and energy on something that nobody needs and just validate, and in some cases unfortunately invalidate, what you strongly believe in. This is especially hard for them to accept. I always tell them to you know what is the thing about your product that you’re mostly confident about. Okay, so let’s look at that. Let’s see if that’s true. And then it’s a hard step for them to make, to recognise that the things in their minds, the facts in their minds, they’re not facts, they’re just assumptions. And if we have time I also have a list of five risks that people have when they actually jump in and do research.
So, one would be, and I see that all the time, especially again with start-up founders, they imagine that they do research based on the fact that they talk with customers. So talking with people is not research. Research is not, again, rocket science but it has some rigour and you have to know some things and how to do some things. So that’s one risk.
Second, I would say is not acting on results because they are unexpected, unintuitive or surprising. That’s how life is. In most cases you are invalidating what your assumptions were. People, in many cases use wrong, the wrong method to answer great research questions; for example, using a survey to figure out if customers would use a product or feature. It’s kind of inherently embedded in the book. I picked the methods to answer the right questions. So when you read the book you don’t have to think about what methods will answer the question that I have. I did that work for you.
And the fourth one is biasing participants in participants with questions, you know, “Would you use this new improved product or app?” – kindly hinting that they think it’s improved. Bias participants with the location of the study. So you invite them to your cool office, they want to be nice back to you so they’ll change their behaviour or opinion. And with body language, if a person participating in research is sitting down and you suddenly stand up above their shoulders and they can feel your breath, that’s biasing participants with body language.
And lastly, focusing on identifying quote unquote “missing” product features rather than uncovering true user needs. Features is what people are mostly interested in and they should but first fall in love with the problems that these features are supposed to solve.
Gerry:
You kind of remind me there, as you’re talking about that, how much the world has changed. I was in a meeting about two weeks ago and one of the chaps who is seconded to our project is a policeman and he said ‘Oh, we started this pilot and a whole lot of things went wrong, which was great”. I thought wow that’s a new world, isn’t it, where you stand up and say things went wrong and isn’t that great?
Tomer:
Yeah, I mean I always like to say that if a research study goes well then we didn’t learn much. We need really things to break.
Gerry:
You need to get the people who hate your product and your idea much more than you need the people who say they love it.
Tomer:
Yeah. That’s true.
Gerry:
The name of Tomer’s book is Validating Product Ideas Through Lean User Research and you can get a 20% discount by quoting UXPOD on the Rosenfeld Media website. I think that applies to any of the Rosenfeld books. Tomer Sharon, thanks so much for joining me again today on the User Experience podcast.
Tomer:
Thank you very much, Gerry.