Published: 30 September 2014
What you can (and can’t) find out in five seconds, and how to design good tests to do so.
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast.
Today’s guest is manger of user experience at Thomson Reuters in Boston. He graduated from Bentley University’s Human Factors and Information Design Masters degree program.
He worked on complex UX projects and learning technology in the legal domain and in intellectual property.
Recently, he published a book, The UX Five-Seconds Rules: Guidelines for User Experience Design’s Simplest Testing Technique.
Paul Doncaster, welcome to the User Experience podcast.
Paul Doncaster:
Thank you very much, Gerry. Pleasure to be here.
Gerry:
I guess we should start off by asking what is a five second test?Paul:
Sure. Well, the best place to find that out if you have a mind to is to go to Jared Spool’s user interface engineering design. There’s an excellent article and podcast there that outlined the originations of the method; how they came to it. So originally it was devised to test content pages only, as opposed to home pages, and it was supposed to answer the question: Is the purpose of this page obvious? And the method was, very simply you put up an image for five seconds, take it away and then ask one or two questions that address that goal. Is the purpose of the page obvious?
It’s so simple and it’s almost criminal how simple and basic the technique is. In fact one of the comments I saw after writing the book when it came out is “It’s basically akin to writing a book about using a fork.” And I would agree with that if people were using it exclusively to eat solid food. But when I saw evidence of people using a fork to eat a bowl of broth and were convinced that they were still full I thought, you know, there was maybe some misunderstanding going on here about what a fork is actually for and the best way to use it. So you know I started digging deeper into it.
Gerry:
I was pleased to see that while you were obviously an advocate of the method you’re also a critical one. Perhaps you could tell us what a five second test is not good for?
Paul:
Sure I’m an advocate for the technique in certain situations as with really any UX method, right? It has to be the right tool for the right job. The big point I try to make is, in here, is it’s not a good technique for using it when longer than five seconds is needed to answer a question in a thoughtful way. So in keeping with the original framework of the inventors of the method for content pages, “What is the purpose of this page?” is an absolutely viable question and a good use for the technique. Another one would be to get an emotional response, to gauge an emotional response to a page or a design. So asking the question “What one word would you use to describe this, the design of this page?” is probably a very viable use of the technique. However, asking questions like, you know, “How would you improve this page?” or “What changes would you make to this page?” after seeing an image for five seconds really isn’t viable for pretty obvious reasons, I think. You need a lot more consideration to be able to answer that question in a thoughtful way.
Obvious things like predicting future behaviour. I saw a lot of tests where people would, you know, put up an image up and say “Would you use this site?” or “Would you use this service based on what you’ve seen here?” and, you know, obviously that’s not going to be a useful use of your research time, using a five second test to answer questions like that.
Gerry:
In fact it’s got to be very hard to answer a question like “Would you be inclined to do something?” or “What are your future actions going to be?”
Paul:
Yeah, any UX method is going to be pretty poor for that. That takes a whole other realm of research to address a question like that.
Gerry:
So couldn’t we sort of address those things by making it a 10-second test instead?
Paul:
Within certain guidelines I’m sure there are certain questions that could be addressed in a 10-second test. You have to be very, very careful when you’re doing anything that’s timed because you’re dealing with memory and memory fade, as you know. You look at something, you take it in, you chunk out what it is, you know your brain chunks out what it’s going to retain and then you’re asking questions about it. And for each question that you ask, your memory, you know the cognitive resources needed to process the question that’s being asked and inform the response and give the response fades, the memory of what you just saw, right? So it’s much more difficult to answer the second question and the third question.
So, you know, extending the test might be good for certain instances but then you risk, you’re further jeopardising what it is you’re trying to learn.
Gerry:
Paul, one of the things I liked about the book was the way that you’ve taken things like the effect of memory fade on question/response rates and examined them and done some analysis and in fact detailed them in the book as well. You’ve obviously put a lot of time into examining this method. Can you tell us a little bit about your research method?
Paul:
So originally the test was meant to be a face-to-face aspect of a larger effort, right, that throw in a couple of five second tests within a larger usability test to get the answers they were looking for about content. Flash-forward to 2011 and 2012 when I became acquainted with the guys at Usability Hub and their tool called fivesecondtest.com which is a way to administer these tests over the web. They have a way to accumulate what they call karma points which allows you to build up points that you can then use for your own tests so people can, so you can get larger and larger response rates for free basically, and the more I took the tests, the more I found myself saying, you know I can’t answer this question or I have no way of answering this question or I can’t remember what it is you’re talking about. And I felt horrible about it, right? I’m a UX person. I’m doing it in good faith. I want to be able to provide people with valuable information but the way I saw questions worded and structured and instructions that were given and images that were presented, I just saw, I realised at that point that they were a lot of people using the method either in an uninformed way or in an incorrect way.
So what I did was I collected a sample of about 300 of these tests on that site over a period of a few months and I put them all on a spreadsheet and I analysed basically front to back, how the instructions were worded and presented. How the images were presented. How questions were worded. How many questions were used, the order they were presented in, the types of questions that were asked, every possible angle I could think of and then I identified trends that I thought were critical or that really stood out and at the end of the day I came up with the conclusion that only about a quarter of the tests that I saw represented reasonable executions of the technique and the rest had varying degrees of problems. So that basically forms the basis for the guidelines that I lay out for using the method, using these online tools.
Gerry:
So in summary, what’s the right way to use the tool?
Paul:
Well, there’s a number of right ways. The original intent is a right way, putting up an image of a content page and asking “What’s the purpose of this page?” That works. But that does not work for home pages, for reasons we’ll get into in here in a minute. You know when you’re using online tools around which research is based it’s good to, it’s good for things like memory dump. If you put up an image and say, “Tell me everything you remember about the image that you just saw” and you don’t go any further than that that’s a good gauge because you can take the responses and analyse them and see if the things that you want noticed about a page or the things that you want retained about a page naturally come forth in the user’s response. It also can be good for target identification questions, so determining whether certain things, visual elements on the page are obvious that you want to be obvious. Say you want to call out, you want to make sure that a special offer label or sticker on a page is obvious. Putting it up for five seconds and asking whether or not something was noticed… can be a good use of the tool.
Attitudinal feedback is another good one: asking the question “How would you describe the page that you just saw?” or “How does it make you feel?” You can then take that data that the user provides and you know make sure that it aligns with the goals that you’ve set out for the page.
Gerry:
So I would gather that the five second test is something that one should be doing when you’ve got a reasonably well visually treated representation of what it is that you want to look at?
Paul:
I’ve seen a lot of instances where it’s been used with wireframes, low fidelity wireframes, and that can be a good, it is viable in certain instances of that but again it’s, the whole idea here is paying close attention to what it is you want to learn and understanding whether or not displaying an image of it for five seconds or a representation of it in five seconds is a good use of your time or a valid use of your time.
Gerry:
So you mentioned that the test is not good for home pages but I’m sure I’ve seen people advocate its use for home pages and landing pages.
Paul:
If you’re true to the original intent and ask “What’s the purpose of the page?” - the purpose of most home pages is pretty obvious. It’s a gateway to more granular data. Most people don’t go to a site to look at a home page. They go to get, they go down deeper into the content pages and understand, get the information they’re looking for there. So to ask the question “What’s the purpose of this page?” really begs the question, begs the answer, I should say, “It’s a home page.” That’s what it’s there for. It’s to identify the company and provide a gateway to further information.
So if you were going to use it to test a home page you would use it in that emotional response sort of way, that’s one example. You could say we’re going to show you a home page, you show it and then say “How does this…,” [or] “Describe this page in one word or two words.” “What is your reaction, your first formed reaction when you see this design?” You know, not getting into specifics about content because a five second test is clearly not good for anything that requires you to read content, but it would be good for home pages if you are just gauging reaction, emotional response or emotional reaction to the visual design of a page.
Gerry:
Sure. Now one thing Paul that would strike many people, I suppose, is the five seconds. It’s a very neat unit of time. Why not three seconds?
Paul:
I’ve asked that question many times and haven’t been able to get a good response to it. I don’t know. I’ve looked in the literature and not come up with a reason why it’s five.
Gerry:
Other than “five works.”
Paul:
Yeah, that’s about the gist of it.
Gerry:
One thing I liked about the book is that you talk about something, you know you take some examples of tests that you’ve gathered during your analysis phase or your research stage, rather. Then you describe some of the issues with it and from that then you derive specific rules or guidelines, as you just referred.
As a matter of fact that comes out to a very neat number as well. You’ve got ten rules in the book. Was that a deliberate aim of yours or did it just come out like that?
Paul:
Well, you know in the early iterations of the research it was going to be the ten commandments of using the five second test and… I’ll borrow from George Carlin here; ten is a psychologically satisfying number. It’s nice and round. People are comfortable with it. I’m sure I could extend this out to a couple more if put my mind to them but I think I started work on that actually at some point but I figured ten was good enough for now and, you know, if we need to extend out obviously people are going to find other guidelines or other rules that would be applicable to those specific methods. I’m sure there are more than ten.
Gerry:
I found the ten very interesting. Obviously they’re specific to the five second rule but a lot of them are more generalizable, if you like, to other fields of research also and given that really, from my perspective anyway, you know correct me if I’m wrong, but this seemed to be the core of the book, the ten rules.
Do you want to go through and maybe give us a little bit about each rule in turn and where it derived from and what the implications are in using it, if that’s not too big a question?
Paul:
I could certainly touch on each one and give a little bit of information. And, as you say, a lot of it is not specific to the technique as used in these online tools but it’s just basic good old-fashioned UX understanding of survey design and research design, right? Going back to the idea of knowing what it is you want to research and limiting it to that and getting the most out of the time that you have and the tool that you have.
So the first one is, again UX research 101; don’t use it when a different method is more appropriate. This is not specific for the five second test, that’s appropriate to any research challenge or design challenge that you have in front of you. You wouldn’t use, you know, a card sort if you wanted to test a usability issue, right?
Gerry:
It’s funny that you say that, we particularly want to pay mind to that for a five second test because I can just see, you know, project managers saying let’s just use the five second test, it’s quick and it’s cheap.
Paul:
Yeah, absolutely. Those who are the uninformed would definitely say that I think.Okay, so the second rule is focus on the specific design item that you want to test and the specific aspect that you want to test and limit the test to that item or that format. I already mentioned the memory dump test, the target identification test and the attitudinal feedback test. Those are three viable design issues or design items that you can viably put this technique to. You can and I have seen people, a ton of people try and mix up those, some memory dump questions and attitudinal feedback questions.
You can do that, again, if you’re very, very careful. If you limit your questions, if you choose them and word them correctly, if you order them optimally it is possible to sort of mix up a question from each one or a question or two from each one. But, again, you’re against the clock in the memory aspect, right? You want to leverage the memory of the person who’s providing the feedback in an optimal way. So, I wouldn’t recommend mixing them very often.
Gerry:
In that regard, Paul, is it okay to mix, you know, to use the five second test as part of say a standard usability test or some other user research method?
Paul:
Sure, absolutely. In fact, Jared’s original method outlines that. It was sort of a throw-in adjunct usability technique thrown in in a larger effort. So, yeah, if you were going to do a usability test, it’s definitely viable to throw in a couple of these.
Third one is don’t give your user any excuse to say “I don’t know” or “I don’t remember.” You’ve put a lot of time and attention into creating your test. There’s nothing worse than launching it using one of these online tools and then getting back response after response after response saying “I don’t know” or “I don’t remember.” That all feeds in to, again, paying very close attention to the test as you’re developing it; your instructions, your image and your questions.
Those are the first three and I think the rest of them, to your earlier point, if you follow those then the rest is basically slight variations on standard UX practice, survey design, testing design.
Number four is devote time to crafting proper instructions. For example, in memory dump tests, instructions can be very general. It’s easy as; “I’m about to show you an image for five seconds” and that’s it; show the image for five seconds and if the question is “Tell me everything you remember about it,” you don’t need to have anything beyond that in your instructions.
If you’re going to do a target identification test you need to say something like; “I’m about to show you an image of a page and I’m going to ask you a question about a specific target or a specific aspect or a specific visual element on the page,” without giving any more than that. Then when you show the image, you can come back and say something about the item that you wanted memorised or see if they remembered.
Number five is optimise the test image so that scrolling is eliminated. I saw a ton of tests where people uploaded images that required not only horizontal scrolling but also vertical scrolling in order to see the entire image and you only have five seconds. You want your participants locked in on the image and any time spent, you know, with eyes wandering all over and moving the mouse over to scroll over to the right or down to the bottom is precious time lost, and they’re not going to remember anything when they come back to answer the questions, or they’ll remember very little… when they come back to answer the questions.
Gerry:
We should clarify for people that what we’re testing is actually an image of the home page… sorry the home page! Excuse me! We’re testing an image of the page, so we’re not looking at an HTML rendering.
Paul:
If you’re using the tools that I use, no, you’re restricted to uploading a PNG or a JPEG or a GIF or something like that. It’s a screen shot that you’re testing, not live code. That may be the next step for any provider of these online tools but I’ve not seen that yet. All of the ones that I’ve seen accept just images.
Number six is there’s no magic number for how many questions to ask but fewer is usually better. And, again, this speaks to the memory fade; the fewer questions that you ask, the more likely you’re going to get a good informed response.
If your test includes one question that’s going to be answered with the best of the user’s ability based on their memory of what they saw. If you have five questions, by the time they get to the fifth question, memory of what they saw is eroded such that only certain types of questions can be asked by the time you get to the fifth. You can ask something like; “Anything else you want to tell me about what you remember?” after asking more specific questions before that. But I think it’s logical that as long as memory’s in play and you understand that memory is going to fade over the course of answering, you know, internalising questions that are asked, forming responses, typing in responses, moving on to the next question, that your memory is going to go, memory of the image that you saw is going to go away and the ability to answer questions is going to be less informed.
So the next, rule number seven is ordering the questions optimally and that speaks to solid, reasonable survey research design. You don’t want to bias your users by asking questions that feed directly into the next question. That’ll affect their response. But you also don’t want to go back and forth between question types and an example in the book talks about a test where there are five questions where the first two were target identification questions. Things like “What was the name of the company on the page you just saw?” and “What was the most prominent element that you saw on the page?” And then it goes into attitudinal questions like; “What did you like most about the design” and “What was most off-putting about the design?” And then the last question goes back to target identification; “Can you name some of the products that this company appears to sell?”
Well question order there… First of all you should have narrowed the test down to either target identification questions that have been only three, or attitudinal questions which would have been only two. But if you’re going to put all five of these in here, which again I would not recommend, the last target identification question should be grouped with the other ones right up front so that the memory is fresher. The memory of specifics of the pages is fresher.
You can answer those questions in an informed way and then, at the end, go back to sort of how you felt about it or what did you like about it or what was most off-putting? That’s what I mean by ordering optimally.
Gerry:
And Paul, one of the other things I liked about the book was the way you would hypothesise for example that that sequencing of questions would not work and then you’d actually test it yourself by running the test and looking at the fall-off in appropriate answers.
Paul:
Yeah, well I mean I figured that was the only way to see whether the rule was valid, right? To run a couple of tests myself using the rules and see whether or not I got the results I expected and by and large, yeah, I did.
Okay, so the next one is writing questions; pay particular attention to how the questions are worded. There’s a number of items here, and this is sort of the biggest in terms of detail, this is the biggest area, the biggest rule in the book and, again, it’s not rocket science. You need to understand what it is you’re asking. You need to be thoughtful in how you phrase it. These tools almost beg you to be very, very concise what it is you ask but in a way that there’s no room for, you know, misunderstanding or begging the “I don’t know” response.
So, I’ll give two examples of what I call out here. First is the prime to repeat. A lot of tests that I saw would basically provide the answer to the first question in the instructions. There were a lot of tests that said something like; “You’re about to see a website about wedding planning.” And then the first question would be “What was the purpose of this site?” Well, you’ve just thrown away your best opportunity for informed feedback and when memory is freshest because you’ve already prompted the user with what the answer’s going to be. So that’s basically a throw-away question.
Another one is, you know, begging the “yes” or “no” answer. A lot of questions asked “yes” or “no” questions; “Did you notice the element that I told you [that] you had to pay attention to?” Well, of course you did because you called it out in the instructions. But worse than that there’s no “why” behind it, right? If you take the “yes” or “no” question and re-write it so that you get the answer, provide them the opportunity in their words to provide the “yes” or “no” answer then you’re getting much more richer data that you can use to go back and refine your designs.
Gerry:
Yeah, there were lots of examples of really, really bad wording and people doing things like cramming three questions into one. You know, the sorts of things that you’d sort of look at and you’d think there was a committee involved in designing those questions.
Paul:
Yeah and you have only… the thing about these tools is they provide response boxes for you to type your answer in but you know they’re one line responses and they, you know by the very visual nature of them encourage brevity in their response. So to ask a question that really had three questions in it you know it’s all the more reason for the user to say “Pass on this question” or I’m not going to answer it because there’s too much going on and not enough area, visual area to provide an informed response.
Paul:
The ninth rule is asking the most prominent element question. There’s a lot of, I saw an awful lot of examples where the test question was “What was the most prominent element on the page?” And I make the argument that if you’ve designed a page in a certain way, you’ve done it with thoughtfulness as to what the prominent element is, right, whether it’s the you know the name of the company or whatever. So, in that sense it’s sort of defeatist to ask the question unless you have a very dense page.
But aside from that, I saw a lot of uses of this question; “What’s the most prominent element on the page?” when there was a large photograph on it, like a sunset or something like that. And it’s, I mean again it’s self-defeating to ask what’s the most prominent element on the page just by size or by colour value if it’s a bright yellow or something like that. Basically, in most cases the answer to the question is inherent in the design. So in certain instances asking what the most prominent element on the page is could be viable but, you know, in the vast majority I’d say it’s basically a throw-away item.
Then the last one is open ended feedback request carry a higher risk of non-responses and low information and so typically at the end a tester will ask; “Is there anything else you’d like us to know about this design?” or “Is there anything else you want to tell us about whether or not you liked this or not?” They’ve already gone through a number of questions. They’ve already answered specifics, again, in most instances. By that point and given the fact that they’ve only looked at something for five seconds, there’s really not much else you’re going to get out of them. In fact this is usually a prime opportunity in the tests that I saw for people to provide snide answers. You know, things like; “Go back to the drawing board. This is horrible” or “I have absolutely nothing to say more about this.” So if getting open-ended feedback using this method is important, ask for a highly focused test restricted to asking for improvement recommendations. Again, you run the risk of not getting very useful data but at least if you put up the image and say something like; “Just tell us anything you want” you know at least that’s going to be sole instance where you’re probably going to get some useful data. Not tackling it as a question, as a fourth or fifth or, you know I wouldn’t go any higher than five questions under any circumstances using this test, but for a fourth or fifth question, you know five seconds just isn’t long enough to provide any thoughtful reflection on what else you can say other than what’s already been asked.
Gerry:
Are five second tests good for doing A/B testing?
Paul:
I wouldn’t think so. That’s... A/B testing is for comparatives and you know I guess you could run simultaneous separate tests asking the exact same questions for two different designs but you know A/B testing to me speaks to usability issues and things like that.
If you ran simultaneous tests for emotional design, if you have two options that you just want to get gut level emotional feedback on, I suppose you could do it that way. But if you’re looking to do A/B testing my guess is your research question is more involved or more you know more deeper than what a five second test can deliver for you.
Gerry:
Sure, so Paul, do you use the five second test yourself on a day-to-day basis?
Paul:
Day-to-day I did actually for quite a while. It’s really good for settling, I’ll tell you what it’s good for, it’s good for settling the internal bar bet type of questions where you have a team or internal stakeholders that have definite opinions about one thing or another. They think that one approach is going to work or one design is going to elicit an emotional response and another faction thinks it’s not and so you do a test and that sort of settles it.
I did in a certain type of project that I was on. A lot of what I would call bar bets came up internally where one faction of stakeholders thought one approach was going to work and another thought another was going to work. I would say; “Alright, give me a half an hour and I’ll get back to you in a minute.” I’d go plug in a five second test. I typically would get results back in 20 minutes or an hour and then later that day I could say well here’s what I’m seeing based on the questions that I asked and, you know, it’s either trending this way or trending another way.
I would never ever use the results of five second tests for sort of the be-all-and-end-all you know going forward with it. I would use those results internally to say; “Well, the direction trends this way based on what I just saw. But we need to test it further using you know in a larger effort, in a more formal effort to get confirmation on it.”
Gerry:
I know there’s several tools that facilitate running these tests. Can you give us an overview of what they are and how they work? I don’t know if there’s any you particularly endorse or not but you can give us the landscape perhaps?
Paul:
I’ll give you three that I’m aware of and I’m sure there are others out there. fivesecondtest.com is the one that got me started on this research effort first of all so that’s the one I’m most familiar with. That’s from the guys at Usability Hub. I’ve used it a lot. They’ve been very helpful and forthcoming in giving me the resources to investigate this further and I’ll re-state; one of the things I really like about it for those of us, and I’m one, who always have to fight the constant battle for research money and budget, things like that, they have it set up so that if you’re a registered user you can accumulate karma points, what they call karma points by taking the tests of others and then you can then use those tests to launch your own tests and you don’t have the ability to, using this, you don’t have the ability to focus, you know, get a special user persona about it unless you take a link to a test and send it to people in your contact list who do fit that persona.
If you use this karma point system, you’re basically getting other testers. But for questions about design, a lot of that can be useful, especially settling those bar bets type of thing. So that’s the one I’ve used most frequently. I know that there’s a Verify app and it’s at ZURB, I think it’s zurb.com that has a version of five second testing. They call it something different and it’s a lot less flexible. I think what they do is allow you to show an image for five seconds and then provide five response boxes that says, you know; “Give me the five things that you remember about it.” So you can’t really customise your questions. And I know the people at UserZoom have a feature that does this very thing. I think they call it “time out testing,” where you have the ability, back to an earlier point you made about customising the time. So you can, you’re not locked into five seconds, you can go to ten or as little as three or whatever and you’re able to customise your answers. That’s a for-fee basis. You have to be a paying customer of UserZoom. They don’t have a free service whereas fivesecondtest.com can be free. There is a paid version as well that gives you more robust functionality and the Verify app I believe also has a pay version as well that probably gives you a lot more to work with rather than just the five response boxes.
Those are the big three ones that I’m aware of.
Gerry:
Okay and besides reading the book of course, do you have any advice for practitioners who may be considering using this technique for the first time or maybe used it and were using it inappropriately and thought it’s not for me and they now want to come back to it based on, you know, this conversation or on what’s happening in general.
Paul:
Don’t be limited by what the initial purpose of the test was. I think as researchers and UX professionals we’re always curious as to whether the tools that are available to us can be stretched and used in different ways. Now they have to be viable and valid ways, right? You don’t want to be using the wrong tool for the wrong purpose, like I mentioned at the outset. But I think in tackling this issue, or in looking at this tool I really got an idea that, you know, not just the question; “Is the purpose of the page obvious?” That’s not the only thing that can be asked based on a five second response. There are other things that can be asked and answered in a viable way.
So, you know, beyond just being curious, seeing if there aren’t any further limits to the types of tools I’ve talked about, the advice I would give for using the method is knowing the background of what it is you want to learn, knowing the limitations of the tool you want to use, knowing what your research question is and just, you know, experimenting out with it. There’s probably a lot, always proceed with caution, you know there’s lots of ways to fall when you’re using any type of online tool that you don’t have complete control over.
But there’s also I think a lot of opportunity if the mind is expanded and, you know, you let yourself think about tools and research questions in different ways.
Gerry:
So cautious enthusiasm.
Paul:
Yeah absolutely. Yeah. Very shortly after the book was released I came across a post online of a guy called Craig Tomlin who’s a marketing conversion optimisation consultant and he has adopted the five second test for use to measure conversion optimisation in a very sort of detailed way on a blog posting. If you Google usefulusability.com and look up five second test within his archives, he took the method and the possibility of using it for different means in a completely different direction that I hadn’t considered.
So you know, more to the point, there’s no limit I don’t think other than your imagination and keeping within good UX practice to take a method like this or tools that are available to execute a method like this and see what else is out there that you can do with it.
Gerry:
Paul’s book is The UX Five-Seconds Rules: Guidelines for User Experience Design’s Simplest Testing Technique. And I must say I really enjoyed the book and I found it was great to be reading something that had robust day-to-day advice included in it on a method that I hadn’t particularly used in the past but will look forward to using in the future.
You get 25% discount on this and other books at Elseviers’s online store by using the code PBTY14.
Paul Doncaster, thanks for joining me today on the User Experience podcast.
Paul:
Thank you very much. It was my pleasure.