Published: March 2014
Aga Bojko talks to Gerry Gaffney about the magic of eye tracking for UX research.
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast. My guest today is a UX researcher, originally from Poland. She’s currently vice-president of ux at Gfk Custom Research in the USA. She’s a former managing editor of UXPA’s user experience magazine. She has a B.A in psychology as well as a Masters in Human Computer Interaction and another in Human Factors.
She’s got a particular interest in eye tracking and has written an excellent book on the topic: Eye tracking the user experience: A practical guide to research.
Aga Bojko, welcome to the User Experience podcast.
Aga Bojko:
Thanks Gerry.
Gerry:
Can you start by telling us what’s meant by eye tracking?
Aga:
Sure. I’m assuming that … most people who are listening to the podcast will already have some familiarity with eye tracking but in a nutshell eye tracking is a process of identifying where someone is looking and how. Typically it’s done with the help of a device called an eye tracker and it can be done for several different purposes. For example, you can control interfaces using your eyes. You can play a game with your eyes or you can type just using your eyes, but when I talk about eye tracking and what the book is about is using eye tracking for… design evaluation basically.
Gerry:
And how did you get interested in eye tracking in the first place?
Aga:
Well it was a long journey and it started in graduate school and that was about 14, 15 years ago at the University of Illinois. Our human factors lab had an eye tracker. It was one of those older eye trackers and it was just a tool that I was doing research with and my research was on cognitive signs of ageing. And just to give you an idea of what kind of research it was one of the papers that I published during that time and it came out in a journal called Psychology and Ageing was titled Age equivalence in switch costs for prosaccade and antisaccade tasks.
Gerry:
Sounds great!
Aga:
Right, right doesn’t sound UX related at all, and I really wanted to use eye tracking… I just wanted to do something more practical. So, at the time I was just using eye tracking for basic psychology research and what I would call traditional human factors research and then in 2003 I joined User Centric, which is now the UX division of Gfk in the U.S, and during my interview, and I think that’s the kind of how they, that’s why they wanted me on the team because during my interview they said, “We have an eye tracker, we just got it, we have no idea how to use it. We don’t know what to do with it.”
So then it kind of became my job to figure it out… the method that I already knew, applying it to something a lot more practical and then my eye tracking research became about understanding how people experience products and interfaces and about improving designs.
And I feel like my mission has always been to bring some of that academic rigour that I was so used to to more practical type of research.
Gerry:
That’s one of the things I really enjoyed about the book. You also co-authored a nice article in UXPA’s User Experience magazine back in 2010 called More than just eye candy: Top 10 misconceptions about eye tracking. I guess that was a precursor to the book?
Aga:
That’s what I called the prequel. That was definitely the prequel because that’s how the book started.
Gerry:
In his foreword to the book Steve Krug refers to the semi-magical ability to know what people are looking at. It is kind of magical, isn’t it?
Aga:
It really is and it’s a great foreword by the way, isn’t it?
Gerry:
It is, yeah.
Aga:
It was really Steve who introduced me to Lou Rosenfeld because he read the article that you mentioned in the UXPA magazine, in the UX magazine. So yeah it’s definitely magical to be able to know exactly what people are looking at because we can’t easily observe this when we conduct studies and also we can’t really ask people about it because we’re not fully aware of where we’re paying our attention. So that information wouldn’t be very reliable if we just asked people and also it wouldn’t be quantifiable very easily. So it does feel a little magical.
But unfortunately, I feel like the magical quality of eye tracking is also its curse because often way too much emphasis is placed on the outcomes of eye tracking and it’s just, you know, one of many tools that we have.
Gerry:
Yeah, I really enjoyed the practical and sort of hands-on nature of the book and your hard-nosed approach which I think is evident and your comments there. You said, for example, in the book don’t conduct eye tracking just for the sake of it. You have three questions that you say UX researchers should ask before deciding to use that methodology. Can you run us through those three questions?
Aga:
Absolutely, yes. And I think by now most practitioners already know that eye tracking’s not going to solve all their problems or answer all their questions, although back in the day when we started doing eye tracking in the UX field it felt like a lot of people placed so much emphasis on it. So the three questions help you figure out, do you need eye tracking? Or is eye tracking just not really useful for that particular study?
So I call it my “To track or not to track” decision tree. There are three questions and you need two “yeses” to pass or to know that eye tracking is right for you.
And the first question that’s also the most important question is:
Gerry:
You’d ask that about any particular method, right?
Aga:
But the problem with eye tracking is that people misinterpret this question and I think this is the toughest question because people think the question is about will eye tracking tell me something interesting? Will it tell me something I don’t know? And a lot of times it will. It will tell you where people are looking but that’s not very actionable. But because it’s interesting a lot of practitioners feel like that just makes it okay to do, to conduct eye tracking just because it’s interesting.
But as practitioners we obviously don’t conduct research to learn interesting things, we need to be able to make some decisions based on our results or somebody else has to make decisions based on our results. So Chapter Two, that’s why I put this question, I know it’s an obvious question but it’s a very important question for eye tracking specifically. And Chapter Two of the book goes into detail about the types of actual insight that eye tracking can really provide.
For example, let’s say that we’re testing packaging for a mobile device and let’s say we have a few different versions of the packaging and we’d like to pick a design that will stand out on the shelf among all the competitors. So in the case of this example, eye tracking will help us compare the designs in terms of how quickly they are noticed and how many people will notice them. So the answer to the first question is a yes. Yes, eye tracking will answer or address this particular objective and I will be able to make a decision based on the results.
Now, if it’s a “no” to the question, if the answer’s a “no”, then you should not proceed with the questions. This is it. It’s either actionable and will address your objectives or it’s just interesting and it’s useless in that way so well then you don’t, if it’s a “no” you don’t go on to the next question.
But if it’s a “yes” then you ask yourself the second question and the second question is:
And this really revolves around the fact that adding eye tracking to a study always means extra cost and time in one way or another. It could be costs associated with the equipment or the sessions maybe are longer because we’re calibrating and doing other things, time related to training time…
So, definitely adding eye tracking increases the cost of the study and traditional usability testing methods are cheap and pretty effective so that’s always something to consider especially when the research focus is just pure usability, eye tracking may not always be, or usually isn’t the simplest method to answer your research questions.
Now in our packaging example, when we were picking the packaging that is the most noticeable, eye tracking is actually the simplest and most reliable method to assess the noticeability. So in this case this would be a “yes” for our example. Now we have two “yeses” and we would be able to go ahead and feel like eye tracking is justified for this study.
Now if the second question was a “no” then there’s one more question we can ask ourselves and that is:
And the reason this question… this is kind of almost a dirty question because it’s really no secret that eye tracking studies get more attention in organisations. I feel it even from my own experience. We typically have more observers attending, more clients attending eye tracking studies than studies without eye tracking.
In a way eye tracking has been a marketing tool for usability testing in a lot of organisations and I wouldn’t consider that as a bad use of eye tracking even though I know this type of use is a little bit controversial but as long as the eye tracking also helps address the study objectives, the study objective, which brings us back to the first question so that’s why the first question needs to be always a “yes”.
I don’t know if that makes sense without the graphic.
Gerry:
Yeah it does.
Tell us, is eye tracking better for qualitative or quantitative research?
Aga:
The book really describes eye tracking being used for both types of UX research, qualitative or formative research and quantitative or summative research. But personally I actually find it more useful for summative or quantitative research and here’s why. Formative research usually tends to focus on finding issues with the design so then the design can be improved, so there’s a lot of iteration and if you use traditional usability testing methods and use them well you should be just fine without eye tracking. That’s what I think. So, traditionally, and what I mean by the traditional usability testing methods is observation of user behaviour, think-aloud protocol and probing. That’s still quite powerful.
I feel like eye tracking in the sort of the qualitative formative research is more of nice to have. The moderator can watch the participant’s gaze in real time and perhaps ask more informed questions. Then the people who are in the observation room, they usually also benefit, it’s kind of like being able to see the participant’s face during a usability study. Does it make or break the study? No. But it sort of helps understand what’s going on. So the same thing is done by eye movements in real time. It’s just another layer of information. It’s not critical but it helps, especially with participants who don’t verbalise well. Just imagine sitting in the observation room when the participant is spending a lot of time on a page and you just don’t know what’s going on. But if you see their eye movements at least you know where they are and what they’re doing.
So, again in a formative qualitative study it’s a nice to have.
Now, I use eye tracking for summative research actually much more often than formative research, and that’s, not many practitioners know this but there are over 100 eye tracking measures, a subset of which can be very useful for measuring the user experience. One of such measures is the average fixation duration. I feel it’s kind of an underdog of measures. A lot of practitioners like to look at dwell time, the number of fixations, not a lot of them look at the average fixation duration. That’s why it’s kind of my favourite measure in a way.
I’m not talking here about the total time spent looking at a particular area, but it’s just the duration of single fixation averaged out across multiple fixations and that is a measure of cognitive processing difficulty.
Meaning that longer fixations indicate that more processing is happening and more processing time is needed and usually that’s because information is harder to understand, like difficult words are being used or there’s high information density, and I often use this measure when evaluating designs that contain a lot of information that needs to be comprehended. Think patient package inserts, instructions, informational websites, things like that. And then you can see if the new content is easier to process than the old version, for example.
I just want to add that with quantitative it’s always about comparisons and so if we measure only one design then the results are not meaningful. That’s why I always say give me something to compare it to, version A and B or a competitor’s design, something that we can compare it to and then it becomes meaningful. I can tell if it’s better or worse.
So I feel, back to your question, I really feel like that the quantitative playground for eye tracking is a little bit more interesting and maybe a little larger than the qualitative one. But that’s just my personal opinion.
Gerry:
I suspect a lot of people think of eye tracking and they think well we’ll have people look at this website or this package or whatever it is and we’ll get a bunch of heat maps and from that we’ll kind of know everything we need to know and that’s it.
But as I read the book I was very, I mean I know a little about eye tracking, but I was very impressed with the amount of detail in there and the extent to which you went to the different measures and what they mean, what they may mean, what they don’t mean and how to analyse them and obviously you’re a bit of a stats geek.
Do you think that practitioners need to have a firm grip of statistical analyses before they can successfully get into eye tracking?
Aga:
Well yes and no. So, qualitative analysis definitely doesn’t require any statistical knowledge. Basically what you do is you look at scan paths and interpret them in the context of what the user does or what the user did and said.
Now quantitative analysis, on the other hand, it definitely requires some statistical knowledge. I don’t think it has to be really extensive but it has to be, you need to be able to use basic inferential statistics when making comparisons and I know Jeff Sauro was on one of your podcasts previously. I listened to that one, and he and Jim Lewis wrote a book called Quantifying the user experience. That’s a really great resource for everybody who wants to learn more about statistics. And I feel like that has to happen before you can really do good quantitative analysis with eye tracking.
Gerry:
Aga, practitioners will be used to using a think-aloud protocol during usability testing, during which they would talk to participants and ask them what’s going through their mind. That’s not necessarily suitable though when we’re using eye trackin. Can you tell us a little bit about that?
Aga:
Yeah that’s actually a very good question and obviously participant feedback is an important part of our UX research. So it is important to understand how to properly obtain it in an eye tracking study and there has been actually a lot of debate about whether or not the think-aloud procedure should be used together with eye tracking. And the answer is not as clear-cut as a lot of people might think. The reasoning here is that when somebody’s talking while performing a task that could change not only their behaviour but also the eye tracking data; the way they pay attention to certain things on the page.
So for example, people look longer at the things that they’re talking about. That’s pretty obvious and they normally might not spend that much time looking there. So if you need the participant behaviour, you as a researcher, if you need the participant behaviour to be as realistic as possible, including the participant’s eye tracking or eye movement behaviour, then it’s best to use a retrospective verbal protocol instead of the traditional think-aloud.
And in a retrospective protocol you basically get the participant feedback afterwards, after the task. So it could be not just feedback but also task commentary. You might… there are certain cues that you can use to obtain that task commentary. You can show the participant the video of the task or the video with their gaze replayed overlaid on top of that and then you can get their thoughts and why they did certain things that they did and they could explain it in a little bit more detail.
So as a general rule summative research will call for this retrospective method. The retrospective protocol does have some limitations as you can imagine. Participants might not quite remember what they did and why they did it during the task. It also, I found that it makes test sessions longer and just not as fun and not as engaging for the observers, the stakeholders in the backroom because first they see the task and then the participant let’s say is watching the task again and talking about it. It really extends the session time.
Gerry:
And presumably the cognitive load on the researchers is significantly increased also.
Aga:
Oh absolutely. So, if I can use think-aloud I will, and let me explain when I would use think-aloud. If I’m doing eye tracking as part of a formative study rather than a summative study, so my goal is to find usability issues, not necessarily to measure user experience then I feel like that’s okay to use think-aloud with an eye tracking study and that’s because in that case you would be using eye tracking mostly for observation purposes and you wouldn’t be generalising the eye movement data afterwards.
So in that case the concurrent verbal protocol or the think-aloud protocol makes the sessions shorter, more engaging and I feel like the value for a formative study, there’s a lot more value in think-aloud than in making participants go through the same task again afterwards. So really the decision of which verbal protocol to use, it boils down to why you’re doing the research and how the eye tracking data will be used and I really don’t like these blanket statements. Sometimes people say, you know, “You must use the retrospective method with eye tracking” or things like, “You need 30 participants for an eye tracking study.” I feel like that really prevents people from thinking about their objectives.
Gerry:
Indeed. Let’s not open up that can of worms as to how many participants you need. <
Aga:
[Laughter.] There is a chapter on that in the book.
Gerry:
Yeah, and I was very impressed also by the inclusion in the book of various strategies for dealing with the very issues you’re talking about in regard to the type of protocol to use and the ability to mix and match and so on.
Aga, can I ask you for people who are newcomers to eye tracking and who interested in beginning to use eye tracking, do you have any general advice for them?
Aga:
Well, don’t be too enthusiastic about what eye tracking can do for you but also don’t be too sceptical. I would say keep an open mind.
I’d definitely start by getting the book, and a lot of content in the book was written with novices in mind, or rather UX researchers who are new to eye tracking. So the book does assume a little bit of knowledge about the traditional, I would say, UX research methods. But a lot of the content is suitable for novices to eye tracking. And then I would read the first couple of chapters. They basically explain when and how to use eye tracking and how eye tracking could add value and then if you decide that that’s something that could be helpful in your research then my recommendation is to start small.
So I’d even recommend renting an eye tracker for the first study instead of buying one. They can be a little pricey still. Just see how it works for you and I would use it with a study that has a couple of very well defined objectives, maybe shorter sessions instead of keeping participants there for an hour and collecting all this data, let’s keep the sessions to fifteen/twenty minutes, and that way you won’t be overwhelmed with the amount of data and all the possibilities. So basically limit the scope and see how it works for you and then take it from there.
Gerry:
Okay and we’ll remind listeners that the name of the book is Eye tracking the user experience: A practical guide to research by Aga Bojko and the 20% discount code at Rosenfeld Media is ETUXPOD.
Aga Bojko, thanks so much for talking with me today on the User Experience podcast.
Aga:
Thanks Gerry.
Published: March 2014