Published: 24 May 2019
Designing questions that answer your data needs, and that humans can understand
Gerry Gaffney
This is Gerry Gaffney with the User Experience podcast.
My guest today is a market research methodologist who specializes in survey design and data quality.
Based in Toronto, Canada, she has a PhD in online survey data quality psychology from Toronto, York University. She wrote an excellent short book called People aren't robots: A practical guide to the psychology and technique of questionnaire design.
Dr Annie Pettit, welcome to the user experience podcast.
Annie Pettit
Thank you very much. I'm glad to chat with you.
Gerry
It's a bit of a chilly afternoon there in Toronto I believe.
Annie
Oh it's a lovely minus 10 Celsius and I'm looking out at some beautiful white snow. We just had a fresh snowfall yesterday, so it's ski time.
Gerry
We’re headed for a 36 Celsius here today.
Annie
I am a little bit jealous cause 36 is just a tad warm.
Gerry
So what got you interested in questionnaire design in the first instance?
Annie
It's been something that's intrigued me right from, gosh, as a teenager. So I don't know where that comes from. Um, curiosity about how people chat with each other. I ended up doing, searching out the questionnaire design classes in university, in grad school and it just continued all the way through. I've just strangely always been interested in how people ask questions on questionnaires.
Gerry
Your book is entitled People Aren't Robots. And the phrase occurs several times in the book. Can you tell us what you mean by that?
Annie
Sure. It's actually turned into a mantra of mine. Especially in market research, social research, we ask people all kinds of questions and they're interesting questions, but many times they are not questions people can ask. If we were robots, we might be able to remember the precise SKU, shape, color, size, package that we purchased at the store last week or last month. Robots can do that. Transactional data can do that. I don't know why human researchers expect human responders to be able to answer those kinds of questions. So to me, it really comes back to reminding people that people answering questionnaires are not robots. They don't have that memory. They don't remember intricate details about intricate products. So we need to stop treating people like robots.
Gerry
We do get snowed with the number of questionnaires, don't we. I'm always amazed when I get something in my inbox that says your recent interaction with Bank X and I think, uh, what?
Annie
Yeah, I don't remember when I last went to a bank. So if you're going to tell me I went to a bank, you know better than I do.
Gerry
You have a real focus on the respondents as being the primary focus. Doesn't this contradict the need to meet various business requirements?
Annie
I'm going to say absolutely not. Our business depends on people enjoying the research experience and looking forward to participating in research in the future. If we cannot create an experience that people want to participate in, we are just shooting ourselves in the foot. So we have to think about people, participants first. If we cannot show them that this is interesting, keep it engaging and help them see why it's important to participate in this, we won't have a business in the future, so people must come first.
Gerry
Perhaps we should back up just a little bit and I should ask you, what do you mean by a questionnaire? Did we want to put some sort of definition around this?
Annie
Ha ha. For our purposes, let's keep it sort of broad. It could be that written out piece of paper, 300 questions in a row that takes you a few hours to answer. But it could be anything where there's a researcher trying to gather some sort of written, verbal answer from people.
When we talk to research participants, we need to communicate with them on whatever it is, a focus group, an interview, even in the middle biometric analysis, you know, when there's machines attached and fancy technology in place, these are people in front of us, people just like we are. So we need to talk to them as if, imagine this, that they're just people themselves. We need to use regular, ordinary language that is meaningful, not just to us but also to them. So this whole talk about, this is one of my favorite examples, when did you last consume coffee? Well, I can't say I ever consumed coffee. You know, I drink it. I eat things, I buy things. I don't really purchase things. We need to find regular everyday language and use it in every communication we have with researchers from questionnaires to interviews, wherever it may be in the research process.
Gerry
When you gave that example of the word “consume,” it reminded me of, I guess a lot of people who are amateurs, not meaning that in any pejorative sense, but who are amateurs in questionnaire or a survey design will tend to adopt this amazingly formal language, and construct these obscure sentences and questions, don’t they?
Annie
Yeah, I call it Charles Dickens language. He can take a whole page and create a single sentence with perfect grammar over an entire page. Some of the problem I think is when we're starting out in, in the research space, we don't really have any history on how to write questions ourselves. So we reach into the archives, we grab the templates, we look online, we look in the textbooks. All of these places still are focused on the formal language. They're still in yesteryears where researchers are in these white lab coats with their big superiority complex and the lab rats need to be treated just as things giving data. So we have to put all that aside and basically switch back to, we're speaking with people, let's use real people words.
Gerry
You write about making things shorter and omitting unnecessary questions. This can in fact be quite a struggle in the real world, can’t it?
Annie
Oh yeah, it's tricky. Like when you mentioned eliminating unnecessary questions, a lot of that comes back to demographics and I completely understand why people want to ask demographics every time. You know, you might want to phrase it just a little bit differently than let's say if you are using panel data, they may have asked age or income or education in one way, but you need to know it in a different way. So therefore you would want to ask that question again. But research participants just don't see it in the same way we do. So we really need to think about which cases make sense to not duplicate those questions. And they do happen. We just need to think more carefully about when we can actually do that. And in terms of shorter questions that this comes back to the formality of language. Again, we do this so often. The old style of writing, the Charles Dickens style of writing is just glorious paragraphs and essay forms with commas and semicolons and examples and quotes all inside one single sentence. People don't talk like that. They don't read like that. They don't communicate in that way. So we need to take care to shorten the questions, take the examples out. That can be a separate line. Take the instructions out. Do we, well, first of all, do we need the instructions? People know how question works. They know I choose one here. So you don't always need to say select one answer here. There's so many ways we can shorten questions and actually improve clarity, comprehension.
Gerry
I guess one of my pet hates is those double-barreled answers, you know, something like, I go on Tuesdays because I love Tuesdays. Whereas my answer might be I go on Tuesdays, you know, because I can’t do it on Wednesday or something.
Annie
Yeah. Politics are the worst for this. They're trying to squeeze a certain answer out. So they phrase it in a way that the options included within one question force you to say things, of course I don't want to kill babies. But do you want valid data or do you want your data? Now obviously you want your data. But if you plan out the research well, whatever the result is, if the result is good or bad in terms of, you know, percentages, scores, agreement, whatever, the result is completely separate from your interpretation and actions of the results. You are responsible for the interpretations and the outcomes. So whether the result is good or bad, you can be responsible for helping a client use that good or bad information in a smart, generalizable way. So don't focus on getting the answer you want. Focus on giving data that can be used in an actionable, valid way.
Gerry
Has the European regulation, the GDPR stuff had an impact on the sort of work that you do and the clients that you deal with?
Annie
Much of my work is very North American, US, Canada. So while I hear about GDPR and some of my clients talk about the fact that they are GDPR compliant, I'm actually not seeing any change really. And some of that may be because they were mostly compliant anyways. If you're careful to comply with Esomar, with ISO standards, with the Insights Association standards, for the most part you ought to be fairly compliant with GDPR and any company that is genuinely concerned about privacy, this is a nonissue for them. They've been on that path anyway. Thankfully those are the companies that suit me better. I'm not too keen to work with companies that aren't keen about privacy. So thus far…
Gerry
It’s a really interesting area, isn’t it? I mean we give away so much data by our very online actions, but also when we’re asked questions we do tend to answer them. I'm often surprised at the willingness of people to answer a question just because some organization has put it in front of them.
Annie
Yes, and this is going to get even more troublesome as a social media data, transactional data, Facebook, Google, all of these massive data collectors are more and more able to connect all this data. We just don't realize what's happening. That's why one of my favorite things to say when I'm talking about social media at conferences is go into your Facebook settings right now and check every last setting because the privacy settings change minute to minute, day to day, so you may have corrected them all last month, You're going to need to correct them all again this month. It's not a static situation.
Gerry
It's a bit of an arms race.
Annie
It absolutely is. I'm in the process of deleting pretty much all of my Facebook data and I'm only staying in so that I can access groups for information purposes.
Gerry
That's an interesting… OK, let's not get sidetracked into that one though it’s an interesting topic.
You're a big believer in providing a verbatim question where respondents can enter free-form text. Now I remember being at a talk a couple of years ago, and I won't say who it was, but this person was saying, no, free-form text is bad and you should never have the option for an “other” entry because it makes your data analysis more difficult.
Annie
I'll be completely honest here. That's the stupidest reason I’ve ever heard. Really, if you're not going to do something because it's hard for you, that's just not a good reason. Think about a questionnaire. Someone has spent 20 minutes doing what you've told them to do. They've gotten frustrated with your bad wording, with your inability to include all the answer options that they need to see included. They’re feeling like you missed bits and pieces here, You missed a question over here. You missed answer options. Some of their answers aren't completely accurate. They deserve a place where they can be honest about their opinions if they want to talk more, help you with other ideas, make sure that their opinion is as honest as they can get it. They deserve a place to say that. And along with that, if you're going to put an open end in there, which you should, you should be genuinely interested in those responses. And you should read every single one, act on every single one. You must put a verbatim in there, but you must also care about the answers that people took the time to put in there.
Gerry
Yeah. A particularly annoying practice is having a verbatim but then limiting it to something like 256 characters.
Annie
There's no need for that. If people want to put an essay in there, let them put an essay in there. They deserve the chance to share their full, honest opinion.
Gerry
It's easy to fall into the trap of designing for desktop. I think particularly when working on questionnaires, it, it's so much easier to do the work yourself on a, on a desktop machine. But you know, we kind of forget how pervasive the mobile is. Even when we, even when we design something that we say, no, this has definitely to be done on a desktop, we still find that a huge percentage of the responses come in via mobile.
How did you manage that?
Annie
This is another one of those cases where we researchers need to remember that people are answering the survey. You can tell them to do anything. In the end they're going to do it the way they want to do it. So if their only free time is on the bus, on their way to work, during while they wait in the lunch room, you know, if these are the times that they have available to answer questionnaires, then that's how questionnaires should be set up. In this day and age, every single questionnaire should be answerable, easily answerable on a mobile device. And if it is not, it is not a questionnaire that should be done. You need to rethink what that question areas and find a way to make it workable, easily workable on a mobile device. There's just no alternative anymore.
Gerry
It's very difficult to do that with things like matrices. If you've got a matrix with column headings and a row headings…
Annie
It's not!
How do you do that?Annie
If you want it to be difficult. You can make anything difficult. But there's, there's really no need for a matrix. Any matrix can be converted into individual questions and there is a benefit to that. So instead of having thirty 5-point agreement scales in a row, you can customize every scale for the question. So you could have an agreement scale, a creativity scale, a knowledge scale, a scale for essentially everything. So every single question can be unique. At the same time that does not mean you automatically turn a thirty-item grid into thirty questions. If you've got a thirty-item grid, you need to rethink, is every single one of those questions essential? Are you actually going to act on the outcome from every single question? Or is it interesting to know, nice to know, a duplicate of something else, somebody's pet question. You know, if you know you're not going to do anything about it because you don't have the funds to change the shape or the size, those questions should just come out, focus only on the questions that must be there. Shorten that list of questions and then it's far easier to turn them into single self contained-questions.
Gerry
Matrices, just say no.
Annie
Yes! Totally agree. I am on that page.
Gerry
How do you screen out people who are not within your demographic but who want to take part perhaps for the incentive if there's an incentive involved?
Annie
Sure. I think screening is one of the trickier things. People need to do a lot more homework on that particular technique. For example, one of the simplest things… Let's start with the most common mistake people make for example is, and since I'm holding a donut, a plastic donut in my hand, we'll ask something like, did you eat doughnuts last week? And if they say yes, we screen them in…
Gerry
Sorry, I have to interrupt you here. Why have you got a plastic donut in your hand?
Annie
Um, I don't know. It's a pin cushion of sorts. And I am playing with the pins in it.
Gerry
Fine. Sorry to interrupt you.
Annie
That's okay. It's a cute little donut. I'll tweet a picture of it after. So when you see a question, did you have donuts last week? It is so obvious that the answer should be yes, of course. Sure. I maybe didn't have donuts, but I had muffins or I had English, I had something kind of like a donut. So I'll say yes, it's not really lying. It's close enough. I'll say yes, I had a donut. Alternatively you can say something like, which of these did you have last week, toast, donuts, brownies, buns, a list of six or eight, either similar, very different things, depending on your needs. A list of six things and include in that list a couple things that almost nobody would have used last week.
And a couple of things that basically everybody would have used last week. So bread, probably most people had some bread last week. So what that does is it gives everybody a place to participate. Yes, I had bread. Yes, I had a bun. Yes, I had cereal. Give people a place where they could feel like they've contributed. Then when your donut option is in there, you know that whoever's chosen that isn't choosing it just because they wanted to participate and be a part of it, it's because they actually had donuts last week. So don't make the screening-in criteria screamingly obvious like what we normally do.
That's the main technique. I usually set that up with a few different questions. I do similar kind of thing with age. I don't just ask are you 21 or older. I'll ask, are you 13 to 15, 16 to 18, 19 to 21, give a bunch of age groups so that you can have more clarity in terms of not just how old are people, but so that people feel they have a place that is for them that allows them to be more honest and at the same time allows you to be more specific in your criteria.
Gerry
Can you tell us about detractors?
Annie
It's a similar sort of thing. So in that list of six to eight items, it would be the less common things, brands or categories that take your attention away from your specific brand or category of concern. So the main purpose is to give people a full list of options so that they can't guess what this particular research is about. Detract them from your topic so they, they can be honest. So they're not focused only on meeting the incentive criteria.
Gerry
Okay. On that point too, one of the things that I noticed when reading the book, there was often an underlying assumption, I think, that people were being paid an incentive to participate. When that's not the case are there additional or different steps to be taken?
Annie
The reason I say that is because, you and I, let's look at our salaries, whatever our salary is per hour, whatever that number is, it is not $2. There is no such thing as an hourly rate for adults in the first world countries that is $2 per hour. So when we say we are incenting people with cash or point equivalents that work out $2 an hour, that is not an incentive to me. So I just don't even classify that. I consider anyone participating in this type of research to be volunteering their time because they could go out, get a job at McDonald's and make minimum wage make far more than this. So it comes back to respecting their time. If they're going to spend half hour or more, I hope you don't do that, I know we do that, but if we're going to be doing that, we need to treat them respectfully. And that comes down to simple, easy to understand clear questions. Basically let's not treat people like robots.
Gerry
You're talking about omitting a certain percentage of outliers. What's the process for doing so?
Annie
So that is, probably much of that is my personal style around doing data quality for questionnaires. What I typically do is insert at least five data quality questions and, and by insert them I might just slightly adjust existing questions so that they have data quality characteristics to them. In many cases I make sure I can find at least ten things throughout a survey to do that. So let's say I have ten criteria throughout a survey and someone could hit one or none or ten of those criteria. So if someone hits one, two, three, my perspective on that is there being a human being, I make mistakes all the time. I Make Typos, I forget things, I spill things, I get distracted. I am not a robot too. And this is no different for people participating in research. They make mistakes all over the place. They don't necessarily read everything as carefully as we want. Because that's how people read. So we have to expect every person will make a few errors. One, two, three errors in every research study. On the other hand, if someone is making eight, nine, 10 errors, something is going on. I'm not gonna say they're cheating necessarily. What may be happening is they're rushing around cooking dinner with little kiddies at their feet and trying to get that survey done at the same time because okay, yeah, they want a couple of bucks or the topic is interesting. They want to help out. If they're making that many errors, I'm going to say today was not a good day for them. Let's not use their data from today. Another day, maybe. Maybe they're able to pay attention much better on another day. So let's hold them aside for today. Put their data side for today and wait for another time and see how they do then.
So I use that 5% as, it's probably going to be somewhere around 5% where we see an unusual number of errors happening. If it's more than 5% of people making lots of errors, you probably wrote a bad questionnaire. So it's not their fault, it's yours.
Gerry
When you talk about, you know, the errors or the data quality characteristics, can you give an example of that?
Annie
So that would be people choosing a fake brand or choosing a lot of brands that are extremely rare. It could be somebody choosing all the “don't know” options or straight-lining or speeding or if there's a lot of, a lot of verbatims, there shouldn't be a lot of verbatims, a couple are good, if they're putting gibberish in there, that would count. Everybody puts gibberish and verbatim so on its own, that is not a problem. But when you add it to don't knows, straight-lining and speeding and choosing red herrings. If you add it to all of that, then yes, it is a problem.
Gerry
You warn questionnaire designers not to let their personal feelings transfer to their question wording. What do you mean by that?.
Annie
So one of my… I have favorites. I've started following on Twitter a lot of groups of people who are disabled. So included in that, and I'm going to say this two ways, there are people who are confined to wheelchairs, those poor, poor people who can't get around and they're stuck with wheelchairs. Alternatively, there are people who have much more freedom because they have access to a wheelchair. They can use a wheelchair to get around on the own, do their things without worrying about having to find somebody to take them around. So your personal feelings could be, oh, that poor person. Alternatively, it could be, awesome, wheelchairs create accessibility. So we really need to think about are our feelings impacting how we're phrasing questions. Are we asking in a questionnaire, Are you confined to a wheelchair, or are we asking, Do you use a wheelchair? Those are really tough to see in a questionnaire. It really helps if you either know people who are in that space. And if you don't, test your questionnaire. Test, test, test, always pre-test. And at the same time go on whatever, whether it's Twitter, Facebook, wheatever it is, join the disability groups. Follow people who are disabled. Look for groups of autistic people, people who are visually disabled. Learn, educate yourself so that you're less likely to insert your personal feelings into your research designs.
Gerry
In the future, you know, when we hook up all the machines, do you envisage that questionnaires would be administered by voice enabled AIs (artificial intelligence) and if so what will that mean for people who design questionnaires?
Annie
Oh, that's a fun question because you just asked about thec ompany Questor. They're already doing AI-moderated qualitative surveys. This is happening. And I think it's only gonna happen even more. For many people AI moderation is a form of accessibility. If you are one of those poor, poor people who doesn't have a dishwasher and has your hands in the soap suds, you can answer an AI survey, talk through a survey while you're doing the dishes, while you're fixing the car, while you're baking, washing the kids, gardening, all that AI interviewing technology is really going to open up the world for questionnaires. So any sort of research design that's people talking back and forth, we are there, it's only gonna get bigger.
Gerry
Wow. It's kind of like having a four year old next you saying but Why? Why?
Annie
Yeah, I just love the accessibility of it. Personally. I'm a typer. I'd much rather type than talk out loud. But for so many people let's you know, extroverts for example, they want to talk a lot. So let them, let them participate in the research the way that makes sense for them, the way they prefer to do it.
Gerry
Okay. Now to finish up and go off topic a little bit, but my final question is you used a self publishing platform rather than a traditional publisher for the book. Why did you do that?
Annie
Because I wanted to do it now and I'm impatient. It gives me a lot more flexibility. So if I want to add a chapter, I can just whip in there, add a chapter and it's done in a couple of weeks. This is not a money maker. I'm not doing this as a money maker. It's just basically a fun side project. I get to put all my thoughts in one page, in one book. And if it helps other people then said, cool.
Gerry
It's refreshingly short as well.
Annie
Yeah, my plan was go factual, not theoretical, so people can just skip to the part that they need to learn about and off to the races.
Gerry
I'll remind visitors that Annie’s book is called People aren't robots: A practical guide to the psychology and technique of questionnaire design and I'd certainly recommend it for anyone who's designing questionnaires.
Gerry
Annie Pettit, thanks so much for joining me today on the User Experience podcast.
Annie
You're very welcome. It was a fun chat.