Published: September 2006
Donna talks about different ways to card sort, the relationship between card sorting and affinity diagrams, and why she doesn’t like dendrograms.
Gerry Gaffney:
My guest on User Experience podcast today is Donna (Maurer) Spencer. Now, Donna is based in I think what they call cool climate wine country just outside of Canberra in Australia, is that what they call it Donna?
Donna Spencer:
They certainly do.
Gerry:
You work as a freelance interaction designer and information architect and you’re also very active in the online design community is that right?
Donna:
Yes, I’m a bit over committed at the moment. Apart from regular day to day work I’m program chair of the Information Architecture summit which is in Vegas next year [2007].
Gerry:
You’re also involved in an IA conference of some sort in Sydney in September [2006]. Is that right?
Gerry’s note:
Donna is also one of the organisers of the 2009 UX Australia conference.
Donna:
Yeah…
Gerry:
The reason that I wanted to talk to you today was because you’ve got something of a specialisation in card sorting. Is that fair to say?
Donna:
You know, I don’t know if I have a specialisation in card sorting.
Gerry:
You’re writing a book about it aren’t you?
Donna:
I am, yeah. I specialise in IA and card sorting happens to be one of those things in IA, and I was just the person who kept going “this is a weird thing… why do I have this feeling that this should be really easy but it’s so hard?” So I started thinking about it and because I was blogging I started blogging about it, so people then started asking me questions about it, so I started thinking about it some more. So, yeah, now somehow I’m a card sort specialist.
Gerry:
Can you start by giving us a brief description of what’s meant by card sorting or what you would say card sorting actually is?
Donna:
I’ve had to do this a lot lately because you know, anybody out of the IA community just goes “what?” [Laughter.]
And what I usually tell them is when you’re trying to organise… Like when you’re working on a really big website or an intranet or something like that with lots of content, and you’re trying to figure out how all of that stuff can be organised in a way that the people on the other end of it can find what they need, you get a pile of index cards, write on them, write content ideas on them, and you give it to people who have used the site and say “you organise those in ways that make sense to you”, and they organise them in the ways that makes sense to them and then I as the information architect can look at the way that they group things and that helps me to think about ways that I could then organise things on the site that makes sense to people, not to me.
Gerry:
You’ve talked about getting somebody to do a cart sort, does this mean you get several representative users to do the card sort or just one, or a group, or what’s the actual mechanics of doing it?
Donna:
I would never ever do it with just one, because then it’s just one person’s opinion, and one person’s opinion is no better than mine, I might do it as with three people in a group and involve maybe three or four groups. Or I might get people to do it individually but still involve a number of people…
Gerry:
…I’m just wondering how you decide, and this is something that I certainly don’t know the answer to, how do you decide when to use groups of three people, for example, and when to use individuals? What are the advantages and disadvantages of doing it as a group activity?
Donna:
Oh, a group activity is fabulous because as you get three people to work together, they talk about stuff. So they’ll talk about the groupings and what they see in the cards and that’s often much more valuable than the statistical results you get at the end. Listening to them talk about the content and talk about the concepts is amazing, hearing their conversation is great. I would always involve teams if I can, the times when I don’t is if, you know, just because of mechanical problems I can’t get people together or because I might have some people who I can involve but they don’t know each other and I know that it’s just going to be weird for them working with people they don’t know.
Team card sorts are fabulous, and I use individual ones just when I need to. Or I might do both and use individual ones to give me a sort of a bit more bulk of data.
Gerry:
Is card sorting the same thing as affinity diagramming or what’s the connection between the two techniques or methods?
Donna:
Well, at the core they’re the same because they’re both about getting a large amount of small bits of data and assembling them into something broader. I have in the past used affinity diagramming after a usability test or after I’ve done some user research and I have all these little observations, and what we usually do is [that] the team and I write down all our little observations and we stand around and chat and group our notes into broader groupings and then again talk about why did this happen and what’s going on here and isn’t that interesting.
So, you know, at the core they’re the same because they’re about getting little pieces of stuff and assembling them into bigger. The difference is with a card sort…and they’ve just come out of history in different ways… in a card sort I would do that with external people or people who will be users of the site, in an affinity diagramming session I would tend to use that with my design team.
Gerry:
Right.
Donna:
You know, really they’re pretty much the same.
Gerry:
Card sorting as you’ve described it is… the core of card sorting is used for creating or developing structures, even I guess when you’re doing analysis of data from user needs analysis or usability testing or whatever you are essentially creating a structure of some sort. But what about card sorting for evaluation purposes? Is that a possibility or is that a technique that’s used?
Donna:
Plenty of people do it, and there’s a technique called closed card sorting. In closed card sorting you give people the categories … that you’ve formed or the team’s come up with, and get them to place the cards into the categories. Loads of people do this and I don’t.
Gerry:
Why?
Donna:
I’ll explain it in a sec, I’m still trying to figure out really whether I’m strange for not doing it or whether you know whether it’s something I’m just missing. My perspective on it is people use it and they say I’m trying to validate my structure, I’m trying to validate my categories, and they’re doing that by asking people to slot content into categories.
Now I think that what I really want to do at that point, when I’ve got something in the different categories, is I want to make sure people can find information in them. S I think that slotting information into categories is quite different to finding it, and to approaching them from, you know, I need to find something.
As an example you might come up with a category called publications, and you might give people a bunch of cards for a closed card sort, and they’ll go “yeah that’s a publication, that’s a publication, that’s a publication” and you’ll go “yep, these things all fit together and my categories are validated”. But then if I look at it the other way and I say how about you go and find information on… how much water Australia has left, to give a very bad example…[laughter]… and they approach this thing and there’s this category called publications, well you know, are they going to look there? Or will they look somewhere else?
Gerry:
So, what you’re saying is that by using this closed card sort method… Well if I understand you correctly, you’re running the risk of blinkering the people who are actually participating is that right?
Donna:
I don’t know if I’d say I’m blinkering them. I just think it’s just the wrong… If you think, okay I want to make sure all my content slots in, then that makes sense. Then I wouldn’t get external people to slot my content in either. But if you’re really trying to say “does this thing work so that people can find information?” then you should ask them to find information.
Gerry:
Rather then to put information in pre-existing slots.
Donna:
Yeah, I would just do something else.
Gerry:
Donna, I think that I’ve read or you’ve said in one of your blogs or whatever that you don’t like cluster analysis or dendrograms which I must admit I’ve found quite useful. Can you tell me why you find them problematic or why you think they’re inappropriate?
Donna:
You’re right, I don’t really like them and I don’t really use them, and you know although I’ve been doing card sorting for five or six years I have some background in statistics from university and then from working in the Bureau of Statistics, so I do know the statistics a bit. But for all of that time I’ve put my data into a tool and spat out the dendrogram and I look it at and I go “hmm, I’m not sure what’s going on here.” And it’s really only in the last couple of months when I’ve been writing about how to do analysis that I really started to understand exactly what they’re representing. So for all of that period that I’ve been doing card sorting… if I do a card sort with the team and I listen to what they talk about, and I watch the groups they form, I learn lots of really good things about potential groupings. And I’ll usually put my results from my card sorting into a spreadsheet. And what I do is I sort it in different ways, I explore it to see what type of groupings people have done, and I look at an individual card and see which categories people put it into, and I look at different categories and compare the difference. So I tend to analyse by more exploring the data. So I still just never found that I need to, you know, look at a dendrogram to help me figure out what I’m going to do with that data or to help me in IA.
Gerry:
I guess one of the things you’re doing is you’re getting a very, very rich data set from the fact that you are doing this generally speaking as a team activity… What you’re doing, it seems to me, is a very rich combination of pure card sorting with user exploration, or almost focus group type activities on top of it.
Donna:
Yep, and while I was writing the book I actually set up a card sort in a software tool, I set up in a bunch of software tools and got people to participate for me. So I have just raw results from these people and you know I can look at them and it tells me some interesting things, but I don’t have that richness that I’m used to, and I wouldn’t really like to be doing much of you know that raw data file with no richness. It just doesn’t tell me what I need it to tell me. My clients probably love it.
Gerry:
Yeah, I must admit I find it useful from time to time but I don’t want to get into, sort of, my take on it. Particularly because I want to get your take because that’s what I’m here for. I really like that rich approach that you’ve got because one of the things that I’ve often thought is, you know, if you get individuals to do card sorts you come up with six or eight or ten alternative card sorts and, sure, there’ll be some overlap. You’re almost better off to do it and get an expert to do it and then evaluate it. Whereas the way you’re doing it you are getting that exploration as well, so you’re getting all of that extra value.
Donna:
Really the discussion that people have and it’s not even what they’re talking about but, you know, they talk in concepts, they’ll talk about “oh this group’s about…” I did one with a department and every single person in the group said “oh this is about agriculture” and you know they put these cards together and they labelled them at the end and not one of those people used the word “agriculture.” But I knew very well that the word “agriculture” was going to be completely fine, because they all said it. So how would I get that out of you know, some raw data?
Gerry:
Indeed. Tell me, Donna, do you have any advice for people who haven’t used card sorting before but who want to start doing so now?
Donna:
Well I guess my first bit of advice would be, don’t think it’s going to give you the magic answer. If you think it’s just going to give you the magic answer and, you know, it’s going to give you the answer to the IA, then maybe you should think about other ways of getting the information you need.
Once you’ve come to that conclusion, some advice to get started… It would be good to get started on something reasonably simple. I made the big mistake of trying to do it on a 15,000 page intranet all at once, and I just messed up everything. But maybe narrowing it down to a small part of a site if you are working on something big, or working on something small to give you the understanding of how the process works, and the type of findings you get from it and it actually helps you to do the IA. Because I find that when most people ask me questions about how to do a card sort, or they say “I don’t know how to… what to do with the data”, what they’re actually saying is “I don’t know how to design the Information Architecture.” And it’s just that they don’t know that that’s what they don’t know. So often it really comes back to that, and if you’re not sure how to design the Information Architecture then thinking about that a bit more, doing some reading, going to workshops or something might actually be better then just running a card sort.
Gerry:
When is the book coming out?
Donna:
The book should be coming out in January next year. I need to get back to the keyboard. [Laughter.]
Donna (Maurer) Spencer thank you very much for joining me on the User Experience podcast.
Gerry:
You’re very welcome.
Note:
Donna’s book, published by Rosenfled Media in 2009, is Card Sorting: Designing Usable Categories
Donna’s website is maadmob.com.au
Audio first published: September 2006