Published: 17 February 2021
Getting your act together with your research data!
Gerry Gaffney
This is Gerry Gaffney with user experience podcast.
My guest today is a freelance experienced strategist and researcher. He’s based in Hove on the English South coast, and it’s a place that Wikipedia describes as tranquil and up-market.
He’s a board member of the Research Ops community and is specifically interested in how to manage research data. He has more than 20 years’ experience in strategy design and research.
Mark McElhaw, welcome to the user experience podcast.
Mark McElhaw
Gerry, thank you very much. Pleasure to be here.
Gerry
Now, lately we seem to see the term “ops” appended to everything. Is Research Ops just a way to make user research sound more like a discipline?
Mark
You know it’s a good question. And one I thought I would leverage the community with literally last minute last night.
It’s interesting because if you think about what Research Ops is, it often is really what I see as sort of the backstage side of things. But it’s also at the coalface and at the same time it’s trying to operationalize general activities and scale. That’s the key one.
But it also is about pushing the bounds of disciplines. So it’s not really research and it’s definitely not research management. But yeah, ops is definitely a word that is coming everywhere. However in academia it is actually a field within medicine, which is where I think we, Kate Towsey borrowed it from and in, in the military, it’s definitely a discipline that goes back centuries, I believe.
Gerry
Can I ask what prompted your interest in a more formal analysis of user research practices?
Mark
[Laughs.] Yeah. A good question.
Essentially my desire is to make research more frictionless and having been a freelance Experience consultant, I find myself having to redo everything from scratch typically, or repurpose things that I’ve already got. So consent forms either from scratch or having to make sure they’re up to date or everything like that. It’s, it’s something that some people think is part of the craft. I would rather just focus on the research and basically make it more frictionless. So that’s my, that’s my desire. It’s not because I’m innately organized,
Gerry
I’m also laughing because I noticed the comment you made on our informal script here, something about a “reports graveyard where insights go to die.” So that’s very telling.
Mark
Yeah. That was basically a frustration having been a freelance. So the fact that you write these reports, no one really reads them.This is the research repository side of research. But I guess what I found is there’s a lot of research that gets created and distilled, and it’s essentially Frankenstein research because it’s based on datasets that shouldn’t necessarily be put together. I’ve seen a persona from a large FMCG [fast moving consumer goods] company re when I really dug down deep into it, I found that really it was based on inaccurate data or inappropriate data. And yet I wasn’t allowed to question it because the research costs thousands. So therefore this is that they want, they want the big document, but then they don’t use question it. And that’s where a repository really comes into its own,
Gerry
Which leads to my next question. Although I will mention for listeners who are not familiar with an FMCG is fast moving consumer goods. So what do you mean by a research repository?
Mark
Yeah, very good question. And I’m going to literally lift a definition that we have used in the research repositories project.
A research depository is any platform, system, drive, database, content collaboration tool, library, knowledge-based Wiki, or filing cabinet that stores research data, notes, transcripts images, videos, recordings, findings, insights, reports, metadata, et cetera, to support consumption and reuse by the entire team.
So besides that sounding like a bit of a terms and conditions that no one reads or a catch-all, the thing is that all of us are actually using research repositories. We went purposely wide because people think of it as this new thing, which it is, there’s a whole new area, but yeah, anything from Google drive is a research repository or a filing cabinet, you know, so we wanted to sort of make people realize that you’re actually using research repositories already. You just don’t necessarily know you’re doing that.
Gerry
Yeah. And I guess the end of that definition is the interesting bit –to support, consumption and reuse by the entire team. And it occurs to me, there are you know, the entire organization would be an interesting challenge as well. I mean, we often sees research silos, don’t we?
Mark
Yeah. And this is another thing that came out of a series of interviews. So as I said, I’m part of this research repositories project. And we interviewed, I think 40 plus people. One of the things was actually the distinction between research silos, but also departmental research and how to bring it together. And there are now… One of the things about research repositories is should it actually be a research repository or should it be a knowledge repository for the entire organization. But essentially interoperability is for me the goal because, and I have heard of one very large company. I’m going to … de-identify it as much as possible that literally was saved by, it was an e-commerce one that was saved itself by literally making the data, the research data interoperable, and it was leveraging insights from marketing and sales and brand. And they found there was a particular date in the year that they hadn’t figured on that was critical to their entire business, some sort of seasonal bit that they hadn’t worked on that saved the company.
Gerry
When you to say interoperability, you’re talking about, can you break that down a little bit? What you mean by that?
Mark
Yeah, it does mean many things to many people. I’m talking about the ability to move information between disparate systems, because as much as research repositories would like to be one repository to rule them all, I don’t think that’s going to happen. And I think as the danger a bit like the AOL garden was, if you remember back in 2000, you know, it’s like a walled garden isn’t necessarily the solution. What you need is open data. You need systems that can actually talk to each other. So whilst there are many systems that incorporate various functions like recruitment and note taking and transcription, what you really need is systems that are able to pass information between themselves. That’s what I see as interoperability.
Gerry
You gave a very good presentation a few months ago now at the Design Research meetup in Melbourne. And you mentioned that our repository was for secondary research. I guess, can you first tell us what you mean by primary and secondary research?
Mark
Yeah. it’s something that is more familiar within academia and you can think of essentially secondary research is the posh word for desk research. To cut to the chase. Primary research has generally the research that most of us think we do as design or user researchers, but really it’s about gathering original data. So an original data set. However, things are changing. And I just wrote an article with Dana Chrisfield about this, how the notions of primary and secondary research are falling apart because really you could actually go to a like Jan Chipchase is going to, which is really, it’s all about gathering and sense-making, and in a sense that’s really what research is about. So whether you have the sense-making as primary research, or actually you have it as secondary research, which is what it is, is another story. But yeah, we can discuss that. That’s a very long, long conversation.
Gerry
Yeah, I imagine so. So you see research repositories as being for holding the secondary research. So before we get onto research repositories in a bit more detail, what happens then to the primary research? You know, to be honest, I’m thinking of a particular example of where somebody said to me, six months after the event, when you were doing all that research, that was blah, blah, blah, did anyone use the printer? You know, and I thought, wow. Okay. And I was able, as it happened to go back to the, to my original notes, which I had online, and I’m kind of very obsessive about note-taking. And I was able to go and look for the word “print” and find them, so the research repository would not have that sort of information. It, if it weren’t a focus of the research at the time?
Mark
Well, this is an excellent point in defining what a research repository actually is. So I’m going to answer your question about primary research, but use it to answer what a research repository is and what the various components are. So on the one end, you’ve got what I call a research library, which is where reports are stored, like the finished report is stored. And I think we’re all quite familiar with that. But then the other end of the spectrum, you’ve got all of the distilled data such as an insights repository, which comes from raw data, which also needs to be stored, where that gets stored is debatable. So and this is something that’s been heavily sort of talked about by can’t remember… Shan Carter, Chris Olah… Research Debt and it’s about how we actually store all this.
So a research repository starts off like literally the final report, but actually you can stretch it all the way down to the insights, which if, you know, the, I find it hard to pronounce, the DIKW pyramid, it’s basically where your data goes to information, to knowledge, to wisdom, where we put insights in between I think knowledge and wisdom. It’s all of that stuff I would say is also part of it. And because of GDPR [General Data Protection Regulation] we need to be we need to have access to that because I’ve spoken to various researchers in government who were saying, this is the opportunity, this is where the efficiency comes in, is being able to go and interrogate the raw data again about a different purpose or lifting up something because as you, you will know, just as well as I, when you’re doing research, you find stuff that’s out of scope or people say yeah, off topic and yet it’s gold or it’s really relevant. So yeah. But that’s a hard one to do. And most people really struggle with that still.Gerry
Yeah. It’s also interesting when you talk about you know, re-purposing that earlier data as well. I mean, that kind of all sorts of ethical issues associated with it. How do you deal with those sorts of concerns? I know that’s a pretty broad question.
Mark
It’s a really big area and it’s something that I’m glad to say it’s changing. I mean, so you’ve got consent. So we all know about a consent form and the things that consent forms are now evolving from being this sort of point in time, like a static thing, which is typically some of you signed away your rights on traditionally to something that’s a lot more fluid and a lot more similar, actually the best analogy I’ve got, it’s actually sexual consent because it’s not something that is fixed in a point of time. It’s something that changes and evolves. It’s actually about a relationship. It’s not about a body or a part or a component. And in a sense, data is an embodied component of ourselves. So this is where if you think about the raw data, you need to be able to review the consent or the person that’s being part that’s participated in this needs to have some sovereignty, to be able to revisit and give permission for a different use.
Mark
And a colleague of mine, Brigette Metzler, who basically has an excellent description of, well, you know, you think you don’t mind your face being used for this, but what happens if it… her measure is, well, what if you put it on the back of a bus shelter, which you might think is, well, what’s that to do with it, but it shows how consent really varies. If your face was some of you are the face of some sort of ad you don’t know about, you know, there might be a government campaign. So yeah, it’s a really big emerging field. And funny enough, I just tweeted something about that yesterday, something I finally got around to reading which really good article on consent.
Gerry
Yeah. I mean, it’s interesting when you say you know, letting people revisit their consent. I mean, there are times when, of course they can’t because they’re dead or they’re no longer available or it’s in our, they’ve been de-identified for, for legal or ethical reasons or whatever. So you can end up with a set of data that, you know, is restricted and must remain restricted, even though it’s potentially very rich and would be useful.
Mark
This is another excellent point, because in a sense… So now we’re talking about a deletion and retention policies, which is in knowledge and information management. Another big area that people don’t look at enough. And I know this literally from speaking to various well-known departments in business and academia and government. Less academia. But it’s more to do with, if you have things that are de identified, then it’s, it’s anonymized, right? So therefore that’s a different thing. And you’ve got a shelf life of, you know, it can be up to two years, is typically done. It could be three months after the project is finished. Within government it’s seven years. Actually. I know that from national archives in the UK, it’s 84 years, I think, for personal information. So there are these timeframes. But the, the question is whether or not the information is anonymous or de-identified is the key thing.
Gerry
No, that I think that, that addresses the point without, without solving the difficult problem. Of course, you know, in its entirety, because it’s, it’s bigger than that. To get to something, I guess, a little bit practical. Can you tell us about some of the different types of research, repository applications? I think you’ve actually got a whole list of them that you can, that you can read off, have you?
Mark
Well, yeah. So being a member on the board as a Research Ops community, I can’t really be seen to be a bias towards any particular provider. However, what I thought as a get-around for to that because everyone’s always asking me, is that well, actually, I’ll just mention the names that are actually on our toolbox because those people that have put their names into that. And that’s where you can find a whole list of research ops apps. So I’m just going to list you 18, these are the ones that have been registered. Anyone who else wants to register, please go ahead and do so. So we’ve got:
• Airtable • ARCS • Aurelius • Condens • ConfirmKit • Confluence • Consider.ly • Dedoose • Dovetail • EnjoyHQ • Feedback Loop • Glean.ly • Handrail • Notion • NVivo • Qualdesk • Sticktail • Tetra InsightsIt feels like a brought to you advert, but at least I was impartial.
Gerry
Indeed you were. I had a look at that list of research ops apps that you guys have, and there’s over 300 items. It really is quite comprehensive. There is a short link I’ll read out and obviously it’ll be on the transcript as well. People can access that full list, it really is comprehensive.
I often think, you know, maybe the cynic in me, there’s a, there’s a couple of lines in Henry the fourth part one, I think Glendower says “I can call the spirits from the vasty deep,” and Hotspur says “So can I, and so can any man, but will they come when you do call them?” And it sort of, to me sums up to some extent the issue that we might have, like it’s all very well to store research data, but how can we ensure that the right people have access and, and that they actually refer to the data and a bit of a mea culpa here on a recent role I was in, somebody said, have you referred back to, you know, X, Y and Z research data? And I had like on day one when I was doing all that background reading, but I hadn’t looked at it again since. it’s very easy for this stuff to just fade into the distance, isn’t it, into the past?
Mark
Another really good question because and one that I kind of answered in the article, I just talked about with Dana Chrisfield. I mean, actually it was with a whole bunch of us on the Research Ops team. But we’ve what we did is we looked at primary and secondary research and actually even coined a new term, which is Knowledge Ops versus Research Ops, because, and that’s basically around knowledge and information management. One of the things that really comes through in what you’re talking about is what we’re calling, we’ve defined a stage called licensing. We’ve really called it out because I think this is becoming the key thing. And actually, you know, following on from our conversation around consent, I now see consent as a distribution license so that, you know, if you think about your, in some country, like on a YouTube channel, you don’t have permission towards something. that’s the way that I think of the information on the research repository or the knowledge base, the knowledge hub.
So the licensing is now a key thing and how it’s negotiated and all the setup of that is, is I think is a whole stage. I would also add that in terms of the information being reused, one of the key things is being able to go back and it’s the provenance in a sense you’re talking about, you know, the provenance of the original data sets. Often the raw data, well, first was deleted because of GDPR. That’s typical one. I’m going to be, I’m going to be interested to see how history judges us on that, on the execution of that, because I think historians would find a lot of this stuff fascinating, but that’s, let’s look at that as that’s a whole conversation in its own right? But I think the point is being able to actually talk to the researcher that did it. And I think that’s something we don’t do enough of, because they will tell you a mine of information that isn’t in that final report. That that’s where I think a research repositiory is the bit that could actually help in the future.
Gerry
And how can we make sure that people access this data though? I mean, it’s there, it’s in a repository, you know, a new person comes in or a new team fires up or a new department is created. How do we make sure that that information is exposed to them?
Mark
One of the things I’ve noticed working in various organizations is how the new person comes in. I’ve got no idea what’s going on. And, and they’re tried to be, you know, informed inducted, et cetera, but basically it’s the handover, and this is funny enough One of my pet pet things is think we don’t have a proper handover. We don’t have a proper handover process. We think we do, but yet we’re always left as the new person arriving in going, well, give me the quick, give me a quick brief, and there’s no one around to do it because you always get it in person. You never read the document. It’s never been developed enough because the exits… do you know what I mean? So, yeah, so I would say this is one of the things that a research repository could really be beneficial for, because it would embed that information and be able to let’s say publish a handover document that would be relevant to the level of the person that needs to hear it.
Gerry
One question I’ve often heard is how often should we update our personas? This goes to the currency of research findings. How do you deal with identifying and flagging currency in research data?
Mark
Well, you’ve mentioned the P word which is an all be a trigger word for me.
Gerry
Persona? [Laughter.]
Mark
Well, let’s maybe move on from that. And we can talk about my Mind States framework. You can invite me back for that and we can discuss that maybe.
This came up again with a couple of interviews. Andy Parker was talking about this was actually shelf life, the shelf life of the data, information, knowledge, wisdom, insights, and how we actually find out what information, how do we groom the actual knowledge hub, the repository for that. And it’s, it’s a problem. It’s actually, it’s actually a problem. A lot of people just go anything older than two years, isn’t relevant. I would challenge that with an example of an insight I’ve got from 10 years ago, working at Max Factor, where we were looking at cosmetics and the persona, she was a Diva. And one of the insights that the researchers came up with was every woman has a cosmetics bag full of cosmetics that they never use, but they can’t throw away. And we were thinking, Oh, this would be great to have like a cosmetics graveyard where you can, you can have an amnesty and Max Factor went, no, but that still stands. I wouldn’t, I’d say the only qualifications. It wouldn’t just be only women now that have that, but that’s an insight that’s, that’s continuous. So it’s yet to be determined is the answer, because it’s now being bluntly used with deletion and retention policies. I don’t think that’s… I think that’s a blunt instrument at the moment and we haven’t actually, I haven’t seen a good answer.
Gerry
Okay. Well, let’s I know it’s tricky when we need to can’t expect answers to everything. I’m just looking at the next question, which I think we’ve already, we’ve already covered off a couple of things here, because you’ve been so comprehensive in what you’ve spoken about. One thing that does occur to me, correct me if I’m wrong, but it does seem to me that when you’re talking about research repositories or that these sorts of activities that you’ve spoken about now you have in mind a larger organization, be it private or public enterprise. Is the whole Research Ops field more suited to those sorts of organizations by definition.
Mark
Hmm. This comes up quite a bit. And the reason being is that in the Research Ops community, we often have what we call the researcher of one, the only researcher in the organization. So No is the answer we, we look at. And that’s why that long definition I gave is basically anything that stores information that you can disseminate and share and reuse is a repository. The key though is basically how you can… but at the same, a lot of it is actually about scaling. So within larger organizations, it’s about scaling research because often if you are the only researcher, you can’t do everything and people go and do research. Now the term we use is people who do research, which Kate Towsey mentioned. Kate Twosey also did I should point out the toolbox originally and I need to give props to her for that, for setting that up the 300 whatever apps, but she called them people who do research, PWDRs.
And the thing was that it’s fine for people who do research to do research, as long as they know what they’re doing within parameters which we can help them with and manage, because any research is not actually good research. And that becomes the tension because they’ll get a researcher in, how do you scale it? You know, you get other people to do the research because they’re not around. So it’s a huge area. This is actually probably one of the core functions of Research Ops I’d say is the ability to scale research according to need. That I think is broad enough to cover all of the parameters you mentioned.
Gerry
Yeah. I guess I’m thinking of the lonely user researcher working, you know, whatever it happens to be, and they’re there on their own. And they, they listen to this. And then Mark McElhow said said, you know, Research Ops is a really good thing. And they’re thinking, Oh my God, you know, I don’t do anything like that. What, what should I do? I mean, what steps should that person take to you know, to, to get whatever benefit is available to them from the whole Research Ops discipline.
Mark
Yeah. And I think from doing this project, I’ve seen that the maturity, this is the technical maturity, not the actual maturity of research. Repositories is, is a good, is a good starter, where you you’ll have a, like a sort of a typical drive that’s all unsorted. And then what happens is you start naming it. And so naming convention is your first step, having a good, clear, accepted naming convention. Trust me, that is not as easy as you think, because it changes. And then it becomes impossible to find. Then the next one is you then have an index of where all my assets, which is a bit similar to an asset registers, like what have I got? And you might then also put dates on it. That could be really handy. Funny enough, there are lots of very large organizations that have had their dates wiped because they were pre you know, different platforms. It’s quite shocking actually.
So you then start putting metadata around that. Then after you’ve got that, make sure you are keeping your research tidy. I’m talking to myself, I feel like I’m looking in a mirror telling myself this, I have to remind myself is always clean up after yourself. Otherwise you’re creating research debt later on, which you’ll never have time to do. You’ll never have time to do it. So it’ll just sit somewhere until someone panics and you just have to delete it, just burn it, you know, so try and incorporate it in your work as opposed to doing it at the end of the project. Then if you can, you know, maybe get a research repository tool.
Tag! That was the other thing, critical thing. Tag your work, start with a folksonomy, which is a nice term which is basically just tagging it with any words you want to use. Evolve that into a structured vocabulary, which then becomes a taxonomy. Don’t get a taxonomy that just becomes vast which is what we’ve just done, mainly my fault, because I’m trying to solve the whole thing. And I’ve created this huge behemoth. I can send you a link to a beautiful Kumu visualization, but don’t do that. [Laughs.] Keep it really simple, very specific to make sure you’ve got the indexing you’ve got, the tagging. That’s when can start when you start considering a research is platform because they can help you with that. And I, I think that was it off the top of my head, yeah? Those would do, I’d say, yeah.
Gerry
That’s a good solid starting point. I’ve got a question completely without notice here. A few years ago I was at Service Design UK. I think it was Service Design UK in London and a couple of presenters, I can’t remember the names for the life of me. I didn’t have a chance to try and look it up before we spoke. They spoke about the authorial voice and you know, how we pretend in the research community that when we, when we present something that, you know, it’s this impartial view that’s emerged from the data and, you know, I, or you as the author has not had anything to do with it. And they suggested, and I was really taken by this concept that you should in fact, go back to the people who were the subjects of your research and get them to validate and verify what you’ve just found. Any thoughts… I know it’s completely out of, you know, not related exactly to what we’re supposed to talk about, but…
Mark
Oh, I’m delighted. This is the kind of challenge I like. The reason why is it’s exactly the type of privilege I used to occupy and that we all do as researchers… or in danger of doing where we become the authorial voice. And you know, the participants are there sort of glean insights from that we then re-present kind of as our own. And I think that, that, there’s a lot of ego in that. And I think we need to check that. And there’s a whole field now. I can’t remember the exact terminology, but it’s all about decolonizing research.
And I really liked the idea of research moving into a collaborative adventure with people that you’re researching with. So it’s more like participatory research. So I think instead of, I would say instead of going back and validating, I would have them on board and available. And so they you’re, you’re there with you, however, yeah, I don’t have the answer fully, but that’s where we need to be heading. And as maybe more of a, sort of you could be the documenter or the scribe, but you are not the creator of this. There’s an ownership quality that goes back to consent. And that’s where this article is that said, I just tweeted about as great about that, because it talks about ownership and contractual stuff that I think this is also being alluded to.
Gerry
Great. One of the things you mentioned you, you spoke about the GDPR a couple of times, which is the EU’s general data protection regulation. And I was wondering whether you notice concerns about the same thing from outside the EU, or is the EU particularly well advanced in its thinking on managing people’s data?
Mark
Well, you know, as a block they’d been quite pioneering, you know, but remember California has also got their privacy regulations as well. It’s really quite advanced and very… they’ve really specified with a lot of granularity the type of privacy. It is something that is occurring and at the same time is not occurring, is going in the opposite way. And I think that lockdown and COVID and pandemic is a wonderful way in to the opposite of that. When we’re looking at contact tracing the way that’s happening within India, for example, in China and various places, your rights have just gone, and you can say, well, you know, when a national emergency comes, you know, we haven’t got time to really go through the consent and that’s a fair point, but I think that it’s really, this was really highlighted. It’s a wonderful double-edged discussion.
Gerry
I mentioned that listeners can find the Research Ops group at researchops.community. Is that correct? And you can find in there a link to a Slack channel, a couple of the articles that you’ve mentioned, Mark will be pointed to from within that general experience as well, I think.
Mark
You can find us on Medium and on Twitter, as well. As @teamreops on Twitter and the Research Ops community on Medium has got a whole swathe of articles. Again, if you want to start, go and read the Eight Pillars which is by Emma Bolton. So that’s a good intro that gives you a whole sort of state of the area in a sense.
Gerry
We will include a couple of those links on the transcript anyway, and as usual, a point of that, you can get a transcript of this episode at uxpod.com
Mark McElhaw. Thank you very much for joining me today on the User Experience podcast,
Mark
Gerry, it’s been a pleasure.