Published: November 2012
Chris Noessel makes space explosion noises, and talks about what we can learn from studying user interfaces in sci-fi movies.
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast. One of the pleasures of working in user experience is it’s quite acceptable to be a sci-fi geek and my guest today is certainly one of those.
He’s a director at Cooper and practice lead for interaction design. He’s worked for Microsoft’s Futures prototyping group in Seattle and at marchFirst he conducted research and design for websites including Apple, Sega and Harman Kardon.
He was one of the founding graduates of the Interactive Design Institute Ivrea in Italy. But the reason he’s here today is to talk geek stuff as he’s the co-author with Nathan Shedroff of Make It So; Interaction Design Lessons from Science Fiction.
Chris Noessel, welcome to the User Experience podcast.
Chris Noessel:
Thanks Gerry.
Gerry:
I imagine that very many of our listeners would be familiar with the expression “Make it so”, which is used by Captain Jean-Luc Picard of the Starship Enterprise. Did you go through many book titles before you got to this one?
Chris:
Actually we did not. When Nathan approached me with the idea in mind he kind of had that title picked out. When he originally had the idea for the book he had been watching some show, and he doesn’t remember which one, even though we’ve been working on this and trying to stoke his memory for the past several years, he can’t remember which sci-fi show it was. But he had seen a couple of interfaces and he thought; “Wow those are things like in the real world”, and then a short while later he saw this Motorola StarTAC phone come out and between the name and the fact that it operates very much like a communicator, or did, he recognised there was some connection.
It was a very short while after that that he made the connection that “Make it so” was a really great title. So, even though the idea sort of bubbled in the back of his head for a long time, it was deeply tied into that title and when he approached me with it he was like; “Okay, so here’s the idea. It’s called ‘Make it So’”, and I was sold both with the idea but with that title being so apropos.
It was a shoo-in.
Gerry:
It certainly is apropos and it’s nicely nuanced as well. Tell me, though, are you guys serious in suggesting that we can learn about interaction design from sitting back and watching sci-fi?
Chris:
I think dead serious. But it takes a certain understanding about how to use science fiction. And part of the reason we’re so confident in making this recommendation to people is that we do it.
I know that we’re in an unusual position having written a book but any time I watch a sci-fi show now, half of my attention is on the storyline of course and the other half is taking a look for the interfaces and what sort of new things are there.
I think one of the things that we don’t have as a practice is a lingua franca about the types of experiences, especially future experiences that we are trying to design.
It’s really easy to reference something in the real world but if it’s in the real world how forward thinking are you really being with your design? If, however, you’re trying to be forward looking it’s really useful to have an example on hand that you can sort of say; “Hey it’s kind of like that interface from Minority Report”, even though if you read the book you’ll understand why I would be cautious about recommending the exact interface from “Minority Report”.
Gerry:
Indeed. Now, one of the things that I must admit I was a bit disappointed with initially, well actually the only thing I was disappointed with because I did enjoy the book very much, but I was disappointed to see that you’d excluded a whole bunch of stuff.
I was looking forward to seeing some discussion, for example, of many of the devices that Philip K. Dick had created in his novels but you excluded books. Just can you tell me the process that you went through, because once I read about it it made sense but I’d like you to tell our listeners why it was that you excluded so many things and what in fact you did exclude?
Chris:
Sure. Knowing that we were going to look at interfaces in sort of the science fiction survey that we were recreating it naturally led to some filters, not of all which pleased us, because we are science fiction fans ourselves.
The first one is we really have to see it; the first example of sci-fi in the entire book is actually a text example from H. G. Wells’ “he Time Machine” and we go through a quote about the time machine as the time traveller is describing it to gentlemen in the room.
And even with that description, you can kind of understand that it’s inadequate to be able to examine that interface. The reader’s mind is just making up too many of the details. So for that reason we had to say, ok, text interface is out because we’d be not critiquing the interface as much as critiquing our own interpretation of the words that describe the interface.
The second thing is that we really needed it to be used over time because of course interfaces are about interactivity, interactivity is about input and output and the system state. And so a lot of things that might have been influential to us, even just like beautiful illustrations of technology or product design, there are tons of examples of that, those had to go by the wayside.
Then the next layer was that we needed them to be consistent from depiction to depiction and that meant that there were two categories that excluded. The first category was a favourite of mine which is graphic novels and it’s because given a different artist or a piece of technology that has evolved over the course of decades in a given story, like the “Fantastic Four”, for instance, the different artists will have drawn it differently.
And even within a single graphic novel there may be some details that change and that would have just taken us too long to say; “Well which version of this technology are we really evaluating?” And for the same reason we decided not to go with hand drawn animation in the survey, just because, again, even though a lot of film makers do really great at keeping those things consistent, we couldn’t rely on it being consistent.
Contrast that with 3D animation where the filmmakers actually have to sort of get in there and work to change the interface from just depiction to depiction. It just makes it much more likely that we’re going to be looking at evaluating the same thing.
So, that was all in. … Even given those giant chunks we still had other things that had to go out, such as the fact that Nathan and I were looking at science fiction, and it’s a gigantic genre, so we also had to cut down for popularity. Originally we were taking a look primarily at influential films and slowly took a look at some of the more obscure films to rough out some of the topics that we had found.
And then lastly, we were limited by the size of the book. We actually have about one third of the material that we had sort of in early development that we had to sort of just get rid of. The category of things, so we had a chapter on doors, a chapter on space ships, a chapter on weapons, a chapter on flying interfaces for spacecraft. And all of that had to go.
So, for every single layer of thing we had to cut out breaks our heart too.
But even given that, this is the one of the largest books that Rosenfeld Media has ever published so we’re actually pretty glad to have gotten to what we did. And it only just sets the stage for future work on our part.
Gerry:
If you could publish an unexpurgated version.
Chris:
[Laughs.] Well we’re kind of slowly building on that on the website that we’ve got going. Right now we’re just sort of publishing our database very slowly. We’ve just completed today actually the review of “The Cabin in the Woods” but I think in the future we’re actually going to build up enough examples there that we can publish some of the things, content, like spaceships, like weapons etc.
Gerry:
How many movies did you actually include in the database for the book?
Chris:
Oh, that’s a good question and the reason is that the number I have in my head is the number of images in the database and that’s over 10,000 and that was actually before we added the last two properties.
So I would have to go and check for the number of movies. I think it’s on the order of, and this is really weird because how you count sci-fi is a tricky question in and of itself. Do we count “Firefly” as one property, or do we count it as the fourteen episodes? Or do we include the movie “Serenity” in that? It gets kind of complicated.
If I had to take a stab at it, I think it’s on the order of three dozen movies and probably on the order of twice the number of that on TV shows. If you want real hard numbers, I’ll have to go look.
Gerry:
No, no I just wanted to get an idea. But it’s certainly a considerable undertaking and I was very impressed as I was going through the book about the methodology that you used.
Tell us a bit about the creation of this database and how you mined it for the information that you present in the book.
Chris:
Well, I think, we wanted to approach this rigorously and to do that we knew that we couldn’t really keep all the sci-fi in our heads. It’s just too much.
Gerry:
Yeah that really struck me because when I got the book I thought, oh this is just a fun topic and these guys are having a bit of fun, and when you get into it you realise how serious you were about your approach.
Chris:
Maybe a little too [serious] because like the more serious you are about sci-fi, the more nerdly you are. But we accept that, I think. So, the serious tack we took, the first thing was actually just gathering examples and it was a pretty complicated process.
The first thing we did is we spent probably about three of the six years just building up examples and that involved getting DVDs of movies and television shows, watching them on our computers and then pausing when we saw interfaces and making screen caps.
And it’s not just one screen cap of a given interface but of course over time, so we created multiple screen caps in order to sort of describe the entire use of the interface and that would include everything from the very mundane stuff like video intercoms in “Gattaca” to the more unusual things like the goofy controls for firing the Death Star, for instance.
And then we sort of built up good descriptions around each of those examples, both from a here is what happens in that scene. But we also sort of reviewed entire movies or entire television shows for categories.
So, what are the communication technologies that occur across “Star Wars”, for instance? We also went through each example and tagged them with a number of different attributes. That included things like what are the physical motions that are involved, what is the user goal of using this piece of interface? And then just kept adding to that custom database over time.
Another embarrassing thing is we will never reveal that working database because it was me coding and I am so not a coder and it’s done in like ColdFusion on a Microsoft Access database. Anyway, any developer who would see that would promptly make sure I never got work doing it but it certainly served our purposes.
So over the course of three years we built up this giant database and at the end of it, even though we sort of kept trickling in examples, we began to really look at it and analyse it and the first thing we did was just take a look at the tag cloud; what are the most tags that we see in the database? And it turns out that was that one of the funniest lessons from the book was that over and above all else, sci-fi glows.
It kind of makes sense, right? Like technology is an embodiment of power and a lot of our real technology does glow, whether it’s just a tiny little blinking LED or of course the glowing screens that we spend most of our days in front of.
But it also included things like we got a sense of the colours that were primarily used in sci-fi. We got use of what are the main goals in sci-fi and even the categories and those categories are sort of driving our deeper investigations and most of the chapters that you’ll see in the book come from that tag cloud.
The first ones that are about UI components like volumetric projection or physical controls are from that tag cloud and the last four chapters are also from that but they’re more categories of human activities like communication and learning.
Gerry:
The section on colours I found particularly interesting because you had one page that was just a colour map, like a heat map of the various colours used in the UIs over a number of years and you extracted what were the dominant colours at any given time and I think “Terminator” caused a red blip in there at some point.
Chris:
Yeah. It was a really fun thing to have done but it was a nut that you could crack a number of different ways and I suspect that you could have approached it from another direction and got kind the same results.
One of the real tricky things was we were counting each interface in a property once but that doesn’t account for screen time. So if you took a look at the amount of red that you saw in “Terminator”, how does that compare to an interface that you only saw once in “Star Wars” or “Star Trek”?
Well, they actually have the same weight in the diagram that we did but as you can see that chart is overwhelmingly blue. So, we sort of think that if anyone else ever does those other colour studies across time, which will take much more computation power than we had access to, we trust that they’ll find the same thing. But of course we’re trying to be objective about it and if somebody finds something different that would be a fascinating thing to see.
Gerry:
I was initially a little bit sceptical about the way that you extracted lessons as you went along in the book but I was won over fairly quickly.
So, how did you extract specific lessons?
Chris:
It’s actually a tough one because lessons are essentially abstractions and you can abstract, I haven’t counted, but let’s say on the order of a dozen different layers, right? Are you paying attention to the physical interface, are you paying attention to the physical motion of the body of the person using it or the alien using it?
Are you talking about the representation that they see of them in the system, like the cursor? It just goes on and on.
Largely the lessons that we derive came from four categories; there was the most sort of personal or artistic ones were just the ones where we had questions and we wanted to go in and answer those things for ourselves.
Another category were things where we just noticed that there were similarities. One question that I had that I actually never answered over the course of the book, so you probably won’t see it in there, is why is it that the red dot that is KITT’s eye in “Night Rider” the same as the red dot in the Cylons?
Now, those properties were sort of about the same time so it may have been just like oh that was around the prop department but of course we want to look at it and ask those questions from an interaction design standpoint.
What was it about that sort of scanning red dot that made them viable faces for those pieces of technology? The tag cloud also sort of gave us that sort of top down thing. It gave us that priority for, hey here are the things we need to look at and so when you look at the cloud itself you see, oh hey, ‘medical’ is one of the big tags that ended up in the database a lot.
So we just tried to find lessons that fit under that category and part of it of course is our experience. I do a lot of medical stuff at Cooper and so I might have glossed over some lessons that somebody brand new to the medical domain might have found useful. But we think we captured that in a really… attention to the structure to the chapters.
So when we wrote up the medical chapter, for instance, it’s actually sub-categorised according to the thing that the physician is doing at that moment, starting with just basic capture of the patient data or doing tests in order to understand what’s going on inside the body or making a hypothesis about what is wrong.
The last category is actually my favourite and this is probably like “Inception” – five levels of nerd down. But it’s apologetics. So apologetics is a term that we stole from theology and in theology it describes a theologian’s attempts to resolve what appears to be broken.
So, the old saw of how can God be infinitely powerful, infinitely knowledgeable and good? An apologist would then go in and explain well, no the systems not really broken, here is what the answer is. And we ended up having to do some of the same thing in the book.
We would look at an interface that on the surface looked really, really broken. We were like, oh that’s just some bad art director’s idea of how that thing should work. But then we would pause and go, no wait a minute. There is something here that works for the movie, and certainly works for the fictional characters.
Is there something that we can pull out that would work for real-world users? And often times by taking that deliberate apologist approach to a particular interface we would go, well wait a minute, that’s exactly how it should work and it often drove the most surprising insights and lessons for the book.
Gerry:
One of the interfaces you’ve already mentioned was the 3D interface in “Minority Report” but it often seems that UIs in fiction are very impractical and that one, for example, holding one’s arm up in the air would be pretty impractical and very tiring very quickly.
Do many filmmakers pay attention to human factors like this or is it just all done for show?
Chris:
It depends of course. [Laughter.] We talked to a couple of art directors over the course of the book. We also did a, this is another thing that had to get cut that I didn’t even mention earlier, but we talked to not only art directors but interface designers for science fiction and even real world makers like folks at NASA who had been deeply inspired by sci-fi and we just didn’t have the room.
So, we had to get that cut out. But the direct answer to your question is I think they have a vision that they want to impart and often times will hide a bunch of stuff that they don’t think is important. In the case of “Minority Report”, and this is something that John Underkoffler, the inventor of that system, and he is now working at Oblong, sort of acknowledges but that they’ve tried to work with since the movie is that you can’t keep your hands above your heart that long.
Composers have, if I remember the factoid right, their hearts are something like 150 per cent the size of a normal person’s because they’re constantly keeping their hands above their heart and most people just aren’t that fit without a lot of training.
And they hid that deliberately from the film. They sort of put it on pause, asked Tom Cruise, or Tom Cruise actually asked, “Hey can we have a break here? Okay, I’m rested up. Let’s start again.” And I think that when they have a vision, especially when it has an importance to the plot, they’re willing to hide a lot of those flaws by clever editing or doing some sort of smoke and mirrors stuff.
Gerry:
What’s the best UI in the sci-fi that you reviewed?
Chris:
[Laughs.] I kind of think that depends on what counts as ‘best’. You could certainly say what is most likely to have helped the character accomplish what their goal is. There is the ‘best’ as in the thing that I found the most surprising lesson from. There’s even ‘best’ as far as coolest combination of technologies.
I’ll answer the last two. One that was sort of a surprise to us was the French film called “Chrysalis”, partially because we didn’t venture outside of English sci-fi enough, as much as we certainly wanted to, or I guess should have. But in this film there is an interface for telesurgery where a world famous surgeon who lives in Paris is working on the most difficult cardiac surgeries in the world but she can’t travel to all of them so she has an interface where she puts on these gloves and she raises her hand and uses a gestural interface that is combined with a volumetric projection in order to see this holographic image of the patient in order to control the surgical tools at the far end and in order to make very… to scale up the heart that she’s working on so that her gross motor movements end up being very precise on the other end.
It’s just like, it’s smart technology, it’s pretty brilliant use of volumetric projection even though there’s a bug in there that I invite the listeners to try and find, and it’s really inspirational as well. It makes both sense about not flying people around to where they’re jet lagged and doing life-critical situations and smart.
So that was one of the best, most surprising.
…So that’s the one that folks may not have known, one that they already probably know that we pulled this great lesson from was “Star Wars”. So you’ll remember the scene where Hans Solo and Luke Skywalker go into the gunner seats at the Millennium Falcon and there are a bunch of TIE fighters, they don’t know from where but who are beginning to attack the ship.
So they get into these semi-ridiculous chairs that swivel about and we’re going to bypass the fact that it’s adding the strange centrifugal forces to something to you know what should be precision motion, that’s okay, we’re going to bypass that. We’re even going to bypass that little goofy orange screen they have in front of them that in no way helps the gunner sort of imagines what’s happening around them in 3D.
But instead we focused on the fact that as the TIE fighter sort of passes the window, and you can see Luke operating the guns and you hear the lasers go [makes laser noises] and you hear the TIE fighter when he gets hit sort of explode [makes explosion noise] and Yay! the day is saved and he can turn and target the next TIE fighter.
Gerry:
I think you’ve missed your calling.
Chris:
[Laughs.] Oh, as a sound effects guy?
Gerry:
That’s right, yeah.
Chris:
Well you’re watching the film, I don’t know that most people have this; I certainly didn’t when I was twelve and seeing the film for the first time but where the heck is the sound coming from? It’s in space, there’s no air to propagate any of that audio back and so at first blush you may think well, that thing is broken until you realise that the sound works for giving the audience a sense of the battle and if it works for us there’s probably something there that will work for Luke in the gunner’s seat as well.
So if the sound wasn’t there, you would want the Millennium Falcon to use whatever sensors it’s got to provide that sound for him so it can detect that there’s a TIE fighter over his left shoulder and if he’s got a Dolby-esque array of speakers it should play something like a spaceship sound that’s appropriate and signals to him the size of the ship, the speed at which it’s coming, the direction at which it’s approaching all through that sort of passive, oh what’s Marshall McLuhan’s term for that? I can’t remember it – that passive medium of audio and give him the opportunity to know, give him a field awareness of what’s happening in the world around him.
It was only studying that, going; “Hey wait a minute, where’s the sound?” before we used apologetics to get to audio augmentation is something that we’ll most likely have to happen if we ever have space combat like that.
Gerry:
Hopefully we won’t.
Chris:
Hopefully we won’t.
Gerry:
Do you think that a consideration of sci-fi should be required for UX people?
Chris:
Hmm that’s a fine question… I don’t think required, only because that sort of raises my authoritarian hackles. [Laughter.] I think that certainly sci-fi is a) a lingua franca; b) it is a very engaging way for students of science fiction to understand the issues.
I can imagine asking students to watch a piece of sci-fi and to do this kind of evaluation. Hey, talk about what the user’s goal is, talk about what the user’s capabilities are, talk about the things that they see to tell them the system state and the controls they have in order to modify that state to achieve those goals.
I think it would actually be, and we haven’t written that, written this material for teachers, but I think it would be a great educational tool. I also think it’s going to be really exciting for students to go, hey wait a minute I thought we were studying something as droll as interface design but wow, we’re actually going to be watching “Star Wars” and “Chrysalis” over the course of this term.
I think you could engage a lot more students that way.
Gerry:
I’ll remind listeners of the name of the book which is “Make It So; Interaction Design lessons from Science Fiction” and I recommend it, it’s very enjoyable and challenging and interesting.
You mention there’s a website associated with it; what’s the name of that website?
Chris:
It’s scifiinterfaces.wordpress.com. As you can imagine, with the phrase ‘make it so’ for being in the popular awareness for so long that URL was long, long gone.
Gerry:
Chris, I believe you’re coming to our neck of the woods, you’re going to run some training for Cooper U in Sydney in December, is that right?
Chris:
Yeah that’s very true. We got enough interest from both folks in New Zealand and Australia in the Interaction Design practicum that we teach that we were able to generate interest. We have I think 16 students now and if your listeners are interested then we will open the class to up to 25 students.
Gerry:
You’re also coming to visit Melbourne where I’m based to visit the UX Book Club I believe.
Chris:
Yeah my first week will be teaching there in Sydney and it’ll be my first time in Australia so I’m being flown to Auckland to speak that first weekend but then my second week is in Melbourne and then my third week my partner joins me and we’re going to drive around and explore the country, even the outback, which is a little scary.
Gerry:
You know we drive on the wrong side of the road here, right?
Chris:
Yes. [Laughs.] I’ve been trained once before, jet-lagged, having to learn that in New Zealand so I think I can manage it again, especially after a week in Sydney I’ll have adjusted so able to get behind the wheel safely.
Gerry:
No doubt. Well, Chris Noessel thanks for joining me today on the User Experience podcast.
Chris:
Thank you very much.
Published: November 2012