Published: November 2009
Shane talks about the principle of the super-real, and about bridging the virtual and physical worlds.
Gerry Gaffney:
This is Gerry Gaffney with the User Experience podcast. My guest today has a long history in usability and user-centred design, going back to 1991.
He has a Master’s degree in cognitive science from the University of New South Wales. He was principal consultant and later general manager at the Hiser Group which is an early and very influential consultancy based in Melbourne, Australia.
Later he freelanced for a variety of organisations, including my own company, Information & Design. For the last few years he’s moved on to bigger and better things as a User Experience Evangelist at Microsoft. Shane Morris, welcome to the User Experience podcast.
Shane Morris:
Thanks Gerry.
Gerry:
The obvious question, Shane: What is a user experience evangelist and how did you get to be one?
Shane:
[Laughs.] Evangelists at Microsoft… Our job is really to get the message out about upcoming and the most recent technology. Basically preparing the marketplace for what’s coming. And by the marketplace generally for me that means talking to designers – user interface designers or designers generally about what Microsoft is up to in the user experience space (Silverlight, Expression, Surface, all that sort of stuff), or talking to developers, a more traditional Microsoft audience, about user experience, about user centred design and why you might want it and how to go about it, as well as talking about what Microsoft is doing to help.
Gerry:
A question without notice: Have you had a play with Microsoft’s Courier, the new tablet, yet?
Shane:
[Laughter.] I saw the video yesterday. It was the first I’d ever seen of it. I’ve already sent my credit card details in and…
Gerry:
You’re anxiously awaiting delivery.
Shane:
That’s right; I’m waiting by the doorstop. Any moment now.
Gerry:
Because I believe that the operating system is compatible with Surface, is that right, or is it the same operating system, or do you know anything about this?
Shane:
Seriously, you know as much about it as I do. We’ve both watched the same video.
Gerry:
OK. I didn’t actually see the video…
At the UX Australia conference in Canberra recently you gave a great presentation on the use of Microsoft Surface as a proof of concept application at the Lonely Plant store at Sydney airport. I guess a lot of people would be familiar with Lonely Planet, the guide books and so on, but they also have an active online presence… Do they have any stores worldwide or was this the first store, the concept store?
Shane:
The Sydney store is the first Lonely Planet store worldwide, opened in July [2009]. What we worked with Lonely Planet on, and also with Amnesia Razorfish, is basically a proof of concept for how technologies like Surface might work in a retail environment. Lonely Planet, as you know, is not just a physical book publisher, they have a large online business. And a large part of what we were exploring is how we could connect the physical environment of the books in the store with the virtual, online environment.
So we’ve worked on this proof of concept from start of this year through to when we announced it in June at our Remix conference.
Gerry:
The store presumably is in the international terminal?
Shane:
Yes, the store’s in the international terminal. You can go there and buy your Lonely Planet guide books and all sorts of travel accessories today. It’s very much a sort of flagship store for Lonely Planet. And I think Lonely Planet prides themselves on adopting new technology and being at the forefront of new trends, and really our proof of concept project with them was about exploring these sorts of future directions.
Gerry:
Can you start by telling us a little bit about what Microsoft Surface actually is?
Shane:
For those who haven’t seen it, it’s basically a computer in a sort of coffee table format, we’re looking at one right now. It’s 30-inch or 32-inch, I can’t remember right now [Gerry's note: it's 30] screen that’s set horizontally into a table-top.
Gerry:
It looks to me as much like a Space Invaders game as anything else.
Shane:
Yes. Short of the 20-cent piece slot on the side, and the joysticks, it looks like a Galaga machine or a Space Invaders machine. It’s deliberately built into this horizontal sort of coffee table format because it’s very much about exploring ideas of social computing, by which I mean computing done by more than one person in collaboration, rather than a traditional one person in front of a screen and keyboard type model.
Gerry:
And it’s a multi-touch surface so one doesn’t use… It’s not by any means a standard Windows operating system that it’s running.
Shane:
There’s no keyboard, there’s no mouse there’s just a multi-touch screen at the top which is also able to recognise objects. So when you place objects or these pre-defined tags on top of the table it can recognise what those objects are. …It’s not a conventional PC in terms of form factor but interestingly enough it’s actually running a standard version of Windows Vista with an extra layer of smarts on top.
Gerry:
You said Lonely Planet is interested in being at the forefront of things, but what specifically attracted them to Microsoft Surface?
Shane:
We met with Lonely Planet and with the developers of the store itself, a company called Lagardère late last year and the agenda was, we were interested in what new technologies were coming up, so that we can plan going forward, including for the store. And we showed them a bunch of technologies including Surface.
And as I recall, the moment that really piqued their interest was when we showed a little application that was built by Amnesia Razorfish in Sydney. All the Amnesia staff, their business cards have these Surface-readable tags on them. So when you put an Amnesia staff-member’s business card on the table, it pops out their Twitter feed and their Flickr feed and stuff like that. And when they saw that application they saw, I guess it came down to they saw an opportunity to bridge they physical and the virtual, as a way to link the static content in a Lonely Planet guide book to the dynamic up-to-date content that’s online on the Lonely Planet website.
One of the guiding catchphrases of the entire project, that we kept repeating, was this bridge between the physical and the virtual. I think that’s what they saw in Surface.
Gerry:
So tell us how the proof-of-concept was typically used in store. I would wander in as a customer and do what?
Shane:
The idea would be that all the books, and other products for that matter in the store would be readable by Surface. Meaning that if you put, as we were doing just before… if you put the Las Vegas Lonely Planet guide on the table it would spill out some information from the book, to sort of bring the content of the book to life, but also link that through to more up-to-date information like maybe weather or exchange rates or, for that matter, Twitter feeds or blog posts.
So in-store it allows users to browse through that content, and learn about more content beyond the realms of the physical book. But the key thing was that we needed to create an experience that had a life beyond the actual table.
One of the most engaging aspects of it is… we had this idea for what we called passport cards, and in the current mock-up they look like passports. These are just pieces of cardboard so we could print thousands of them. What a visitor to the store could do was place one of these cards on the table, and then drag, out of the books, content that they’re interested in, whether that’s destinations or photos or maps or videos, whatever it is. On the Surface, it feels like you’re literally dragging the content onto this card. The photo shrinks and disappears behind the cards, and then what the visitor can do is take that card with them. Presumably they’re on their way to some exotic location. When they arrive at the location or when they get home, they can take that card, enter a URL into a web browser, and go to a similar experience that’s online using Silverlight, and fetch back that same content, and then link through from that to Lonely Planet online to the Lonely Planet site proper.
So for Lonely Planet it’s a great way to drive people from the physical books that people know well through to the digital property, which of course Lonely Planet is growing over time, and it creates a much more engaging, much more personalised experience. I’ve actually done a little bit of work in store to customise my passport, so it’s a really compelling reason for me to check it out and visit the Lonely Planet website later on.
Gerry:
Now Surface recognises the passport because it’s got these little roundy things printed on the back of it, but how do you tell the web what passport number you’ve got?
Shane:
The passports also have a number-number printed on the front of them…
Gerry:
Just a unique code?
Shane:
Basically that’s all it is. So when you go to the website, we have a Silverlight application that has the same look and feel as the Surface application, type in your number up there – we’re just looking at it now – where it says “passport number”. We go back out to a Windows Azure service, we pull down the content that was saved in the store and we redisplay it on the website.
Gerry:
And I guess there’s all sorts of interesting issues around that to do with copyright and so on. People going in and dragging everything from the book onto their passport and saying now I’ve got it all online.
Shane:
[Laughs.] Yeah, I don’t imagine for a moment that we would present all the content from a book, if only for the reason that it would actually make the user experience in-store much less rewarding, because if we overload people with too much choice, I actually think that would be detrimental to the experience.
Throughout the design process one of the things that Amnesia Razorfish was really conscious of is designing an experience that was engaging up to a point. What we didn’t want was people coming to the store and monopolizing the machine for hours on end. We wanted an experience that took 5-ish minutes. So we deliberately set some boundaries around what you could do, so that it was delightful, but you were left wanting more.
Gerry:
How is designing for a horizontal multi-touch interface different to designing for a traditional WIMP-type UI?
Shane:
It’s been really interesting and I was really lucky to work on this project to explore some of these issues, because I don’t think that we really know the answers to all these questions. So a bunch of issues came up, the first one being the obvious one: When a UI is embedded in a horizontal table there’s no logical top, so whenever you’re designing an application for Surface the first thing you have to think about is how or whether to orient the elements on the screen.
And the typical solution that most of the Surface applications we’re seeing today do is that when they initially populate the screen with controls and content they actually just randomly distribute these things on the screen and then let the user or users negotiate amongst themselves about how they want to arrange or share that content. Rather than having a traditional Windows application with “File, Edit, View, Help” across the top. You can’t do that on Surface, because you don’t know which direction people are facing as they use it, and you’re going to want to share those controls. So orientation I think was one of the most interesting things.
The other interesting design challenge is that you have to design bearing in mind that because it’s a multi-user application, different people will be doing different things at once. In a traditional user interface we can assume there’s only one user. When the user says they want to do a particular thing, we can assume – until they tell us otherwise – that that’s what they’re doing…
Gerry:
This is one of the things that particularly interested me when you were talking in Canberra last month, because you said… This issue of contention, which is a very interesting area, it’s something I’ve been interested in for a while. For example Tom Vanderbilt and I talked about it when we talked about his book Traffic – how you deal with contention.
And you were talking about situations where I might go up and drop a book about Buenos Aires and start looking at a video, and someone else would come in and drop a book about Las Vegas and they might start looking at a video. Can you tell me about how you decided… did you place limits on that; if you didn’t, why did you not do so? What was the thinking behind that?
Shane:
Well we explored a bunch of different options. Because while it’s possible to play the videos simultaneously, obviously the audio channels would… we could play both audio channels simultaneously but obviously they would interfere with each other and it wouldn’t be a great experience.
And so we explored different options. Possibly the natural reaction from a typical systems design point of view is that we would block the playing of the second video…
Gerry:
Yeah, pop up a dialog saying “Video already running!”
Shane:
Exactly. Please wait, or please close the existing video. But the problem with that is that we felt it would really degrade the delight of the experience, because suddenly we’ve reminded you that there’s a computer here. And really what we’re trying to do with Surface is create as naturalistic experience as we can.
So after some debate and exploring different options what we actually decided was it was better to leave the people using the application to negotiate amongst themselves. So if you’re playing a video and I want to play my video, I might start playing my video, hopefully realise that it was competing with yours and then a social negotiation has to happen. Either I stop my video and wait, or maybe I’m confident enough to reach over and stop your video so that I can listen to mine. Or we negotiate. And I’m quite excited by the idea that we leave it to the people around the table to resolve these sorts of issues.
Gerry:
That areas is really interesting, I don’t know if you’re familiar with Hans Monderman’s work, the traffic engineer who pioneered the concept of pulling out signals and road signs and so on, in many cases increasing throughput and almost universally decreasing the number of accidents, simply by saying there is contention and we can try to make it rule-bound, in which case that can work sometimes, but then people enter rule-based behaviour, so we take out all the social norms. He said well let’s take out the rules and just rely on social norms so you have this constant interaction and negotiation going on at junctions and intersections.
Shane:
It’s a very similar idea. We had a similar problem with mapping, although we didn’t manage to implement it at the time. An obvious thing that you would want to do with a Lonely Planet application is show maps. We didn’t want to have multiple maps simultaneously because maps by their nature need to be big and zoomable so the logical thing was to appropriate the entire background of the application for a map. But that means we can only show one map at a time. So how were we going to negotiate that?
The solution we came up with for that was different for the video. What we proposed to do was have a mapping target, a floating little round doughnut-shaped target called “map me” or something, that would only accommodate one destination, so when you dragged a destination onto this little target it replaced the previous map, and that was the way we ensured, that we would indicate to users that you can only map one thing at a time. So a different approach to solving a similar problem.
Gerry:
But nice and elegant also.
Shane:
Well we hope so. We never, as I say unfortunately we didn’t get as far as implementing the mapping feature. It would have been really interesting to see how that played out in real life.
Gerry:
I just had a play with the Surface for a few minutes before we started to talk, but one of the things I found quite challenging at times was there are all sorts of different objects that you’re representing – they might be parts of the body for surgery, there might be financial modelling, there might be destinations in the case of Lonely Planet. When you look at these things, each of them may have certain actions that I can take with them – I may be able to play a video, that’s kind of fairly easy because I may be able to click on a Play button, but there’s lots of things that we don’t know how to do. So we need a whole new language almost…
Shane:
Yeah. And I think, ironically, this is what is exciting me about designing for Surface today. While there is no language, no conventions and no standards, it’s… I probably shouldn’t be saying this but it’s a much more interesting design problem right now, when we’re really, literally exploring new territory.
Compare that to designing a conventional web application today where a lot of the conventions are so well known that the design space is pre-constrained.
But to answer your question, I think that what we’re going to see evolve over the next… I won’t put a time frame on it but what we’re going to have to see evolve is some conventions. But I’m happy that we’re letting the industry, and I don’t just mean Microsoft but all the various partners and organisations who are building touch interfaces, let them thrash that out amongst themselves. But yeah, today if you look at touch-based applications across various platforms there’s no consistency at all between gestures or manipulations.
But I’m happy to let that… I certainly wouldn’t want an organisation like Microsoft or whomever to say “this is the standard”. I’d like to see the community work that out.
Gerry:
You mentioned earlier on something about people being reminded there’s a computer there, and that’s one thing that I found as well. When I was playing with something that was interactive and graphical or audible or whatever, it was a very rich experience, but then occasionally, like in the financial planning app for example I’d suddenly be down dealing with text and boring stuff. And it really… it’s hard to describe but it was a big cognitive shift from one… almost modality to another. Any thoughts on that?
Shane:
It’s interesting because we don’t think Surface is good for every type of application. And I don’t think, for example, that you should use Surface for a text-intensive application like filling out a form. Because, as you say, you break from this idea of manipulating real-world-ish objects to do something that is much more sort of workaday computational.
But we definitely think that Surface is good for certain types of applications and not necessarily for others.
Gerry:
You described in Canberra as well as “delightful” the way that photos auto-rotate. Can you tell me about – why is it delightful?
Shane:
[Laughs.] Why is it delightful? You don’t think it’s delightful?
Gerry:
Well, when we were doing it didn’t work the whole time, Shane…
Shane:
[Laughter.] Well that’s not delightful. We tried… One of the underlying design principles of Surface we call the principle of the super-real. When we represent objects on Surface what we try to do is imagine if that was a real object in the real world, how would it behave? And then enhance that behaviour just a little bit, but not so much that it’s unrecognisable.
So one of the little tricks which I thought was delightful in the Lonely Planet application is when you touch on a photograph, Surface attempts to work out which direction your finger is facing based on the shape of the imprint and the shadow. So we have an idea of which way the finger is facing and therefore we can rotate the photo to face you. And to my mind that’s an example of where we’ve taken an object from the real world, a photo, and added a little bit of the super-real, without taking it to the point where it’s unrecognisable. So that’s what I was hoping would be delightful… when it works.
Gerry:
And I think it is quite delightful, but there were times when, and possibly it was a bit unfair the way that I was playing with it and touching, once it had rotated to touch it again and confuse it and maybe rotate back the other way. It’s definitely nice to have something recognise that Hey, this is the person that’s speaking to me at the moment and rotate that way.
In the Lonely Planet store, how did customers react to the prototype?
Shane:
Well we haven’t been lucky enough to deploy it in the real world… We have shown it in exhibition spaces so while that’s not a representative sample because we were showing them at tech conferences, certainly that… dare I say that aspect of delight showed through in what we’ve seen. The idea that a book can be more than a book, that a photo can be more than a photo, that a piece of cardboard can be more than just a piece of cardboard, is really engaging to people.
And of course, you know, bearing in mind that there isn’t a lot of these Surface technology devices around right now so it’s novel to most people. It will be interesting to see how five, let’s say five-ish or ten-ish years out, when this sort of technology is much more pervasive, then that novelty will hopefully have worn off and maybe people will approach these things differently.
Gerry:
You already said that there are a lot of applications that you don’t think are suited to Surface. Where do you see Surface-like applications being developed, and where is the Surface most suited for?
Shane:
Today with the form factor of Surface… Obviously as I said it’s deliberately meant for situations where people can come together, whether that’s people who already know each other, or people who may be strangers but can collaborate on something.
So that necessarily probably means public spaces. Not exclusively, but that’s an obvious place. So most of the interest we’re seeing in Surface today is sort of the retail/public service type environments. In America today they’re being used in AT&T phone stores, for example, so that customers can learn more about phones. They’re being used in Sheraton hotel lobbies where people can browse maps and explore local tourist attractions. Deliberately, hopefully, people can work together and collaborate and work together on these sorts of things.
I think certainly in the first generation of Surface applications that’s going to be where we see the most adoption.
Gerry:
It’s interesting you say retail and retail spaces. I was in the new Westfield London a few months ago, looking at their online kiosks. The Westfield London is beautiful – have you seen it?
Shane:
I haven’t. I’ve heard that it’s quite glamorous.
Gerry:
It’s a beautiful shopping centre. The information kiosks are beautiful but almost utterly useless. I spent about half an hour watching people using them and I estimated there’s a 90% failure rate on finding anything, so people go to the kiosk, spend a minute or so and then they either ask a security guard or a human at the information desk to find that they want.
Shane:
So this is a problem of design or with the technology or…
Gerry:
I think it’s a problem of failing to understand the context in which people are looking for stuff, and I guess maybe something like Surface might be ideal because it struck me as a failure to use ego-centric rather than geocentric maps, so you see people trying to auto-rotate Westfield. [Gerry's note: I blogged about this elsewhere in January 2009.]
Shane:
Oh, right, try to map up to left…
Gerry:
Yeah, up to left and level 3 and so on instead of the device giving them some more information about where they actually are.
Shane:
I think we’re going to see a lot of really interesting examples of these sorts of technologies, and not just the touch-based technology, you know also more use of physical objects to manipulate technology going forward.
It’s going to be interesting to see how these things evolve, not just on big-format screens like this but on small portable screens as well.
Gerry:
If there’s anyone listening who’s interested in dabbling in this area, whether it be with Microsoft Surface specifically or multi-touch or, I guess, just beyond the GUI or the WIMP user interface paradigm which is getting a bit long in the tooth at this point, what should they be doing?
Shane:
There’s a couple of things come immediately to mind. Firstly there is, as you know, plenty of discussion evolving on the web in blogs and stuff about designing for gesture, designing for multi-touch. And tangible UIs not so much but there is some stuff. So there is plenty of stuff to find on the web, there’s people like as you know Dan Saffer has written a book on designing for gesture, so a body of knowledge is starting to evolve.
From a Microsoft-specific point of view, I guess there’s a couple of things. First we should mentioned that Windows 7 is multi-touch enabled as well, so any Windows 7 device, if you have a multi-touch display you can design for multi-touch…
Gerry:
So if I run Windows 7 on say the newer Dell machines I’ll get the multi-touch experience?
Shane:
Yeah. So that’s a relatively accessible way to start playing with multi-touch. If you’re specifically interested in Surface technologies you can, if you’re a Microsoft partner, and it’s free to become a Microsoft partner, you can download the SDKs and the simulators and start playing for yourself.
You can go to the surface.com website and follow your nose to the partners’ section. You can join up and download a whole bunch of guidance and software and sample applications and stuff that we’re putting together.
Gerry:
Shane Morris, thank you for joining me today on the User Experience podcast.
Shane:
You’re welcome Gerry.
Published: November 2009