The future of games in museums: what should we be doing?

A confluence of related projects and talks has got me thinking about where games in museums should be going in the future. There have been some notable successes in museum games to date, and some failures. Where to go from here? Here are some assembled thoughts on the types of games and game design practices I would love to see more museums exploring.

Collaborative games

Now, I love competitive games, but not everyone does, and competition can be off putting and disruptive in, say, family situations (I’m sure you all have stories about the game of Monopoly that ended in tears). Collaborative games are perhaps more suited to the mixed audiences and interests that are represented by museum visitors.

For example, Spaceteam is absolutely one of my favourite games of the last few years. It manages to be ridiculous, hilarious, breathlessly exciting, social and visually striking, all at the same time. Go find 3 other people with smartphones or tablets and have a go, it’s hard to explain. What’s particularly brilliant about it is the way that players become instantly collaborative through the mechanic of needing to convey information to each other rapidly. There is no competitive element, you are all working together to stop your spacecraft coming apart at the seams.

There are also lots of board games that work this way, (I tried Forbidden Island the other week, which is a good example), search for co-operative play on Boardgame Geek. The UVA Bay Game, a team based sim about sustainability, is another interesting case, and seems to have resulted in genuine behaviour change as players realised they would have to work together to solve the issues, both in the game, and in real life.

Discussion based games

Where discussion is happening, thinking is happening. Most museums want to be in some way thought-provoking, and recognise that seeing people deep in conversation about the objects or display they are looking at is a good sign. But many people don’t feel entirely comfortable sharing opinions about art, or history, or science, feeling they lack knowledge or will say something that will be ridiculed. Or, perhaps, it isn’t part of their normal group nature to have discussions of this sort. This means if you want to encourage discussion, you may need to scaffold it in some way, and games can be great for this.

One of the most interesting game experiences I’ve had was playing Liliane Lijn’s Power Game  at the ICA a few years back. The atmospheric set helped, but at it’s heart it was a sort of poker game (actually based on Chemin de Fer, I believe) where you had to make the case for why the word you had been given was more powerful that another. Everyone at the table would then vote, and if you won, you got the chips in the middle. It’s more complicated than that, in truth, but what it means is that you end up having to make arguments for abstract concepts you wouldn’t normally think about (i.e. is “war” more powerful than “love”?).

An online discussion game which I’ve enjoyed playing is the Foresight Engine. In this, there a future scenario to which players must respond by playing different types of card that discuss the outcome or effect of that scenario. You get points for each response you play, for responses to discussions that you start, and for having your response highlighted by a moderator/judge. It is effectively a collaborative crowd-sourced future forecasting tool and it’s been applied in all kinds of real world situations (they used it in Christchurch to help citizens work through the implications of certain scenarios for their earthquake damaged city, for instance). I’m certain a card based game that was a mix between this and Power Game could work really well in a museum setting, perhaps as part of events.

Rapid, casual games

Fast, casual games with the right mechanics can be super addictive. This is why I don’t open Bejeweled Blitz unless I know I have at least half an hour to lose to it, even though each game only takes a minute (and is totally brainless, yes, I know, I don’t care). They can also manage to convey a simple point effectively, as we found with the Axon game at Wellcome. In Axon, players click to move a neuron forward via protein targets, with games lasting sometimes just a few seconds, and in doing so (as we found via a survey of players) they learn something small but interesting about foetal brain development, get a sense of the aesthetics of modern brain imagery, and have their interest piqued sufficiently to follow up with a visit to Wikipedia.

Also look at super fast task based or point and click based games such as Wario Ware or McPixel. Each level takes seconds. This format can work really well, be exciting even if the task is incredibly unsophisticated (and therefore work for a range of abilities) and can also be fun to watch. Try also Tenya Wanya Teens if it comes to a place near you. It’s physical, quick, and fun for spectators as well. Oh and whilst we are talking about crazy fast games that are also hilarious, if you haven’t tried QWOP, or any other of Bennett Foddy’s games, you really must.

The temptation is for museums is to create games that convey a lot of information (because we have so much interesting content, so it’s understandable), or that are a bit worthy. But those games are really really hard to do well, and risk turning off a lot of people. For in-gallery games, museum visitors may be time conscious, or assume it will be serious, or just want to watch, and fast, funny games could be the answer.

Pervasive games

Museums are great spaces. And recently Lates events have become very popular. Many of these are already using pervasive games to get people interacting with the spaces in new and different ways. Hide and Seek’s Sandpits do something similar, as at the National Maritime Museum a while back. Capture the Museum (Thoughtden and National Museums Scotland) is a team based pervasive game that also uses smartphones to deliver puzzle based challenges (if I’ve got that right, I haven’t had a chance to play yet).

So this isn’t a new idea, but it would be great to see museums do more of this. These sorts of games are often easier to run and develop (and therefore cheaper) than digital games but are frequently overlooked. If you can turn them into a card or board game or just a set of instructions, you could also distribute online which helps it reach a bigger audience.

Yes, many museums don’t want people running around and yelling during regular museum hours, but not all games have to involve this. At SFMOMA, their ArtGameLab crowdsourced games to be played in the public spaces that were more sedate, but still fun (e.g. go around the museum critiquing artworks using only your facial expressions).

Card/board games

Having mentioned card and board games above, I think it’s worth highlighting them. Are there any examples of existing museum card or board games? I wonder why not.

Locative, mobile games

Taking the game out of the museum is not new. We tried it with Magic in Modern London, Tate did Magic Tate Ball (more toy than game, I guess, but still), and others have released games on mobile. But it feels like the potential of mobile is not yet being fully exploited, particularly around location, and the relationship between objects and the landscape, whether urban or rural.

Developing bespoke apps of this type can be expensive and difficult (but if you get it right, wow). But there are other platforms you can use, I recently looked into using the SCVNGR platform for a museum project and was disappointed to find out that it was now unsupported, which is a shame, because it would have been perfect. But you could use other trail apps to add challenges or mini games, or work with Junaio and its AR functionality to create something playful. In researching SCVNGR I found another ex employee has created a similar platform called Edventure Builder, which could be worth exploring.

Online, casual games

This has probably been the most successful approach to date. Tate, Science Museum and our Wellcome Collection games High Tea and Axon have all demonstrated that online casual games distributed to portals can be successful in reaching large audiences, and having an impact in terms of learning (read the High Tea evaluation here). The Science Museum had a rarer success with Launchball, a game that was only on their website yet reached a large audience (via a post on Reddit, if I recall correctly), but in general distributing to portals seems to be the most effective approach.

What this means is that there is already a good existing model for doing this. Use it! The potential audience is in the millions. And if you develop in a way that your game can be released to mobile as well (using Unity, say), it’s even bigger. This approach isn’t the cheapest, and you need to work with real game design pros, but it can be very effective. When you look at the value of High Tea, it was working out at about a penny per play, and had a genuine impact on players in terms of learning and thinking about the subject matter.

Games based on 3rd party platforms

I’ve already mentioned this in terms of locative games, but there are many game development tools out there that are relatively simple to use (I list some in this previous post on making games on a budget). There are also tools that were designed for other things but could be repurposed to playful ends.

I saw an online production recently (the Nightvision Experiment) that frankly didn’t work for me in terms of plot or acting, but used a clever mechanic of delivering the entire story via twitter and youtube. And The Dark Room used just Youtube and it’s annotation function to create a smart, funny sort of video text adventure. There is no reason social media can’t be used to create games, or interactive experiences, and in fact Liliane Lijn was at one point running her Power Game over twitter as well.

Console game partnerships

I don’t think anyone has tried this yet in museums (although the Wellcome Trust did try it as part of their broadcast and games funding work), but I’m sure there is potential here. Obviously museums are unlikely to be able to afford to develop a AAA console game of their own, they cost millions. There are increasing numbers of indie devs producing games for consoles (especially since Unity can output to several of them) for lesser budgets, but still, it’s expensive.

However, museums do have content. They have stories, settings and objects, and they have all kinds of experts. Many are respected global brands. Might a canny partnership be possible between a console game producer and a museum? Perhaps where a museum can do and fund some development work into the factual elements (or even non factual elements), provide some information, setting, or idea inspiration?

Evaluation

I do go on about this, but this needs to happen more. I’ve just read a fascinating evaluation for a Science Museum game (that I hope will be shared soon) and it was *really* illuminating. I wish I’d seen it earlier. It made me think that we’ve all done enough games now that there should be a pretty good body of knowledge about what works, and what doesn’t, and we should be building on this. Some of this information does get shared at conferences and so on, but it still feels like a lot of museum game development (and other digital development, in fact) happens in the dark.

We can copy other apparent successes, but without knowing in depth information about the player responses, we may just be repeating an empty exercise in gathering hits.

Games people

Museums need to focus on working with people who are good at making games, and not get so hung up on the platforms or technologies. This essay by Suzy Glass isn’t about games specifically, but it could be, and it is absolutely spot on about this issue.

Museums also need to be working with those people right from the start of a project, not waiting until they’ve put together an extensively scoped funding proposal without any games expert input. And that means hiring someone, freelance, or full time and paying them from the start. It will be worth the investment. I’d also like to see less of small games and other digital agencies time being wasted by having to jump through multiple time consuming (and therefore expensive) hoops as part of the procurement process, when they have a worse than 5 to 1 chance of getting the commission.

Those are my assorted, random, thoughts, please feel free to add yours!

Electronic visualisation and inspiration: the EVA conference 2013 #evalondonconf

I spent Monday and Tuesday of this week at EVA, the Electronic Visualisation and the Arts conference for which I was lucky enough to get a bursary to attend. It’s a multidisciplinary event, which describes itself as being about “electronic visualisation technologies in art, music, dance, theatre, the sciences and more…” It was certainly diverse. Some presentations were technical or academic, others were more practical, and they were on a range of subjects from a wide variety of institutions and individuals. “Electronic visualisation” covers an lot. I left quite inspired, if wondering on earth to do with all the ideas now whirling around my brain. I highly recommend it.

Here are some of my highlights from the two days I spent there. For the full proceedings go to http://ewic.bcs.org/category/17656#.

Creating Magic on Mobile

My paper kicked off the day, so let’s get that out of the way. I presented (with Alex Butterworth of Amblr) on “Creating Magic On Mobile”. You can read the full paper here, which discusses our experiences creating the Magic in Modern London app for Wellcome Collection (when I worked there as Multimedia Producer). It’s a geolocated treasure hunt set in Edwardian London so the paper covers the use of GPS, audio-visual content and a map based interface to create a historical semi-fictionalised narrative.

It also contains a call for more imaginative use of mobile in cultural organisations, and shows how we attempted to do this with Magic in Modern London, but also some of the challenges we faced along the way (such as trying to overlay a 1908 map onto google maps). It was a pretty experimental project, but also very ambitious, and we learnt a huge amount along the way. It played a large part in my drive to help set up ME:CA (Mobile Experiences: Cultural Audiences).

Keynote: Steve DiPaola “Future trends: Adding Computational Intelligence, Knowledge and Creativity to Interactive Exhibits and Visualization Systems”

Steve DiPaola (a “computer based cognitive scientist, artist and researcher”) gave a fascinating keynote that covered several of his projects. The notes from his talk are here http://dipaola.org/eva13/. These included.

  • A beluga whale pod interactive simulation at the Vancouver aquarium being used both in gallery and as oart of scientific research. This installation allows visitors to observe particular behaviours and test out scenarios.
  • Teaching evolution through art: using “evolved” artworks to help people really get what evolution means, particularly challenging in America, where 60% of people don’t believe in it (apparently). This was done as part of Darwins 200th birthday celebrations.
  • Analysis of Rembrandt’s works to understand his compositions and techniques and their affect on the eye path of the viewer. This work came to the conclusion that Rembrandt had an impressive understanding of vision science 200 years before anyone else.
  • A project to create life-like (ish) 3D avatars that people can speak through to address people remotely, and perhaps anonymously (e.g. scientists having a digital conversation with the public from another country).
  • An analysis of Picasso’s “Guernica” 40 day period of creativity. This mapped all the works created during this prolific period onto a timeline to try and understand the nature of his creative process, how ideas evolved, and how he was influenced by other works.

Timelines

Timelines were also feature of other talks, such as this about “representing uncertainty in historical time”, I didn’t see this presentation but did see the demo later, which looked at turning the works of a composer into a timeline that mapped them against their first performances. Can’t find a link for this, but I was impressed, as with the Guernica project above, at the way a putting data into a timeline can reveal new insight.

More museums and mobile (with added AR)

There were a couple of other museum mobile projects, both of which used augmented reality (AR) in different ways. Timeline Newcastle looked interesting, and quite similar to Streetmuseum in overlaying archive photos onto the modern city (if I understood it correctly). I was particularly interested to see that the Smithsonian Natural History Museum have a very ambitious app in the works called Skin and Bones. It will focus on 14 objects, giving different levels of engagement with each, based on their framework for visitor preferences called IPOP (ideas, people, objects and physical). So for visitors who want to hear about people, they have a meet the scientist option, for physical – an activity, for ideas – an exploration of a scientific concept, and for object – a detailed study of the skeleton some of which used AR to overlay the skeleton with an animation.

I was interested in their concept of using it to raise “visual literacy” ie, helping visitors to understand and interpret what they were seeing  and in doing so increase dwell time in the galleries. They had recognised that users were spending little time in this gallery, except for at a few “hero” skeletons, and surmised that people weren’t finding it interesting because they didn’t know how to make sense of what they were seeing. So in the app, for example, it will show you how a particular venomous snake’s jaws hinge back into their mouth and how you can see that in the skeleton.

They have an impressive (and expensive sounding?!) user testing plan. They are creating two apps,  one without AR and one with to see how that affects visitor interaction, and will be using a beta app path analyis tool called Look Back to help with this (although this feature is also apart of GA mobile analytics as well).

Keynote: Linda Candy “Creativity and Evaluation: Supporting Practice and Research in the Interactive Arts”

This was an impassioned call for making evaluation a key part of the creative process in a keynote from writer researcher Linda Candy. It’s not just about evaluating impact, but also creating new knowledge and new works, she said (but impact also still important, I would add!) and should be fully embedded in practice. For artists creating digital interactive works there are also usability issues they should be testing (do people understand how to interact with it, for example? Is it within reach of children/wheelchair users?).

Obviously, as a huge fan of evaluation, I am very much down with this. Evaluation isn’t just about fulfilling the tedious criteria of funding bodies, but is more about understanding and improving your own work. The evaluation work I’ve done on games has been some of those more interesting and thought provoking work I’ve done.

Other assorted interesting presentations: critical robots, doomsday clocks and more

There were several other interesting demos and presentations.

Stop, my brain is full

There was a whole other day of this that I missed, probably for the best, as by the end of it my head was spinning.

Why I’m fed up with digital projects (and why I’m not): a rant

I’ve felt a little rant bubbling up in me over the last few months: a sense of disquiet about digital, a jaded annoyance about wasted time and resources and opportunities squandered. Today I was reminded about an old project that was the epitome of digital idiocy, one of those thoughtless knee-jerk “we must have an app!” projects that make me want to throw a toddler tantrum, kicking and screaming “but who is it for?” until someone agrees to at least do a bit of audience research or string together a minimally viable set of objectives. And that reminder seems to have brought it all to the surface, so here goes.

I am fed up of seeing people and organisations produce digital rubbish: poor apps, clunky games, badly designed microsites and other half-arsed online, mobile and technological systems and whatnots. I am fed up of people who are smart about digital, who think about users, who ask the right questions at the start, who embrace technology without fear and understand how to apply it, being over-ruled by people who’ve just bought an iPad for their kids and now assume that everyone has them and that this is all the justification they need to insist upon spending £50k on a new app for their organisation (an app for who?! For what?!). Moreover, I am fed up of projects that fail being brushed under the carpet, with no evaluation or debrief that enables everyone involved to learn from mistakes and build something better the next time.

It makes me sad to see money and time being wasted, when it’s usually avoidable, and even more so if nothing is then learned from this. It makes me especially sad, because the technology we have available to us today is AMAZING. My phone is able to do astonishing things and the computers I use have incredible firepower (I still remember how exciting it was to get a floppy disc drive for my BBC micro, now look at them!).  It feels like there is so much potential and so much more that the technology we have could allow us to do.

It’s not like there aren’t enough examples of brilliant products, apps, games, and other applications of these technologies out there to learn from. Too many to list. But many many many more that have been an utter waste of everyone’s time from the beginning because they just didn’t ask some basic questions about who and what their product was for, and test it with that in mind along the way. Isn’t that a bit sad?

I know it’s not easy. That there is a degree of luck in rising above the huge amounts of competition online or on on the app store, that things beyond your control can go wrong during the development of a project, and that there are some times you need to go on your gut instinct, and that’s not infallible. Goodness knows I’ve made every kind of mistake creating digital things over the years, as have most people working in this constantly changing field. And I will continue to make mistakes, as will most people etc.

So I’m not claiming there is a perfect process, but there are better and worse ones, and there are definitely massive howling alarm-bell-ringing clangers that should be sending up the red flag right from the start. The “we need an app, never mind who for”, the “we’ve got all this content, let’s just stick it online, people will come to us”, “let’s create a whole new portal for something that already exists”, and so on. And sometimes it feels like we aren’t moving on from this, these whack-a-mole stupidities keep popping up over and over again and just will not die.

Why is that? In a recent conversation with a fellow person-who-works-somewhat-indeterminately-within-“digital” we discovered we’d both been feeling this slight ennui with the field. This sense that it wasn’t meeting its potential, that a lot of rubbish was being made, and that it was hard to see how to make things better. We wondered if it it might be that digital is constantly being driven by the hardware makers, by Apple and console or computer companies and that their vested interest in creating new stuff for us to buy means we’re always just playing catch up. Why do we let this dictate the pace? Where is the chance to take stock and develop best practice?

My experience as a freelancer over the last few months, applying for funding and responding to ITTs, has also been eye-opening. The systems and funding in place for making digital stuff are really broken. If you were trying to design a process that would result in crappy digital products, you’d make people (who may not be particularly digitally literate) scope everything out and pin it down before they’d ever been able to build anything or ask their audience about it, only then engage an agency who are the real experts but can’t change the scope, and then make the project team go through a month long change request process if they want to develop the project in a different direction from the scope based on the evidence as they start building and testing things. Which is exactly the process in place in a frighteningly high number of places. The old models for procuring new infrastructure etc just do not work for digital projects, and it’s hamstringing them from the very start.

Plus money, of course. Not enough money, or not tailoring the project to the actual budget available.  I’ve had this rant elsewhere. I seem to spend a lot more time these days telling people not to make a digital thing. Would it work just as well as a piece of text? As a printout? As a physical activity? Or a really simple website? Don’t overreach if you just can’t afford it, do something straightforward and proven.

But at the same time I love working on and seeing others working on brilliant digital projects. The reach you can get, the amazing response in terms of both scale and thoughtful quality, the sense of shared global experience, the playfulness and the utility, when done well. So, how do we do more of these?

How to make games on a low budget

So you’re convinced, as I am, that games are a great way of reaching new audiences and engaging them meaningfully with your message or content. But you have little or no money, what to do? It’s all very well when you have upwards of £40k to spend (and ideally even more), but what about when you don’t? Are you excluded from creating or commissioning games?

Well no, I don’t think you are. But you’re going to have to do things a bit differently. You may have to give up a certain amount of control, and be relaxed about where games end up. But you probably should be anyway, the point is to get your message out there, right?

What not to do

I’ve recently heard about two separate organisations that were looking for proposals for educational games for between £500 and £2500, which made me gasp, I must admit. Taking this approach to doing games on a budget – just not paying much for them – is a very bad idea, for a number of reasons. I speak from experience, believe me.

Low budget projects are ALWAYS the projects that cause the most problems and take the longest. Your game will naturally take a back seat with the agency or developer when higher paying jobs come their way, and fair enough. As the commissioner, you won’t have much clout or sway over how the project develops, and aren’t likely to have much opportunity to change it as changes=time=money. Unless the agency/developer is bad at handling that equation, in which case they are likely to go bust trying to complete your game.

It’s the old cost/time/quality triangle. If the cost goes down, either the amount time goes up or quality goes down. Or, more likely, both. And since time=money, as established, the true cost actually becomes much higher for everyone involved. No-one wins and it’s bad for everyone’s business.

Instead, maybe try one of these options.

Find more money: other funding sources

There are a number of sources of funding for games out there. My old employer, the Wellcome Trust, is keen to encourage more games on biomedical subjects and offers a number of potential grants (this development grant, for example). Other public engagement funds are also likely to cover games projects, even if they don’t explicitly state that. Look at what people like Nesta or the Arts Council are currently offering in the way of funding. IdeasTap appears to have quite a lot of funding bodies listed on its website too.

I’ll add in more examples here as I come across them, but please do suggest any you know of.

A tip. Always always always, with any grant application, get in touch with people at the funding body to find out more and get advice on whether your project is suitable, or find out what you might need to do to make it suitable. Public engagement funding in particular is likely to need some sort of decent evaluation of impact built in, so don’t just say you’ll count the number of hits, give it some thought.

Find more money: partnerships and co-investment

You might not have the money, but presumably you do have something great to offer – content, domain expertise, a well respected brand etc. All of these might well be appealing to someone who does have money. A games agency might be interested in developing something in return for profit share. A company might see a good fit with their aims and want to sponsor your project. Another similar organisation (arts, cultural, educational?) might be interested in a partnership, or perhaps a group of you could club together and pool budgets. A broadcaster such as the BBC or other online platform might be interested too.

Partnerships of this sort can be tricky, true. All parties need to be very clear about their roles, where IP rests, where the final sign-off lies and so on before getting too far along with it. Get an agreement in place as soon as you can, and you may have to be prepared to relinquish some control. Regular communication is obviously really important too.

There are potential benefits beyond just the extra budget though. Your partner might also have additional expertise or resources, for example in marketing, that could be very handy. They might have their own large audience which you would then have access to. Choosing partners for what else they can offer is therefore wise as well.

Use existing game creation tools

You don’t necessarily have to start from scratch when creating a game, there are a number of tools out there that can simplify the process. Perhaps you or someone in your organisation could even have a go at making one yourselves.

I haven’t tried many of these tools, so can’t vouch for them. I know GameStar Mechanic has been successfully used to get kids building games, as has interactive fiction creator Quest (which I am a big fan of), which should be a good indicator of their ease of us. GameMaker: Studio by YoYo games is another possibility.

Googling for game creation tools throws up loads more options, Sploder, The Game Creators, Game Gonzo and more. Anyone used any of these and had a good/bad experience?

Run games jams or competitions

Games jams involve inviting lots games designers or games development team to spend usually 24 or 48 hours rapidly building a game on a particular theme. They’ve become more and more popular recently. Done well, this could be a good way of getting lots of games about your subject matter of interest out there. Done badly, though, they can feel exploitative and yucky.

It’s probably wise to work with someone who has run successful games jams in the past, and understand what it takes to make them work. Or work with a regular games jam event. You need to make sure people have a reason to be there, that their time is valued and that they are in some way compensated for it. Oh, and that they keep hold of the IP. The ones that Wellcome ran simply paid developers for their time, and then ran it as a competition so that the winners got money to develop their game further.

Update: This “Hack Day Manifesto” provides some useful advice for running events such as these (via @oonaghtweets and @dannybirchall on twitter)

Game design students

There are many game design courses out there. I haven’t actually tried this myself, but another option might be to work with a university or other teaching organisation that has a games design course to provide a brief for students.

Are you involved with a game design course that would be up for doing something like this? Let me know in the comments!

Try something different

Does it have to be a slick online game? Could you create a paper or card game and then make it available online? Add new rules to an existing game a la the brilliant Boardgame Remix Kit? Maybe if you want to get it into schools, you should just do something simpler like provide pictures and lesson plan suggestions and upload it to TES. Or do the same with instructions for a live or pervasive game that students could play?

And finally

Thanks to Sharna Jackson, Phil Stuart and Kim Plowright for their suggestions, which I’ve incorporated into this post. If you have any more thoughts, funding sources etc, please do add these in the comments and I will update the post.

Update

Updating to add some very useful additional thoughts from others following a discussion on the LinkedIn Games Based Learning Group.

Dustin Chertoff: The more the commissioner has completed up front, in terms of both art assets and game design, the cheaper it will be to actually develop the game. But that also means the commissioner is likely to be more resistant to the inevitable change requests from the developer saying “such and such feature doesn’t work or is too complicated.” A co-design between the commissioner and the developer is very useful in this regard, unless the commissioner already has a strong game development background (which is usually not the case).

As far as keeping project costs down, engines such as Unity3D are a huge development boon. Devs can start working on game system development much faster and there is a huge library of tools and assets available that further cut down development time.

Even so, from my experience it still takes a team of 3 (2 devs, 1 artist) working full time around 20k USD about 1.5-2 months to design, develop, test, and polish a relatively simple game. It does become faster and cheaper to add more content once you get the core game systems developed and tested though. But the low-cost commissions I’ve seen never have money beyond that first version.

Mathew Georghiou: Budget is always an issue and most people do not understand the complexity and effort required to develop a game as compared to other types of applications. There has been some industry research done that suggests the average mobile app costs $20,000-$40,000 to develop, and that seems to mirror my experience and that of Dustin’s comment above.

Some very basic apps can be done for less, but there will always be some significant compromises required as you have identified in your article.

The best advice is to always consult with someone with experience before designing your specifications or budget, and certainly before issuing an RFP, so that you can make sure to develop a plan that is feasible.

Peter Stidwill:  Although not a tip, this visual document from the Games for Impact academic consortium here in the States has some ballpark figures for professional game development. This is helpful to point to when trying to set expectations about what can be achieved.

http://gamesforimpact.files.wordpress.com/2012/09/gamesforimpact-bestpractices.pdf – see ‘Production cost estimate’

Evaluating online video

Whilst I was Multimedia Producer at the Wellcome Trust I began a programme of evaluation for our online videos. I left before I was able to see most of the results from that, but in this post I’ll share the plan of action I had for the evaluation, which used a number of different approaches. I was building upon work that Danny Birchall and I did in games evaluation, using some of the same methods, but evaluating video presents a somewhat different set of challenges. If you have any thoughts about other techniques one could use, or ways of improving on my methodology, I’d love to hear that.

Wellcome Trust YouTube Channel
Wellcome Trust YouTube Channel

Why evaluate videos?

At the Trust we (me and the other Multimedia Producer, Barry J Gibb) were creating short documentaries about the research funded by the Trust, educational films, and so on. The audience varied, but was usually considered to be an interested lay audience or a GCSE/A-level student. Our video dissemination strategy was fairly straightforward, we posted them on our site (where they frankly got a bit lost) and on YouTube (which I always considered to be far more important especially because of the much greater chance someone would find them there, and the ability to share).

I set the YouTube channels up in early 2009, but had no formal plan of evaluation (oh! to turn the clock back) and simply kept an ad hoc eye on analytics and feedback. The analytics were sometimes interesting, but commentary on YouTube, you may have noticed, is rarely that worthwhile. When we got lots of it, it was generally polarised and sometimes unpleasant. When we got none, which was usually the case, it was like putting the videos up to the sound of tumbleweeds and crickets. We had no real idea who was watching our videos, whether or not they liked them, how they were using them, and therefore it was hard to know how to improve them.

That was the key, for me. How could we develop a strategy for the future when we didn’t really know what had worked or not in the past? But all was not lost. A lot of information was just sitting there, in the form of analytics, and to find out more, we just had to ask.

My approach

I split the project up into a number of different areas:

1. Analytics

2. Staff feedback

3. Audience feedback

4. External organisations

Analytics

My plan was to track several paramenters for the lifetime of each video, ideally broken down into years. I had decided to go for calendar years rather than years since posted to be able to compare videos across the same period, but was somewhat in two minds about it. Youtube analytics offers information about views, traffic sources, and playback locations, which I had planned to include, and demographics and audience retention, which I hadn’t. Demographics on YouTube are only tracked for those people who are signed in and have added those details to their profile, so are largely worthless. I guess they may be less so now that it’s owned by Google and more people are likely to be signed in, but I still wouldn’t place much stock in them. The audience retention graphs are interesting, but hard to compare.

The analytics presented the biggest challenge for the project, though.  The first challenge was that the analytics were split between Google Analytics for the videos on our website and Youtube’s own analytics. The latter are much more restrictive than the former, though they give more detail on the attention patterns, for example. GA depends very much on how it’s been set up for video, but thankfully when GA was set up for the Wellcome Trust site at least someone had the sense to track the number of actual video plays as events, rather than rely on pageviews. But still, marrying GA and YouTube together wasn’t going to be easy.

The second issue was that getting data out of YouTube isn’t easy. You can download spreadsheet reports for one parameter over one time period at a time, but when you have well over 50 videos and want to try and track several parameters over several years, well, I didn’t fancy doing that kind of epic legwork. There is a solution, but it’s not ideal either. Google have an experimental API for YouTube analytics, and our IT department were as I left on the case of trying to write an application that would use it to built reports.

I did have a couple of initial findings from my more ad hoc monitoring of stats over the years, though these aren’t going to be hugely robust and I had hoped to interrogate them further with more data.

  1. On YouTube, most people were arriving after searching for relevant keywords *in Youtube*, (ie, not through Google organic search), or from related video links. The latter were definitely responsible for hits on our most popular videos, that were coming up as related links to other high profile videos from others.
  2. Many more people were arriving directly at video pages rather than the Channel page.

Staff Feedback

The Wellcome Trust has several hundred employees and, as with our external audience, we didn’t know what all of them thought of our video output. We didn’t even know if they were all aware of it, that they could use it or even commission their own films. Their opinions and needs were also very important to developing our video strategy. I put together a survey in Survey Monkey, had it checked by our UX expert (always worth doing)  and posted it on our intranet. I wasn’t around for the results of this one.

Audience Feedback

This was the bit I was most interested in, qualitative feedback from our audience. For our games evaluation work we had put up a survey at the end of the game, followed up with telephone interviews, and also looked at community commentary. As we’ve established, for Youtube the latter is not particularly useful, and we also didn’t have comments open on our site. But the survey was still an approach we could take, except instead of one game, we had ninety three videos. And instead of several hundred thousand plays over the launch weekend, most videos would have a few hundred views over their lifetimes.

This meant that we would need a much longer data gathering period as only a small fraction of people will answer a survey. As always, we ran a prize draw with this to help the numbers, but even so, it seemed wise to assume it would be six months (at least) before we had enough responses to come to any conclusions.

I drew up some survey questions, with the help of colleagues and again had them checked over by Nancy Willacy, our UX expert, to make sure they were understandable and that we were asking the right questions. There is lots out there on survey design, including some points from our own experience in this paper on games evaluations that I wrote with colleagues at Wellcome, The Science Museum and the National Space Center. Being very clear about what you are trying to find out is very important, it’s easy to creep beyond that and try and find out everything, but nobody wants to answer a huge survey online.

My intention was to add the survey as a link on our site next to all videos, and also to put them up as linked annotations on YouTube videos. I did the YouTube links myself before I left, which was a pretty tedious job. You can see an example at the end of this video. I have a feeling that you may have less options in the annotations if you aren’t a YouTube partner, and that adding links like this might not be possible. I guess you’d have to add one in the text below, but that’s not ideal.

External organisations

Finally, I wanted to know what other people were doing with video at similar organisations (and also what they thought of our output, if they’d seen it). This was to be much less formal, simply identifying other organisations and writing to them with a list of questions, at least in the first instance.

And you?

I hope some of that is interesting or useful, but it would also be great if other people share their experiences of evaluating video. What do you do in your organisation? What could we have done better here? Even better, does anyone have any results they can share?

An encounter with Patrick Moore (people are complicated)

A story I would like to share, on hearing that Patrick Moore passed away today (or perhaps yesterday). Already twitter is filled with a mix of sadness at his death and discussion of his positive legacy for science, but also reminders of his sexist comments and fairly extreme right wing views. I think it’s OK to talk about both these things, challenging the latter and celebrating the former.

When I was much younger I was a huge astronomy nerd and Moore was obviously an icon. When I was perhaps around 11 or 12 I was at an astronomy conference in London with my mother, who would escort me to astronomy weekends, observatory viewing sessions, and events like this. Yeah, I know, I was a weird kid. During a coffee break at the conference, Patrick Moore was there just sort of hanging out, and my mother encouraged me to go say hello.

I did, and Moore was charming and friendly. He was obviously pleased that I was so enthusiastic about the subject, apologised that he couldn’t talk for longer, and invited me and my mother to come visit him in Selsey. We arranged a time to do so via post (imagine, using the postal service to arrange a meeting!). I even kept his typewritten notes to me, see below, so I must have been a bit starstruck (pun intended).

A message from Patrick Moore
A message from Patrick Moore

We went to visit during a holiday nearby, and he was a great host. He showed us his telescope (no jokes please, seriously) and various bits of astronomical equipment and garden observatory. Finding out that I played music, he played his xylophone and recordings of some marches. We had tea and cake. He was clearly keen to encourage my interest in astronomy and science. We left after a couple of hours of visiting and that was the end of any correspondence, beyond a thank you letter from me of course, but the encounter left a huge impression on me.

Years later, I was immensely disappointed to discover his political views, and especially disappointed to hear his sexist comments about women ruining TV and so on. How to square this with the avuncular character I’d met who was so supportive of my enthusism for science? People contain multitudes, I guess. His views are clearly more complex than the impression you get from what is reported, but this isn’t a defence of them. It’s disappointing that he wasn’t challenged more on this in his lifetime.

We sometimes forget that people in the public eye are as nuanced, messy and complicated as any of the rest of us, and we shouldn’t expect them to be otherwise. We can be grateful for Patrick Moore’s kindness and great work in popularising astronomy and angry about his views at the same time.

On the BBC #FusionGames Summit, changing the tune about educational games, and Susan Bloody Greenfield

As I sat in the audience at a session called “Learning To Game or Gaming To Learn?” at last Wednesday’s BBC Academy’s Fusion Summit on games in Salford, I became rather frustrated. The session had been billed thusly:

How far should broadcasters use games as a vehicle for learning? Join some of the finest minds in the field as they wrestle with the future of learning games and the controversial subject of gamification.

On the panel were Mark Sorrell (Hide and Seek), Carlton Reeve (Play With Learning), Tom Kenyon (NESTA), Phil Stuart (Preloaded), John Milner (Bitesize, BBC Knowledge & Learning), and they were being questioned by Kate Russell of BBC Click. Undoubtedly a great line-up, yet it was all rather unsatisfying, and also rather familiar.

This is my attempt to explain and unpick this frustration, which is actually a more general frustration with the way educational games are so often treated with scepticism and distrust. This isn’t really a criticism of those involved in the panel, since it was just playing out in the same way that these sorts of discussions always do, and probably are always expected to.

Starting the session with a discussion about gamification didn’t help. Russell acknowledged that it wasn’t going to go down well with some of the audience, and winced as she said it (as I did writing it, ugh). So much has already been said on this subject, whether gamification is just pointsification, whether it mistakes the extrinsic trappings of gaming for the reason why people enjoy games, whether it should be reclaimed as just meaning adding game mechanics to content and so on.

Most people do seem to understand it to mean pointsification, and I can’t see this having any more than limited value. In this context, it’s also a total distraction. This isn’t really what people are talking about when they talk about games based learning in my experience, so it’s a shame it took up so much of the panel’s time. In fact, it’s a shame gamification has taken up so much time on so many conference panels and sessions over the last couple of years, can we possibly move on from this now?

But it was the next few questions that troubled me more. Russell asked “do educational games work?” And “where’s the evidence?” Now, it’s not that I think we shouldn’t ask these questions, but it seems that these questions are all anyone ever asks about educational games. The implication always seems to be that one should be hugely sceptical of such an outlandish and possibly even NEUROLOGICALLY DANGEROUS (more on that later) concept, and that educational games exponents had better have some seriously good evidence up their sleeves if we are to countenance allowing their nutty ideas into our schools and homes.

This makes me weary. It is really such a leap to see that an activity so absorbing as playing a good game, could be harnessed for learning of some sort? An activity whose very essence is about learning, as you must do to improve in any game? Does it trouble people that much that it might be possible to have fun whilst learning? (Note: not that learning in and of itself can’t be fun, but if there was a better way of educating children that all of them would really enjoy, shouldn’t we be really happy about that and keen to explore it further?) Some of the panel did indeed make some of these points, but were rather on the back foot in the face of this slightly negative questioning. Asked to think of evidence off the top of their heads, they were unsurprisingly unable to cite any academic papers in support of their position.

Gran Turismo Academy was mentioned (by Mark Sorrell IIRC) where players were put in real cars, and performed brilliantly, despite only ever having played the game before.  It’s a great example, but in the rest of the discussion little other evidence was mentioned and the conclusion was that there wasn’t much out there. I couldn’t think of any either off the top of my head at the time, but a google search shows there’s a fair bit out there. I’m pretty sure the military wouldn’t be so keen on using games in their training if they didn’t have some good evidence, for example. But maybe all of us involved with games for educational purposes should be better versed in the literature (and there is definitely a discussion about better dissemination about this sort of research and evaluation to be had at some point).

Testing the efficacy of games in learning is always going to be tricky though. For example, testing existing games may show that some of them are poor learning tools, but you couldn’t conclude from that that all games are poor learning tools or that it’s impossible for games to work in this way. And perhaps this is a rather back to front approach anyway for those making the games. Using what we know about the science of learning and good gameplay to make great educational games, with the majority of the testing taking place in formative stages would surely be more effective?

This week I read the Nesta Decoding Learning report, which makes a similar point about starting from good learning principles when creating digital educational tools, and which I highly recommend. I’ve also had a number of discussions recently with people who are indeed working on this basis, which is great, and it may be that this line of thinking isn’t news to many people working on educational games and technology.

So why do we always end up on the back foot in discussions like this? Why is the default position apparently one of scepticism? Responding with examples of the odd good game based learning initiative or stats about how the games industry is huge, broadly equal in terms of gender, not just played by teenage boys in their…snore… doesn’t appear to be making a difference. I hear this defensive tone so frequently and, hands up, have definitely been guilty of doing this myself in the past. Especially to audiences I assume will be sceptical (science types, for example). I promise to stop doing that now. But I have heard this discussion about whether or not educational games can work so frequently in conferences, articles and from people outside the “industry”, and it never seems to move on.

Perhaps we’re doing it wrong. As I tweeted at the time, perhaps all this defensive navel gazing is counter-productive. Perhaps it’s merely reinforcing the impression that the scepticism is right. I think many others in this area, like myself, know in our hearts that there is much potential here; that there is something in games which could work really well for increasing people’s understanding of subjects, situations, and systems. There may well be aspects of learning that games are not good at too, of course, but I don’t think it’s so far-fetched to think they could be very powerful educational tools.

Let’s not allow the likes of Susan Greenfield and her Daily Mail pleasing nonsense about video games set the agenda around this. She might not be able to point at any actual evidence for her claims, but we can. (On another note, if she thinks games have so much potential for evil, they must be powerful things, and therefore have potential for much good too. And of course they change the brain, as any repeated activity will, and this is not necessarily a negative response, as well she knows. But I digress). Let’s not sit back and wait for others to confer respectability on this area, let’s set examples, continue to do great work, and let’s talk instead about how to deal with the real and meaty challenges facing educational games: reaching teachers, funding projects, being heard above the noise, and so on.

On that note, we do actually get into the genuine issues around games based learning at the London Educational Games Meetup group (LEGup). Please do come along to talk games, share learnings, tell me I’m full of crap, or tell me how you think we can change the tune on this issue.

Bring out your games evaluations

Games evaluation page on the museum games wiki
Games evaluation page on the museum games wiki

So, the games evaluation page on the museum games wiki is looking a little sparse. The only thing up there at the moment is the High Tea evaluation. I should be getting a couple of others up there soon, but I’m hoping there are others out there we could add.

Do you have any sort of games evaluation that you’d be happy to have published/linked to from here? It doesn’t have to be a museum game to be useful, any sort of game evaluation would probably be relevant to those developing museum games. It could be a formal document, blog post, presentation. Just something that talks about what you did with your game, what your aims with it were, and what you did to test whether it achieved those aims.

Let me know in the comments or via twitter, where I’m @marthasadie. Thanks in anticipation…

Notes from #SXSWi: Public Lab: Mapping, DIY Activism & Civic Science #PublicLab

I’m attempting to write up every single session I went to at SXSWi. Will be mostly about games, but also how tech can kill, neuroscience, digital anthropology, civic science and more.

A video introduction to the Public Laboratory of Open Technology and Science.

Public Lab: Mapping, DIY Activism & Civic Science

http://schedule.sxsw.com/2012/events/event_IAP9516

This session was an introduction to an extremely interesting organisation: The Public Laboratory for Open Technology and Science (PLOTS). In their own words, they are “a community which develops and applies open-source tools to environmental exploration and investigation”. They provide simple tools and DIY solutions that enable people to collect data about their environment. The applications of this are absolutely fascinating and potentially very important.

As the panel explained, much environmental data is effectively owned by agencies and major organisations. Moreover, the standard tools for collecting this sort of information are expensive and proprietary, requiring serious investment in hardware or software. This creates an imbalance, especially in communities where the environment is under stress as a result of the action of big companies, think of the BP oil spill (more on that later). Those communities don’t usually have the means to collect their own data. PLOTS aims to change this.

They call what they do “civic science”: enabling projects which are community developed and community owned. They were keen to distinguish this from “citizen science”, which they defined as crowdsourcing data that then goes back to a big agency or research team, out of the hands of the people that collected it. The tools they develop are low cost and open source and information is shared under a creative commons share-alike license.

To give an example, during the BP Deepwater Horizon oil spill, people along the Gulf Coast wanted their own monitoring tools to see what was really happening to their shores. They clearly mistrusted the official line, and wanted to see for themselves. So PLOTS provided the information and tools to do this, particularly in the form of aerial mapping, and funded the project via Kickstarter. The rig for the aerial mapping technology is deliberately basic, uses balloons, a soda bottle and a camera, and yet can produce data “an order of magnitude better than Google maps”, they claimed. You can see the rig here, and more information about the Gulf Coast project here.

The aerial mapping tools have also been used by protestors to monitor demonstrations. For example this article on The Verge has pictures of balloon rigs being used by Occupy Wall Street protesters. This and other mapping projects can be found on the Grassroots Mapping site. From the PLOTS website:

Maps are often used by those in power to exert influence over territory, or control territorial narratives. “Grassroots mapping” attempts to invert this dynamic by using maps as a mode of communication and as evidence for an alternative, community-owned definition of a territory. To date, our tools have been used to contest official maps or rhetoric by enabling communities to map sites that are not included in official maps. In Lima Peru, members of an informal settlement developed maps of their community as evidence of their habitation, while on the Gulf Coast of the US, locally produced maps of oil are being used to document damage that is underreported by the state.

Mapping can be hugely political, and as they say, this has traditionally been a tool only for those in power to wield. It’s exciting to see the tools being made available to try and redress this imbalance, but the challenges don’t end there. Firstly, it is important, they said, for people carrying out aerial mapping activity to consider how what they are doing might be perceived. After all, people may be unsettled by unmanned flying objects taking pictures of them if they don’t know what it’s for.

More significantly, though, there is the hurdle of getting your data recognised by authorities. Particularly, I imagine, if it is data that contradicts the official information. They have to be careful about the chain of custody for data to make sure that its legitimate and can’t be called into question. Also, they have to go through a legal process for advocacy which they described as “frustrating”.

But it sounds like they are doing good work, and it’s an inspirational idea. I did a bit of digging, but haven’t turned up any similar projects here in the UK. Anyone heard of any?

The whole session was recorded and you can listen to it here. You can also follow PLOTS on twitter: @publiclab.

Notes from #SXSWi: A New Culture of Learning: Gaming, Tech, Design #newlearn

I’m attempting to write up every single session I went to at SXSWi. Will be mostly about games, but also how tech can kill, neuroscience, digital anthropology, civic science and more.

Trailer for TiltWorld

Heather Staker, Nicole Lazzaro and Scott Stropkay: A New Culture of Learning: Gaming, Tech, Design

http://schedule.sxsw.com/2012/events/event_IAP9607

This fascinating session invited us “to imagine a future of learning that is as powerful as it is optimistic”. It was a discussion on “exploring play, innovation, and the cultivation of the imagination as cornerstones of learning” with three speakers:  Heather Staker, of the Innosight Institute, a “a nonprofit think tank devoted to applying the theories of disruptive innovation to problems in the social sector”, which sounds pretty interesting; Nicole Lazzaro, founder of XEODesign, a “Player Experience Design” consulting company; and Scott Stropkay, a “Lateral-thinker, envisioneer, strategist, designer, prototyper, builder” and founder of Essential Design. You can listen to the whole thing on the link above, and I recommend it, but here are some of my highlights from the session.

Nicole Lazzaro introduce us to her 4 keys to fun, or principles of game design, available as a poster. It sounds like XEO design do a lot of interesting research as part of their design process, including something called facial emotion coding (this?). She also mentioned the work of Clayton Christensen, author of Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns. And she introduced us to the new release from her company, Tiltworld, a iPhone game that aims to both educate players about the environment but also have a direct impact in that points gained result in trees being planted in Madagascar. Games in general were much discussed as effective learning tools.

The panel discussed innovation in the classroom. A good example, apparently, is work being done in Singapore where large scale efforts are underway to change the way kids learn. Another was the Carpe Diem school which offers online as well as campus learning and puts the emphasis on students taking control of their own learning. I liked the concept of teachers as coaches, helping individuals to “get unstuck”. The Carpe Diem school is an example of something called Blended Learning, which is based around using a mix of different learning environments.

Design based learning also came up, which sounds intriguing. It appears to be based around setting students problems to solve, which “empowers kids to think of themselves as creative problem solvers”. That from Scott Stropkay, if I remember rightly. This article seems like it may explain more. Somewhat inevitably, the incredibly successful Khan Academy was also brought up.

But currently, this innovation isn’t making it into the mainstream in the US, as Stropkay pointed out. Heather Staker agreed that it’s very hard to break into the traditional learning system, as it’s so entrenched. She also noted that we shouldn’t forget that actually not all students want to learn in this way.

Lots of food for thought, then, in this session, which opened my eyes to the possibilities for taking education in new directions. Listen to the whole thing here.