Reaching a new audience through gaming

Below is a paper by Danny Birchall and I that we wrote in 2012 about our work with games at Wellcome, and what we’d learned along the way. It was written just before I left, and so it felt like a nice wrapping up of what was a long and interesting process. It was only published in print, though, and I asked permission to post it online, which they granted as long as I waited 6 months. It’s been more like 18 months, but anyway, here goes. I hope it’s interesting and useful to anyone thinking about how they could use games as an engagement or learning tool, or to reach millions of people.

Reaching a new audience through gaming

Martha Henson and Danny Birchall

This article first appeared in Henry Stewart Publications 2047-1300 (2013) Vol. 1, 4 351–356 Journal of Digital Media Management 351

Wellcome Collection is an unusual public venue, based on the Euston Road in London. It is part of the Wellcome Trust, which was founded in 1936 according to the wishes of Sir Henry Wellcome, a biomedical entrepreneur and fanatical collector, as expressed in his will. The aims of the trust are to improve animal and human health, largely through funding biomedical research, but they also have an interest in public engagement with science. This is where Wellcome Collection comes in.

Together with a very fine medical library, the building houses two permanent exhibitions. One displays a small fraction of Henry Wellcome’s original collection of artefacts loosely connected in some way to health, but with this remit interpreted in such a broad way as to include items such as Charles Darwin’s walking stick, a chastity belt, prosthetics, surgical instruments and a Peruvian mummified corpse. The other looks at more modern medical concerns such as genetics and obesity through art and objects such as sequencing machines and diet books. There are also temporary exhibitions and events that deal with a wide range of topics related to life, medicine and science. The most recent, Superhuman, challenged ideas about what constitutes human enhancement, while the next one, Death, will tackle the emotive but endlessly fascinating subject after which it has been named.

The tone of all of this is very much non-didactic: the aim is to engage visitors with ideas, ethics, history and science in a thought-provoking way rather than merely to convey factual information. This approach also underpinned our thinking when it came to games: they had to be genuinely fun and engaging, and the factual content had to be embedded in a meaningful way. The idea was that people should have their curiosity and interest piqued by playing, not feel like they had received a thinly disguised lecture with a game mechanic crudely stuck on to it. It was also felt that the diverse subject matter that Wellcome Collection covers, combined with resources such as the image library and archives, had the potential to form the basis of really great original games.

But why games? First, a well-constructed game can be an incredibly engaging experience. Even those who do not consider themselves gamers will likely recognise the feeling of losing hours to playing Solitaire or Tetris on their computer, or perhaps Angry Birds on their phone. This state of ‘flow’, as it is known, is deep engagement and it is hard to think of many other activities that create this.

In addition, play is, in and of itself, all about learning. One must learn to progress in any game and there is no reason why this learning cannot also be factual. Furthermore, ‘people who play games’ constitute a huge audience, spread all over the world and yet using the same platforms and devices, all brought together by their desire to play. Reaching this audience, one that probably would not otherwise be aware of or be able to come to Wellcome Collection, was a significant reason for choosing this particular route — not because the aim was to convert those players into visitors, but to widen the reach of Wellcome’s activities.

THE GAMES AND THE PROCESS

We started relatively simply. The first couple of games were based on existing successful and enduring game mechanics that it was thought tallied particularly well with the museum’s content. We worked with an agency called Smile Your Eyes Off to create an online Flash game called Memory, which used the old game of Pelmanism as a basis. In this game, the player must turn over two matching cards to clear them away, the aim being to clear all the cards presented (1). The difference in this game is that all the card faces have pictures from Wellcome Images, a vast and varied picture library that contains both historical and contemporary images (2)

Having different themed levels made it possible to convey the breadth of the collection, with around 30 themes available for ten levels, such as masks, skeletons, cells and advertising. The levels also get increasingly difficult as the images get more numerous and more similar. For example, at the end of the game, the player might be unlucky enough to get the forceps level, which is not only very difficult but also very representative of Henry Wellcome’s habit of collecting a large amount of the same type of object.

It was a good starting point — the game was not too complicated and we got to know the process for making games. We found that collaboration was particularly important, it could not have been done without the involvement of image researchers in Wellcome Images and we worked closely with the developers. This spirit of collaboration was to inform the process for future games as well. The game was placed on the Wellcome Collection website, and continues to attract players, with around 60,000 views since launch.

The next project was a quiz engine that allowed the creation of multiple choice quizzes that could bring in images, audio and video in both question and answer (3). This was not particularly original, but was very useful and perhaps most notable for being our first collaboration with Preloaded, an award-winning games agency who focus on ‘games with purpose’ (4). Soon after the quiz engine project, an opportunity came up to do something a bit different and we turned again to Preloaded to help realise the idea.

In the winter of 2010–2011, Wellcome Collection held an exhibition called High Society, which looked at the history of drug use. One small part of the exhibition dealt with the Opium Wars, a shameful episode in the history of the British Empire in which opium was smuggled into China by the British, in order to raise funds to buy tea. It is a fascinatingly murky and under-discussed period of history, but it also has at its heart a central trading mechanic: buy opium; sell opium; buy tea. It was this mechanic that became the basis for High Tea.

In High Tea, one plays as a British trader, attempting to buy and sell opium at a good price and make enough money to keep the people back in Britain in tea (5). The player must avoid getting their ship impounded by the Chinese authorities, who are very much against this trade. The game starts in 1830 and ends in 1839, if the player survives that long, at which point they are informed how many Chinese people are now addicted to opium because of their actions and also that the result of this trade was the First Opium War, which ended rather badly for the Chinese.

Again, collaboration was very important to making High Tea work. It is a controversial subject and it was important to make sure that it was factually accurate. Fortunately, one of the curators of the High Society exhibition, author Mike Jay, was involved from the start. He helped make sure that the facts and names were right, but more than that, he worked with US and Preloaded to develop the fundamental concept, which came out of a brainstorming discussion about the subject matter.

THE SUCCESS OF HIGH TEA AND EVALUATING IT

Preloaded’s distribution strategy for online casual games is to build the game to be self-contained and therefore easy to download and ‘rip’ to other sites (6). At the same time, the company also makes it easily trackable. This means that the game can be placed anywhere and one can still see the play statistics and information, in this case using Google Analytics. The game is then seeded to major portals such as Kongregate, Newgrounds and Armor games, from which, if it shows any promise, it will be immediately ripped to tens if not hundreds of other games sites around the world. On occasion they will ask permission, but usually not. This is what happened to High Tea when it was launched in at the beginning of February 2011.

Initially, a goal was set of around 100,000 plays in the first few months, but this was exceeded in just one day. The game got high star ratings, was featured on the front pages of all the big portals to which it had been, was ripped to around 70 more on the first day and hundreds of comments were left by players. This ‘opening weekend’ was beyond all expectations. After a few days the game left the front pages and plays dropped off, although still to a consistent few thousand plays per day, which continues to this day. To date, it has had over 4 million plays (7).

Why was it so successful? What did the players make of it and the unusual subject matter? Some clues could be found from the analytics. We could see that the distribution strategy had worked and that over 50 per cent of plays were on sites that had ripped the game. Around 47 per cent of plays were on the officially seeded portals and only 3 per cent were on the Wellcome Collection site.

We also saw that social media had driven very little traffic, although share links were placed for Twitter and Facebook in the game. It was therefore fortunate that we had not relied on putting it on its own site and hoping it would ‘go viral’ over social media. We also looked at the Google Trends results for searches for ‘opium wars’ over several months before and after the game was launched. There is indeed a spike that corresponds with the game, but does not appear to have any other news event associated.

Although this is all very circumstantial, a survey that was put up with the game suggested that the game was inspiring people to do more of their own research on the topic. While the analytics are interesting, they often raise more questions than they answer and a more qualitative approach is necessary to find out more. Collaborating again with another department within the Wellcome Trust that specialises in evaluation, a plan was formulated to do this.

First, players were asked to answer a series of questions not only about the gameplay and about themselves, but also about their learning from the game and how they felt about the British Empire after playing it. Based on the answers from this, an interview script was devised and was followed up with seven players in depth over the phone and in a small focus group.

The results were illuminating. The most pleasing result from the survey was that over 50 per cent of players said they were likely to go on to find out more about the subject under their own steam. This was mostly via internet searches, but one player even said that they had read a book about Chinese history as a result. This was a huge success. As the aim had been for the games to engage rather than didactically educate, to generate interest rather than lecture, this measure seems to demonstrate that this had been achieved. Players were asked whether they knew much about the Opium Wars before playing and how the game had affected how they felt about the British Empire. Over 60 per cent were already aware of this history, but in the interviews most said they felt they had learnt something from playing.

Surprisingly, given how shocking it was felt that the actions of the British Empire were, around 57 per cent of surveyed players felt the same about it after playing and around 10 per cent even felt more positively. This indicated poor survey design as there was no baseline to judge any change in opinion from before the game. As such, this was explored further in the telephone interviews. It turned out that players who felt more positively towards the British Empire did so because the game allowed them to empathise with the traders involved by putting them in their shoes. They felt that their decision making was purely economic — they were just trying to put food on the table and were at enough of a remove from the trade’s impact to make ethical factors seem very distant. It was not that players were condoning their behaviour, but understanding it made them more sympathetic. This was very interesting, that even such a simple map-based game could engender feelings of empathy towards characters in it.

The final avenue for evaluation was the commentary left by players on the games portals — many hundreds not only across all the sites on which it was played, but also in blog posts about and reviews of the game (and even comments on those articles), on social media, in YouTube walkthroughs and sites like Metafilter and Reddit. What makes this particularly interesting (although harder to analyse) is that one has no control over it and it can make for unexpected reading. For example, some commentators discussed the maths or economics of the game, not something that had been particularly considered while developing it.

Many commented on the gameplay, which they largely enjoyed, but frequently had suggestions for improving (multiplayer or sandbox modes, ability to buy more boats and so on). A significant number discussed the content in some depth and the commentary was often quite thoughtful and measured. A tiny number, many less than had been thought, took issue with Wellcome Collection supposedly glorifying the actions of the British by making it fun to play.

It was a deliberate aim to try to keep the game fairly neutral in tone to let players make up their own mind, up until the slightly over-the-top results at the end showing the millions of Chinese people the player has caused to be addicted to opium. We had worried that this would be misinterpreted as a pro British Empire game, but most people understood that this was not what was going on here.

In terms of the player demographics, an international audience had indeed been reached, with a large proportion of players not only in North America, but also all over Europe and a large chunk in Brazil. The age range was wide, but more focused on 16–24 year olds than other age ranges, which is a younger audience than is usual for Wellcome Collection activities. One disappointment was a 90 per cent male skew, according to the survey, despite female players liking the game equally well. This is probably a disadvantage of targeting these online portals, which have a mostly male demographic. We are is looking at ways to rectify this for future games, perhaps by including mobile platforms, which have a more even gender split.

WHAT NEXT?

This year has seen production of two very different projects, Axon and Magic In Modern London. Following the success of High Tea, the winning formula was repeated for online casual Flash games with a game about neuroscience inspired by the exhibition Brains. As before, we collaborated with Preloaded, the exhibition curator, who this time was Marius Kwint from the University of Portsmouth, and another domain expert, neurobiologist Richard Wingate from Kings College London.

There was a fruitful day-long session of learning and discussion, during which Richard Wingate gave a crash course in neuroscience and Preloaded in game design. We were looking in particular for some rules within the brain science field that could be turned into the rules of a game. A key moment was when Richard showed a video he had created of neurons growing in a foetal chicken brain (this video can now be seen in the game itself). The neurons are growing towards protein targets and are also competing with other neurons to make the best connections and survive. This mechanic, along with the ‘brainbow’ style aesthetic of bright neurons on a dark background, became Axon (8).

The distribution strategy was more or less identical to that of High Tea and it again met with huge success. As before, we were keen to find out more about players reactions and surveyed them. Although the evaluation is not yet finished and the telephone interviews have not been started, it has been found that over 76 per cent of players learned something from playing the game and, again, a large proportion were moved to go and do their own research after playing.

Magic in Modern London is a very different and much more experimental project (9). It was inspired by a recent exhibition at Wellcome Collection in which an artist, Felicity Powell, took hundreds of amulets and incorporated them into her own work. The amulets, now in the Pitt Rivers collection, were originally collected by folklorist Edward Lovett in the early 20th century. Lovett wrote a book, Magic in Modern London, in which he told the stories that came with the amulets, which were bought or cajoled off ordinary Londoners (10).

This time, the subject matter suggested a different approach. As the amulets were collected around London, the idea of a treasure hunt around the city, where the player ‘collected’ the amulets themselves, seemed natural, and the obvious platform seemed to be mobile. As it happened, an agency called Amblr was building a platform that used GPS to trigger audiovisual content, which was used to great effect in an app called Hackey Hear (11).

In the Magic and Modern London app, the user navigates a 1908 map, overlaid on modern London, to find ‘areas of enchantment’ and collect the amulet at their centre. The aim is to assist an elderly and ailing Lovett to reassemble his collection and the stories that go with it. Once in an area of enchantment, music, imagery and voiceover lead the player to the amulet. Finding the amulet in question will unlock a reading from the book or inspired by the book related to each amulet.

The app has only just been released, although an evaluation will be carried out in the same manner as with the other games. As this is in many ways a more complicated and involved experience for the player, it will be very interesting to see how the platform affects distribution and what players make of it.

WHAT HAS BEEN LEARNED?

As we hope is clear, this has been journey of discovery, during which we have been both surprised and delighted by what has been found. The success with reaching and engaging a new audience could be put down to a number of key factors.

Collaboration has obviously been important, as has the distribution strategy. Picking subject matter that was both intriguing and also lent itself to a game was crucial, as was making sure that the factual content was fully embedded in the game mechanic. Working with people who know how to make fun games is hugely important, as is testing them throughout development to make sure they are well balanced. Evaluation is vital, there are many simple methods available to check what players are learning from a game or how they are reacting to it and it is perhaps the most interesting part of the process.

References

1. Wellcome Collection (undated) ‘Memory’, available at: http://www.wellcomecollection.org/interactives/memory/ (accessed 6th November, 2012).

2. Wellcome Images (undated) ‘Wellcome Images’, available at: http://images.wellcome.ac.uk/ (accessed 6th November, 2012).

3. Wellcome Collection (undated) ‘High Society Quiz’, available at: http://www.wellcome collection.org/whats-on/exhibitions/high-society/quiz.aspx (accessed 6th November, 2012),

4. Preloaded (2013) ‘Preloaded’, available at: http://preloaded.com/ (accessed 6th November, 2012).

5. Wellcome Collection (undated) ‘High Tea’, available at: http://hightea.wellcomeapps.com/ (accessed 6th November, 2012).

6. Stuart, P. (2010) ‘How we publish an online game’, available at: http://preloaded.com/blog/2010/08/16/how-we-publish-an-online-game/ (accessed 5th May, 2011).

7. Birchall, D. and Henson, M. (2011) ‘High Tea Evaluation Report’, available at: http://museumgames.pbworks.com/w/file/fetch/44614076/HighTeaEvaluationReport.pdf (accessed 15th January, 2013).

8. Wellcome Collection (undated) ‘Axon’, available at: http://axon.wellcomeapps.com/ (accessed 6th November, 2012).

9. Wellcome Collection (undated) ‘Magic in Modern London’, available at: http://www.wellcome collection.org/explore/play/magic-in-modern-london.aspx (accessed 6th November, 2012).

10. Lovett, E. (1925) ‘Magic in Modern London’, printed at the Advertiser offices.

11. Hackney Hear (undated) ‘Hackney Hear’, available at: http://www.hackneyhear.com/ (accessed 6th November, 2012).

Henson and Birchall

356 Journal of Digital Media Management Vol. 1, 4 351–356 [1] Henry Stewart Publications 2047-1300 (2013)

The future of games in museums: what should we be doing?

A confluence of related projects and talks has got me thinking about where games in museums should be going in the future. There have been some notable successes in museum games to date, and some failures. Where to go from here? Here are some assembled thoughts on the types of games and game design practices I would love to see more museums exploring.

Collaborative games

Now, I love competitive games, but not everyone does, and competition can be off putting and disruptive in, say, family situations (I’m sure you all have stories about the game of Monopoly that ended in tears). Collaborative games are perhaps more suited to the mixed audiences and interests that are represented by museum visitors.

For example, Spaceteam is absolutely one of my favourite games of the last few years. It manages to be ridiculous, hilarious, breathlessly exciting, social and visually striking, all at the same time. Go find 3 other people with smartphones or tablets and have a go, it’s hard to explain. What’s particularly brilliant about it is the way that players become instantly collaborative through the mechanic of needing to convey information to each other rapidly. There is no competitive element, you are all working together to stop your spacecraft coming apart at the seams.

There are also lots of board games that work this way, (I tried Forbidden Island the other week, which is a good example), search for co-operative play on Boardgame Geek. The UVA Bay Game, a team based sim about sustainability, is another interesting case, and seems to have resulted in genuine behaviour change as players realised they would have to work together to solve the issues, both in the game, and in real life.

Discussion based games

Where discussion is happening, thinking is happening. Most museums want to be in some way thought-provoking, and recognise that seeing people deep in conversation about the objects or display they are looking at is a good sign. But many people don’t feel entirely comfortable sharing opinions about art, or history, or science, feeling they lack knowledge or will say something that will be ridiculed. Or, perhaps, it isn’t part of their normal group nature to have discussions of this sort. This means if you want to encourage discussion, you may need to scaffold it in some way, and games can be great for this.

One of the most interesting game experiences I’ve had was playing Liliane Lijn’s Power Game at the ICA a few years back. The atmospheric set helped, but at it’s heart it was a sort of poker game (actually based on Chemin de Fer, I believe) where you had to make the case for why the word you had been given was more powerful that another. Everyone at the table would then vote, and if you won, you got the chips in the middle. It’s more complicated than that, in truth, but what it means is that you end up having to make arguments for abstract concepts you wouldn’t normally think about (i.e. is “war” more powerful than “love”?).

An online discussion game which I’ve enjoyed playing is the Foresight Engine. In this, there a future scenario to which players must respond by playing different types of card that discuss the outcome or effect of that scenario. You get points for each response you play, for responses to discussions that you start, and for having your response highlighted by a moderator/judge. It is effectively a collaborative crowd-sourced future forecasting tool and it’s been applied in all kinds of real world situations (they used it in Christchurch to help citizens work through the implications of certain scenarios for their earthquake damaged city, for instance). I’m certain a card based game that was a mix between this and Power Game could work really well in a museum setting, perhaps as part of events.

Rapid, casual games

Fast, casual games with the right mechanics can be super addictive. This is why I don’t open Bejeweled Blitz unless I know I have at least half an hour to lose to it, even though each game only takes a minute (and is totally brainless, yes, I know, I don’t care). They can also manage to convey a simple point effectively, as we found with the Axon game at Wellcome. In Axon, players click to move a neuron forward via protein targets, with games lasting sometimes just a few seconds, and in doing so (as we found via a survey of players) they learn something small but interesting about foetal brain development, get a sense of the aesthetics of modern brain imagery, and have their interest piqued sufficiently to follow up with a visit to Wikipedia.

Also look at super fast task based or point and click based games such as Wario Ware or McPixel. Each level takes seconds. This format can work really well, be exciting even if the task is incredibly unsophisticated (and therefore work for a range of abilities) and can also be fun to watch. Try also Tenya Wanya Teens if it comes to a place near you. It’s physical, quick, and fun for spectators as well. Oh and whilst we are talking about crazy fast games that are also hilarious, if you haven’t tried QWOP, or any other of Bennett Foddy’s games, you really must.

The temptation is for museums is to create games that convey a lot of information (because we have so much interesting content, so it’s understandable), or that are a bit worthy. But those games are really really hard to do well, and risk turning off a lot of people. For in-gallery games, museum visitors may be time conscious, or assume it will be serious, or just want to watch, and fast, funny games could be the answer.

Pervasive games

Museums are great spaces. And recently Lates events have become very popular. Many of these are already using pervasive games to get people interacting with the spaces in new and different ways. Hide and Seek’s Sandpits do something similar, as at the National Maritime Museum a while back. Capture the Museum (Thoughtden and National Museums Scotland) is a team based pervasive game that also uses smartphones to deliver puzzle based challenges (if I’ve got that right, I haven’t had a chance to play yet).

So this isn’t a new idea, but it would be great to see museums do more of this. These sorts of games are often easier to run and develop (and therefore cheaper) than digital games but are frequently overlooked. If you can turn them into a card or board game or just a set of instructions, you could also distribute online which helps it reach a bigger audience.

Yes, many museums don’t want people running around and yelling during regular museum hours, but not all games have to involve this. At SFMOMA, their ArtGameLab crowdsourced games to be played in the public spaces that were more sedate, but still fun (e.g. go around the museum critiquing artworks using only your facial expressions).

Card/board games

Having mentioned card and board games above, I think it’s worth highlighting them. Are there any examples of existing museum card or board games? I wonder why not.

Locative, mobile games

Taking the game out of the museum is not new. We tried it with Magic in Modern London, Tate did Magic Tate Ball (more toy than game, I guess, but still), and others have released games on mobile. But it feels like the potential of mobile is not yet being fully exploited, particularly around location, and the relationship between objects and the landscape, whether urban or rural.

Developing bespoke apps of this type can be expensive and difficult (but if you get it right, wow). But there are other platforms you can use, I recently looked into using the SCVNGR platform for a museum project and was disappointed to find out that it was now unsupported, which is a shame, because it would have been perfect. But you could use other trail apps to add challenges or mini games, or work with Junaio and its AR functionality to create something playful. In researching SCVNGR I found another ex employee has created a similar platform called Edventure Builder, which could be worth exploring.

Online, casual games

This has probably been the most successful approach to date. Tate, Science Museum and our Wellcome Collection games High Tea and Axon have all demonstrated that online casual games distributed to portals can be successful in reaching large audiences, and having an impact in terms of learning (read the High Tea evaluation here). The Science Museum had a rarer success with Launchball, a game that was only on their website yet reached a large audience (via a post on Reddit, if I recall correctly), but in general distributing to portals seems to be the most effective approach.

What this means is that there is already a good existing model for doing this. Use it! The potential audience is in the millions. And if you develop in a way that your game can be released to mobile as well (using Unity, say), it’s even bigger. This approach isn’t the cheapest, and you need to work with real game design pros, but it can be very effective. When you look at the value of High Tea, it was working out at about a penny per play, and had a genuine impact on players in terms of learning and thinking about the subject matter.

Games based on 3^rd party platforms

I’ve already mentioned this in terms of locative games, but there are many game development tools out there that are relatively simple to use (I list some in this previous post on making games on a budget). There are also tools that were designed for other things but could be repurposed to playful ends.

I saw an online production recently (the Nightvision Experiment) that frankly didn’t work for me in terms of plot or acting, but used a clever mechanic of delivering the entire story via twitter and youtube. And The Dark Room used just Youtube and it’s annotation function to create a smart, funny sort of video text adventure. There is no reason social media can’t be used to create games, or interactive experiences, and in fact Liliane Lijn was at one point running her Power Game over twitter as well.

Console game partnerships

I don’t think anyone has tried this yet in museums (although the Wellcome Trust did try it as part of their broadcast and games funding work), but I’m sure there is potential here. Obviously museums are unlikely to be able to afford to develop a AAA console game of their own, they cost millions. There are increasing numbers of indie devs producing games for consoles (especially since Unity can output to several of them) for lesser budgets, but still, it’s expensive.

However, museums do have content. They have stories, settings and objects, and they have all kinds of experts. Many are respected global brands. Might a canny partnership be possible between a console game producer and a museum? Perhaps where a museum can do and fund some development work into the factual elements (or even non factual elements), provide some information, setting, or idea inspiration?

Evaluation

I do go on about this, but this needs to happen more. I’ve just read a fascinating evaluation for a Science Museum game (that I hope will be shared soon) and it was *really* illuminating. I wish I’d seen it earlier. It made me think that we’ve all done enough games now that there should be a pretty good body of knowledge about what works, and what doesn’t, and we should be building on this. Some of this information does get shared at conferences and so on, but it still feels like a lot of museum game development (and other digital development, in fact) happens in the dark.

We can copy other apparent successes, but without knowing in depth information about the player responses, we may just be repeating an empty exercise in gathering hits.

Games people

Museums need to focus on working with people who are good at making games, and not get so hung up on the platforms or technologies. This essay by Suzy Glass isn’t about games specifically, but it could be, and it is absolutely spot on about this issue.

Museums also need to be working with those people right from the start of a project, not waiting until they’ve put together an extensively scoped funding proposal without any games expert input. And that means hiring someone, freelance, or full time and paying them from the start. It will be worth the investment. I’d also like to see less of small games and other digital agencies time being wasted by having to jump through multiple time consuming (and therefore expensive) hoops as part of the procurement process, when they have a worse than 5 to 1 chance of getting the commission.

Those are my assorted, random, thoughts, please feel free to add yours!

Electronic visualisation and inspiration: the EVA conference 2013 #evalondonconf

I spent Monday and Tuesday of this week at EVA, the Electronic Visualisation and the Arts conference for which I was lucky enough to get a bursary to attend. It’s a multidisciplinary event, which describes itself as being about “electronic visualisation technologies in art, music, dance, theatre, the sciences and more…” It was certainly diverse. Some presentations were technical or academic, others were more practical, and they were on a range of subjects from a wide variety of institutions and individuals. “Electronic visualisation” covers an lot. I left quite inspired, if wondering on earth to do with all the ideas now whirling around my brain. I highly recommend it.

Here are some of my highlights from the two days I spent there. For the full proceedings go to http://ewic.bcs.org/category/17656#.

Creating Magic on Mobile

My paper kicked off the day, so let’s get that out of the way. I presented (with Alex Butterworth of Amblr) on “Creating Magic On Mobile”. You can read the full paper here, which discusses our experiences creating the Magic in Modern London app for Wellcome Collection (when I worked there as Multimedia Producer). It’s a geolocated treasure hunt set in Edwardian London so the paper covers the use of GPS, audio-visual content and a map based interface to create a historical semi-fictionalised narrative.

It also contains a call for more imaginative use of mobile in cultural organisations, and shows how we attempted to do this with Magic in Modern London, but also some of the challenges we faced along the way (such as trying to overlay a 1908 map onto google maps). It was a pretty experimental project, but also very ambitious, and we learnt a huge amount along the way. It played a large part in my drive to help set up ME:CA (Mobile Experiences: Cultural Audiences).

Keynote: Steve DiPaola “Future trends: Adding Computational Intelligence, Knowledge and Creativity to Interactive Exhibits and Visualization Systems”

Steve DiPaola (a “computer based cognitive scientist, artist and researcher”) gave a fascinating keynote that covered several of his projects. The notes from his talk are here http://dipaola.org/eva13/. These included.

A beluga whale pod interactive simulation at the Vancouver aquarium being used both in gallery and as oart of scientific research. This installation allows visitors to observe particular behaviours and test out scenarios.
Teaching evolution through art: using “evolved” artworks to help people really get what evolution means, particularly challenging in America, where 60% of people don’t believe in it (apparently). This was done as part of Darwins 200^th birthday celebrations.
Analysis of Rembrandt’s works to understand his compositions and techniques and their affect on the eye path of the viewer. This work came to the conclusion that Rembrandt had an impressive understanding of vision science 200 years before anyone else.
A project to create life-like (ish) 3D avatars that people can speak through to address people remotely, and perhaps anonymously (e.g. scientists having a digital conversation with the public from another country).
An analysis of Picasso’s “Guernica” 40 day period of creativity. This mapped all the works created during this prolific period onto a timeline to try and understand the nature of his creative process, how ideas evolved, and how he was influenced by other works.

Timelines

Timelines were also feature of other talks, such as this about “representing uncertainty in historical time”, I didn’t see this presentation but did see the demo later, which looked at turning the works of a composer into a timeline that mapped them against their first performances. Can’t find a link for this, but I was impressed, as with the Guernica project above, at the way a putting data into a timeline can reveal new insight.

More museums and mobile (with added AR)

There were a couple of other museum mobile projects, both of which used augmented reality (AR) in different ways. Timeline Newcastle looked interesting, and quite similar to Streetmuseum in overlaying archive photos onto the modern city (if I understood it correctly). I was particularly interested to see that the Smithsonian Natural History Museum have a very ambitious app in the works called Skin and Bones. It will focus on 14 objects, giving different levels of engagement with each, based on their framework for visitor preferences called IPOP (ideas, people, objects and physical). So for visitors who want to hear about people, they have a meet the scientist option, for physical – an activity, for ideas – an exploration of a scientific concept, and for object – a detailed study of the skeleton some of which used AR to overlay the skeleton with an animation.

I was interested in their concept of using it to raise “visual literacy” ie, helping visitors to understand and interpret what they were seeing and in doing so increase dwell time in the galleries. They had recognised that users were spending little time in this gallery, except for at a few “hero” skeletons, and surmised that people weren’t finding it interesting because they didn’t know how to make sense of what they were seeing. So in the app, for example, it will show you how a particular venomous snake’s jaws hinge back into their mouth and how you can see that in the skeleton.

They have an impressive (and expensive sounding?!) user testing plan. They are creating two apps, one without AR and one with to see how that affects visitor interaction, and will be using a beta app path analyis tool called Look Back to help with this (although this feature is also apart of GA mobile analytics as well).

Keynote: Linda Candy “Creativity and Evaluation: Supporting Practice and Research in the Interactive Arts”

This was an impassioned call for making evaluation a key part of the creative process in a keynote from writer researcher Linda Candy. It’s not just about evaluating impact, but also creating new knowledge and new works, she said (but impact also still important, I would add!) and should be fully embedded in practice. For artists creating digital interactive works there are also usability issues they should be testing (do people understand how to interact with it, for example? Is it within reach of children/wheelchair users?).

Obviously, as a huge fan of evaluation, I am very much down with this. Evaluation isn’t just about fulfilling the tedious criteria of funding bodies, but is more about understanding and improving your own work. The evaluation work I’ve done on games has been some of those more interesting and thought provoking work I’ve done.

Stop, my brain is full

There was a whole other day of this that I missed, probably for the best, as by the end of it my head was spinning.

Why I’m fed up with digital projects (and why I’m not): a rant

I’ve felt a little rant bubbling up in me over the last few months: a sense of disquiet about digital, a jaded annoyance about wasted time and resources and opportunities squandered. Today I was reminded about an old project that was the epitome of digital idiocy, one of those thoughtless knee-jerk “we must have an app!” projects that make me want to throw a toddler tantrum, kicking and screaming “but who is it for?” until someone agrees to at least do a bit of audience research or string together a minimally viable set of objectives. And that reminder seems to have brought it all to the surface, so here goes.

I am fed up of seeing people and organisations produce digital rubbish: poor apps, clunky games, badly designed microsites and other half-arsed online, mobile and technological systems and whatnots. I am fed up of people who are smart about digital, who think about users, who ask the right questions at the start, who embrace technology without fear and understand how to apply it, being over-ruled by people who’ve just bought an iPad for their kids and now assume that everyone has them and that this is all the justification they need to insist upon spending £50k on a new app for their organisation (an app for who?! For what?!). Moreover, I am fed up of projects that fail being brushed under the carpet, with no evaluation or debrief that enables everyone involved to learn from mistakes and build something better the next time.

It makes me sad to see money and time being wasted, when it’s usually avoidable, and even more so if nothing is then learned from this. It makes me especially sad, because the technology we have available to us today is AMAZING. My phone is able to do astonishing things and the computers I use have incredible firepower (I still remember how exciting it was to get a floppy disc drive for my BBC micro, now look at them!). It feels like there is so much potential and so much more that the technology we have could allow us to do.

It’s not like there aren’t enough examples of brilliant products, apps, games, and other applications of these technologies out there to learn from. Too many to list. But many many many more that have been an utter waste of everyone’s time from the beginning because they just didn’t ask some basic questions about who and what their product was for, and test it with that in mind along the way. Isn’t that a bit sad?

I know it’s not easy. That there is a degree of luck in rising above the huge amounts of competition online or on on the app store, that things beyond your control can go wrong during the development of a project, and that there are some times you need to go on your gut instinct, and that’s not infallible. Goodness knows I’ve made every kind of mistake creating digital things over the years, as have most people working in this constantly changing field. And I will continue to make mistakes, as will most people etc.

So I’m not claiming there is a perfect process, but there are better and worse ones, and there are definitely massive howling alarm-bell-ringing clangers that should be sending up the red flag right from the start. The “we need an app, never mind who for”, the “we’ve got all this content, let’s just stick it online, people will come to us”, “let’s create a whole new portal for something that already exists”, and so on. And sometimes it feels like we aren’t moving on from this, these whack-a-mole stupidities keep popping up over and over again and just will not die.

Why is that? In a recent conversation with a fellow person-who-works-somewhat-indeterminately-within-“digital” we discovered we’d both been feeling this slight ennui with the field. This sense that it wasn’t meeting its potential, that a lot of rubbish was being made, and that it was hard to see how to make things better. We wondered if it it might be that digital is constantly being driven by the hardware makers, by Apple and console or computer companies and that their vested interest in creating new stuff for us to buy means we’re always just playing catch up. Why do we let this dictate the pace? Where is the chance to take stock and develop best practice?

My experience as a freelancer over the last few months, applying for funding and responding to ITTs, has also been eye-opening. The systems and funding in place for making digital stuff are really broken. If you were trying to design a process that would result in crappy digital products, you’d make people (who may not be particularly digitally literate) scope everything out and pin it down before they’d ever been able to build anything or ask their audience about it, only then engage an agency who are the real experts but can’t change the scope, and then make the project team go through a month long change request process if they want to develop the project in a different direction from the scope based on the evidence as they start building and testing things. Which is exactly the process in place in a frighteningly high number of places. The old models for procuring new infrastructure etc just do not work for digital projects, and it’s hamstringing them from the very start.

Plus money, of course. Not enough money, or not tailoring the project to the actual budget available. I’ve had this rant elsewhere. I seem to spend a lot more time these days telling people not to make a digital thing. Would it work just as well as a piece of text? As a printout? As a physical activity? Or a really simple website? Don’t overreach if you just can’t afford it, do something straightforward and proven.

But at the same time I love working on and seeing others working on brilliant digital projects. The reach you can get, the amazing response in terms of both scale and thoughtful quality, the sense of shared global experience, the playfulness and the utility, when done well. So, how do we do more of these?

Evaluating online video

Whilst I was Multimedia Producer at the Wellcome Trust I began a programme of evaluation for our online videos. I left before I was able to see most of the results from that, but in this post I’ll share the plan of action I had for the evaluation, which used a number of different approaches. I was building upon work that Danny Birchall and I did in games evaluation, using some of the same methods, but evaluating video presents a somewhat different set of challenges. If you have any thoughts about other techniques one could use, or ways of improving on my methodology, I’d love to hear that.

Why evaluate videos?

At the Trust we (me and the other Multimedia Producer, Barry J Gibb) were creating short documentaries about the research funded by the Trust, educational films, and so on. The audience varied, but was usually considered to be an interested lay audience or a GCSE/A-level student. Our video dissemination strategy was fairly straightforward, we posted them on our site (where they frankly got a bit lost) and on YouTube (which I always considered to be far more important especially because of the much greater chance someone would find them there, and the ability to share).

I set the YouTube channels up in early 2009, but had no formal plan of evaluation (oh! to turn the clock back) and simply kept an ad hoc eye on analytics and feedback. The analytics were sometimes interesting, but commentary on YouTube, you may have noticed, is rarely that worthwhile. When we got lots of it, it was generally polarised and sometimes unpleasant. When we got none, which was usually the case, it was like putting the videos up to the sound of tumbleweeds and crickets. We had no real idea who was watching our videos, whether or not they liked them, how they were using them, and therefore it was hard to know how to improve them.

That was the key, for me. How could we develop a strategy for the future when we didn’t really know what had worked or not in the past? But all was not lost. A lot of information was just sitting there, in the form of analytics, and to find out more, we just had to ask.

My approach

I split the project up into a number of different areas:

1. Analytics

2. Staff feedback

3. Audience feedback

4. External organisations

Analytics

My plan was to track several paramenters for the lifetime of each video, ideally broken down into years. I had decided to go for calendar years rather than years since posted to be able to compare videos across the same period, but was somewhat in two minds about it. Youtube analytics offers information about views, traffic sources, and playback locations, which I had planned to include, and demographics and audience retention, which I hadn’t. Demographics on YouTube are only tracked for those people who are signed in and have added those details to their profile, so are largely worthless. I guess they may be less so now that it’s owned by Google and more people are likely to be signed in, but I still wouldn’t place much stock in them. The audience retention graphs are interesting, but hard to compare.

The analytics presented the biggest challenge for the project, though. The first challenge was that the analytics were split between Google Analytics for the videos on our website and Youtube’s own analytics. The latter are much more restrictive than the former, though they give more detail on the attention patterns, for example. GA depends very much on how it’s been set up for video, but thankfully when GA was set up for the Wellcome Trust site at least someone had the sense to track the number of actual video plays as events, rather than rely on pageviews. But still, marrying GA and YouTube together wasn’t going to be easy.

The second issue was that getting data out of YouTube isn’t easy. You can download spreadsheet reports for one parameter over one time period at a time, but when you have well over 50 videos and want to try and track several parameters over several years, well, I didn’t fancy doing that kind of epic legwork. There is a solution, but it’s not ideal either. Google have an experimental API for YouTube analytics, and our IT department were as I left on the case of trying to write an application that would use it to built reports.

I did have a couple of initial findings from my more ad hoc monitoring of stats over the years, though these aren’t going to be hugely robust and I had hoped to interrogate them further with more data.

On YouTube, most people were arriving after searching for relevant keywords *in Youtube*, (ie, not through Google organic search), or from related video links. The latter were definitely responsible for hits on our most popular videos, that were coming up as related links to other high profile videos from others.
Many more people were arriving directly at video pages rather than the Channel page.

Staff Feedback

The Wellcome Trust has several hundred employees and, as with our external audience, we didn’t know what all of them thought of our video output. We didn’t even know if they were all aware of it, that they could use it or even commission their own films. Their opinions and needs were also very important to developing our video strategy. I put together a survey in Survey Monkey, had it checked by our UX expert (always worth doing) and posted it on our intranet. I wasn’t around for the results of this one.

Audience Feedback

This was the bit I was most interested in, qualitative feedback from our audience. For our games evaluation work we had put up a survey at the end of the game, followed up with telephone interviews, and also looked at community commentary. As we’ve established, for Youtube the latter is not particularly useful, and we also didn’t have comments open on our site. But the survey was still an approach we could take, except instead of one game, we had ninety three videos. And instead of several hundred thousand plays over the launch weekend, most videos would have a few hundred views over their lifetimes.

This meant that we would need a much longer data gathering period as only a small fraction of people will answer a survey. As always, we ran a prize draw with this to help the numbers, but even so, it seemed wise to assume it would be six months (at least) before we had enough responses to come to any conclusions.

I drew up some survey questions, with the help of colleagues and again had them checked over by Nancy Willacy, our UX expert, to make sure they were understandable and that we were asking the right questions. There is lots out there on survey design, including some points from our own experience in this paper on games evaluations that I wrote with colleagues at Wellcome, The Science Museum and the National Space Center. Being very clear about what you are trying to find out is very important, it’s easy to creep beyond that and try and find out everything, but nobody wants to answer a huge survey online.

My intention was to add the survey as a link on our site next to all videos, and also to put them up as linked annotations on YouTube videos. I did the YouTube links myself before I left, which was a pretty tedious job. You can see an example at the end of this video. I have a feeling that you may have less options in the annotations if you aren’t a YouTube partner, and that adding links like this might not be possible. I guess you’d have to add one in the text below, but that’s not ideal.

External organisations

Finally, I wanted to know what other people were doing with video at similar organisations (and also what they thought of our output, if they’d seen it). This was to be much less formal, simply identifying other organisations and writing to them with a list of questions, at least in the first instance.

And you?

I hope some of that is interesting or useful, but it would also be great if other people share their experiences of evaluating video. What do you do in your organisation? What could we have done better here? Even better, does anyone have any results they can share?

Bring out your games evaluations

So, the games evaluation page on the museum games wiki is looking a little sparse. The only thing up there at the moment is the High Tea evaluation. I should be getting a couple of others up there soon, but I’m hoping there are others out there we could add.

Do you have any sort of games evaluation that you’d be happy to have published/linked to from here? It doesn’t have to be a museum game to be useful, any sort of game evaluation would probably be relevant to those developing museum games. It could be a formal document, blog post, presentation. Just something that talks about what you did with your game, what your aims with it were, and what you did to test whether it achieved those aims.

Let me know in the comments or via twitter, where I’m @marthasadie. Thanks in anticipation…

The High Tea game: An education

High Tea instructions: Buy opium, sell opium, buy tae — The instructions for High Tea

Have been meaning to write something up about our (Wellcome Collection and Preloaded‘s) High Tea game for some time. However, Danny Birchall and I have now published our evaluation report, which pretty much says it all. It’s been an incredibly informative process, which I hope to repeat for all future games. Some headline stuff from all the analytics, surveys and interviews:

High Tea has had over 3 million plays
Users found it fun and educational, and liked that it was based on historical fact
Over 50% were inspired to go find out more about the subject matter after playing
Distributing it to just three or four portals in a rippable format meant it is now on hundreds if not thousands more, but still trackable
Social media was useless for bringing people to the game

And lots more, all in the the evaluation. Whilst you are there, please sign up and contribute to the Museum Games wiki. We hope the latter can become a really valuable resource for all of us working in this area.

Tag: evaluation