Everyone knows data driven design is important. Zynga do it, so it must be good. But it’s not a golden ticket. Sometimes a design is just bad, and as we say in the North of England, “You can’t polish a turd”. Which translates as: despite your best efforts there are some things you just can’t fix or improve.
This post is about how to not polish a turd. Or more accurately, the risks of relying on AB testing, and how find inspiration for great game designs.
I like MY sandwiches
I like sandwiches. Not all types of sandwiches, and not all fillings. But I always like the sandwiches I make myself.
You see, when I make a sandwich there’s no salt beef, no pickles, no mayonnaise, no coronation chicken, no fatty ham, and definitely no egg. That’s just a few of the things I don’t like (the list is long).
If you were to make me a sandwich you’ll probably make me a sandwich I don’t want. Not that I’d tell you, I’m too polite. I’d eat the sandwich, but I’d remember not eat a sandwich you’ve made again.
That’s the problem. It’s much easier to make things for ourselves – I know what sandwich fillings I like. The same is true of games. I know what I like (Monkey Island, Fallout, and Deux Ex) so I know how to design a game I want to play (and quite probably other 30-year-old men like me want to play).
Making games for kids isn’t so easy. I am not an 8-year-old girl. I never have been, and I don’t believe in reincarnation, so I never will be. So how do I know what an 8 year old girl will like?
Don’t trust your instinct
Every day at Dubit (my company) we make games for 8-year-old girls. Strictly speaking we don’t just make games for 8-year-old girls, we make games for a wide range of kids. Nevertheless, the problem is the same, we are not kids any more – and we’re not allowed to employ kids (something about labour laws). So we can’t just design games that we want to play. We have to design a game that someone completely different that ourselves will enjoy.
Designing for kids means you can’t rely on your instinct – what you think will be fun – you’ve got to involve kids in the design. Instead, watch them play, and test your concepts. But you don’t just ask them what they like. They’ll lie (mislead is a more pleasant way to say it).
This is a really important point. Kids will tell you what they think you want to hear. The only way to know what kids actually think is to observe what they do. Online this means using data, AB tests, and analytics to see how they use your game, where they get stuck, and what they like.
Zynga are famous for this type of data driven design. At any point in time they’re running 500 different experiments. Each experiment is an AB test – they compare two versions of a design to see which performs best.
Trust the data
These types of tests don’t lie. If homepage A converts more visitors into players than homepage B, then homepage A is the better homepage.
In a earlier post, 7 month gamble: 2 months design, 5 months development, but is the game fun?, I shared some of the quick and dirty data driven techniques we’ve used to test how kids react to different parts of a game design without having to build the whole game.
The bad news, while the tests I described don’t lie they can mislead. The good news, there are plenty of resources explaining how to design tests that don’t mislead. There’s little I can add.
But running tests is easy, what I care about is knowing I’ve got the best homepage, the best game design, the best theme, or the best narrative. Not the best out of the two, three, or four tests. But the very best.
While the examples that follow are for a homepage design, the same process applies to game design, theme, and narrative using the principle explained in this post.
Watch the pennies
A typical AB test looks like this: design a homepage, then make a change. Usually a small change. Maybe change the sign up button from “Register Here” to “Play Now”. You let that test run for a while. Probably the second one wins, then try another variation: “Adopt Now!”
This continues with gradual incremental improvements. Over time all these little changes add up to a big improvement – it’s like watching the pennies so the dollars will take care of themselves.
Except, more often that not, it doesn’t work this way. Maybe at first you see improvements, but pretty quickly it gets hard to tell if the changes are making any difference at all!
I wouldn’t have started from here
The problem here, is that you’ve found the “local maximum”. Which is just a fancy way of saying you’ve made that original design as good as it can be.
All those little improvements have added up to a better homepage. But once you’ve found the local maximum any other changes wont improve performance, indeed, more changes can only result in worse performance!
The real question is whether you’ve found the “global maximum” – is the design the very best of all possible designs?
Usually the first design is not the best design, and you won’t get to the best design with incremental improvements. More often than not, the global maximum design is very different to the one you start with.
Yet data driven incremental improvements are just that, small improvements. And small improvements don’t lead to big design changes, because big design changes can (at first) lead to worse performance. But AB tests always choose the design with the best performance. The idea is illustrated with the chart.
To find the global maximum you’ve got to start with a design that has the potential to be the best possible design. Which reminds me of the old Irish joke:
Paddy stopped cutting the hedge as the big car drew up beside him and an English visitor enquired,
“Could you tell me the way to Balbriggan, Please?”
Paddy wiped his brow.
“Certainly, sor. If you take the first road to the left? no still that wouldn’t do? drive on for about four miles then turn left at the crossroads? no that wouldn’t do either.”
Paddy scratched his head thoughtfully.
“You know, sor, if I was going to Balbriggan I wouldn’t start from here at all.”
And that’s the problem with the first homepage design. If we want to find the global maximum we shouldnt have started with that design at all!
Where to start from
To find the best starting point is quite simple. Make several very different homepage designs, layouts, color schemes, and positioning statements. Then test those design. No small incremental tweaks until you’ve found the most promising starting points.
The problem is this advice is simple to write, but much harder to execute. How do you come up with a great starting point? Where does the inspiration for a great design, a great game, or a great story come from?
It comes from the extremes.
Inspirations at the edges
At Dubit we believe insights come from extreme users and not from center of the bell curve. There’s little inspiration in the average, there’s a huge amount of insight at the edges.
What does this mean for virtual worlds and games?
If it’s an existing children’s brand then start by finding the super fans, find the kids who run the fan sites. Talk to the adults who love the brand. These people are the extremes, they represent 1% of the target audience. But talking to these people is going to tell you much more than the kids who don’t really care about your brand. I’m always surprised by the super fans.
Then find the kids who hate you. The ones who take to vocalise their hate. They’re just as passionate as your super fans. You want to find out why they hate you, not so you can minimize why they hate. Knowing why they hate helps you build a game the fans will love. Because it’s always better to build a game that polarizes than a game that excites no-one.
The same principle applies to the game mechanics. Once you know who your super fans are, then find out what kind of games they play. What do they love, what do they hate? Find the extremes.
Maybe the love pet games. There are already dozens of different pet games online, on consoles, on mobile devices. What’s going to make your game great?
Again start with the super fans; the kids who run the fan sites for other pet games. Which games have the most passionate fans? Why do they love that game? Try asking them how they’d describe the game to their friends. You’ll be surprised. It’s usually the little things.
Remember to observe them. How they play the game. If its a cartoon then watch when they laugh.
Overtime, you’ll start to build up a vivid picture of your fans and haters. You’ll see the little things that are important, the quirks, and the insights that you can turn into designs, themes, narratives, or mechanics. Now you can start to design, now you know what to design. Only then is it time to go back to data driven design.
Almost by definition data driven design tests against the centre of the bell curve – the “average” people. To be financially successful you need the middle of the bell curve to love the game, but you’ll never find inspiration in the middle.
In a nutshell. Start with the passionate fans, be those fans of your brand, fans of the genre of game, or fans of your genre of narrative. Find the super fans. Talk to them, and observe them. And only then test what you’ve learnt with data driven design.