Split-testing in Game Design

I’ve been reading about the practice of split-testing lately, which is essentially Darwinism for content such as advertisements or web pages. For instance, say we want to increase the conversion rate we get on visitors to our websites actually creating characters and getting involved in the games themselves. We could a) use our best guess at what will be appealing, b) solicit feedback from potential users via a focus group or some other method, or c) use split-testing to actually find out what users prefer. (I’m sure there are other options, but I am not really a marketer.)

Option ‘a’ is what most web page developers default to, and that has previously included us. Option ‘b’ is fraught with difficulties. Asking someone what he or she prefers is a very poor method of discerning what the person actually prefers when push comes to shove. Putting together a proper focus group is also not without expense.

Split-testing, on the other hand, is perfect and is simplicity itself. Stick two web pages side by side, direct visitors to them in as random a fashion as you can manage, and then watch the results. If (and almost certainly when) a statistically significant difference appears in the conversion rate between the two, discard the lower-performing page, do a new page, and repeat over and over. Iterate your way towards perfection.

Iteration is nothing new in virtual worlds, of course: many of the smaller/more agile ones iterate on literally a daily basis and even a behemoth like World of Warcraft manages it, albeit much less frequently. The problem is, without any control to compare your newest iteration against, there’s no way of knowing if your latest batch of changes is actually improving the user experience. I’m sure at least some people in SOE believed that the NGE expansion for SW:G was a positive iteration.

Now, granted, there are real problems with measuring the value of an iteration in something as complex and interconnected as a virtual world. With something like a banner ad selling a physical product or stand-alone digital goods like an ebook, you have essentially a linear process: view, click through to website, build interest on the site itself, buy.

The hardest part of that is building interest on the site itself. One way to make this easier is to offer a free trial or simply outright free play in order to keep a player interested in learning about your world. This, of course, introduces an entirely new element to the chain, which is the game’s context. While the context (the financial world) in which the Wall Street Journal sells subscriptions is, barring some global catastrophe, pretty secure, people won’t (and really can’t) value the context that a virtual world provides until they experience it for themselves. The trick, then, is getting them to both create a valuable context AND to get people to stick around long enough to grow to value it.

My theory, as of today, is that split-testing can greatly assist with the latter in particular. The former’s complexity and connected web of players often acting with meta-interests at heart makes it quite difficult to isolate a particular element to split-test against. In the latter case – decreasing the drop-out rate - one simply has to extend the amount of time a player will play a free trial or an outright free game for before giving up and quitting (and ensuring you’re not just giving up the farm for free if it’s a subscription game we’re talking about). While it’s not really feasible to constantly build new game introductions to split-test against each other, it IS feasible to re-write your intro copy and split-test that way. This is more relevant for text MUDs in that a larger proportion of the experience is rooted in text (often 100% in many text MUDs) but given that conversion rates for something like a Google text ad can be increased by, in some cases, over 50% simply by capitalizing a single letter, I have no doubt whatsoever that simply split-testing the written copy for the game introduction could have a huge effect, iteratively. Imagine split-testing the order of tasks learned or even different intro quests against each other. I think the possibilities are enormous, and we’re going to consciously employ split-testing in our newest project’s newbie experience.