Comments on: Split-Testing Games in China http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/ A blog on virtual worlds, games, and digital content, from Matt Mihaly Thu, 20 Nov 2008 12:57:37 +0000 http://wordpress.org/?v=2.0.3 by: Plaguelands » Blog Archive » A new take on MMOG Development http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35161 Wed, 10 Oct 2007 14:05:31 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35161 [...] Hat Tip: The Forge. This is where I learned about the whole split-testing mmogs in China thing. [...] […] Hat Tip: The Forge. This is where I learned about the whole split-testing mmogs in China thing. […]

]]> by: BobSugar http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35142 Wed, 10 Oct 2007 05:08:21 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35142 Yeah - we're finding almost the exact same thing. Split-testing and focus-testing is extremely useful for streamlining the initial experience (and we're still using it for that) - but ranges from useless to actively harming the process when applied to anything further down the gameplay learning-curve. Yeah - we’re finding almost the exact same thing. Split-testing and focus-testing is extremely useful for streamlining the initial experience (and we’re still using it for that) - but ranges from useless to actively harming the process when applied to anything further down the gameplay learning-curve.

]]> by: Matt http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35140 Wed, 10 Oct 2007 04:55:39 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35140 Yep, the reason that I specifically mentioned split-testing for a newbie experience in an online game is because you have a very measurable behavior: How long a player engages for. You don't need to (and shouldn't) ask them what they like because you're looking for objective data. Does a change result in a higher stick rate for newbies? Then it's almost certainly the right choice. When it comes to the game experience as a whole, I agree it'd be very hard to do. --matt Yep, the reason that I specifically mentioned split-testing for a newbie experience in an online game is because you have a very measurable behavior: How long a player engages for. You don’t need to (and shouldn’t) ask them what they like because you’re looking for objective data. Does a change result in a higher stick rate for newbies? Then it’s almost certainly the right choice.

When it comes to the game experience as a whole, I agree it’d be very hard to do.

–matt

]]> by: BobSugar http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35139 Wed, 10 Oct 2007 04:51:57 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35139 On the project I'm working on, we've been doing a lot of user-interface split-testing, to determine the most accessible and fun controls for the game. I have noticed one very dangerous problem with split-testing (and any form of focus-testing, for that matter): it accords an unreasonably high weight to the initial experience. We've been trying to use your example of quickly picking a 'winner,' and then quickly iterating to the next split-test, but this heads pretty rapidly towards an easily accessible, very shallow gameplay experience. Choices which aren't easily understandable to players are not only often ignored, but are sometimes disliked initially. I've been playing alot of Halo 3 lately, and when I first started, I couldn't see any perceivable strategic difference between the assault rifle, the submachine gun, and the spiker. In fact, if presented with all 3 in a focus-test, I probably would have commented that they were unnecessarily overlapping, and one or two should be cut. I think that's a particularly dangerous sort of example, because not only are your testers not going to grasp the deeper gameplay features initially, but they might even dislike them (or prefer a shallower version you also present), giving you poor feedback. The only way to avoid this is to increase the time spent under each split-test, which unfortunately fights the quick iteration ideal. On the project I’m working on, we’ve been doing a lot of user-interface split-testing, to determine the most accessible and fun controls for the game. I have noticed one very dangerous problem with split-testing (and any form of focus-testing, for that matter): it accords an unreasonably high weight to the initial experience.

We’ve been trying to use your example of quickly picking a ‘winner,’ and then quickly iterating to the next split-test, but this heads pretty rapidly towards an easily accessible, very shallow gameplay experience. Choices which aren’t easily understandable to players are not only often ignored, but are sometimes disliked initially.

I’ve been playing alot of Halo 3 lately, and when I first started, I couldn’t see any perceivable strategic difference between the assault rifle, the submachine gun, and the spiker. In fact, if presented with all 3 in a focus-test, I probably would have commented that they were unnecessarily overlapping, and one or two should be cut. I think that’s a particularly dangerous sort of example, because not only are your testers not going to grasp the deeper gameplay features initially, but they might even dislike them (or prefer a shallower version you also present), giving you poor feedback.

The only way to avoid this is to increase the time spent under each split-test, which unfortunately fights the quick iteration ideal.

]]> by: Matt http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35130 Tue, 09 Oct 2007 22:23:11 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35130 Yep, those are all potential problems with just doing one pass at it. Split-testing is most effective when you can do it over and over and over. Yep, those are all potential problems with just doing one pass at it. Split-testing is most effective when you can do it over and over and over.

]]> by: wowpanda http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35128 Tue, 09 Oct 2007 22:18:16 +0000 http://forge.ironrealms.com/2007/10/09/split-testing-games-in-china/#comment-35128 There is a chance that the smaller half might be pissed that the stuff they are familiar with is gone in the official lunch, and off creating private servers. And sometimes most popular version might not be the best one. For example I started playing WOW because my friend from college days are playing it, and I didn't compare any other games with it. Another thing will be, not everything is 50/50. There might be too many choices to do split testing (2^N), so they might end up discarding some of the good stuff in the discarded version, and adding them back upon user feed back. There is a chance that the smaller half might be pissed that the stuff they are familiar with is gone in the official lunch, and off creating private servers.

And sometimes most popular version might not be the best one. For example I started playing WOW because my friend from college days are playing it, and I didn’t compare any other games with it.

Another thing will be, not everything is 50/50. There might be too many choices to do split testing (2^N), so they might end up discarding some of the good stuff in the discarded version, and adding them back upon user feed back.

]]>