Second Life A/B orientation testing, and how to mess it up

A/B testing has been used by Linden Lab for some time to gauge the effectiveness of the new user orientation experience in Second Life. A/B testing (also known as Split Testing) is a fairly straightforward system where a set of users are given the A experience (basically what already exists) as a control group and a random sampling are shunted to an alternative (the B experience) that differs in only one respect. The results of the two groups are then compared to look for statistical improvement. Through a series of A/B tests, the particular elements that tend to produce the greatest improvements can be determined.

It's a great way of improving new user experience over time. Unfortunately, the efforts of Linden Lab's VTeam (in charge of the volunteer programs) create results that are not only misleading, but actually statistically worthless.

Read Full Story >>
The story is too old to be commented.