GWA Project/HI201 H210
July 2008
Some tests are going at the HI's 201 - 210. In this case the HI's are the entry islands for new residents.
Blue Linden is involved in this as being a part of the NewUserExperience team. Overhere a collection of articles and transcripts about the subject gathered from various places.
By: Lisa Lowe 12:00, 25 July 2008 (PDT)
In a comment on above article:
Blue Linden, on July 24th, 2008 at 12:45 pm Said: Common Sense vs. Metrics
I have to disagree that what needs to be done is to rely solely on “common sense”. Over the past 9 months of polling, live focus testing and A/B testing, one thing is very clear: *Everyone* has an opinion regarding what’s best for newbies. Despite the great majority of them sounding perfectly sensible, many of these opinions are contradictory and only a very small percentage of them are supported by any data at all.
Mentors have contributed significantly to the discussion, and are very good at relating a specific kind of observed data, but it’s valuable to consider a number of sources when looking at complex problems such as “orienting new users in Second Life”. So I’d like to urge people to avoid making blanket statements of support or condemnation based solely on personal experience in new user spaces.
———
The HI as 1st Location A/B test.
When looking at 7 key metrics related to retention and indicating a commitment in SL, we found that the HI has better numbers than the OI on 5 of them.
We can also look at the effectiveness of changes made to HI to address specific problems. For example, Mentors regularly tell me that one of the most common questions is “how do I get to (SL)” where (SL) = “the city”, “the mainland”, “the game” etc., which we see in live focus testing as well, so we included more ways to leave the HI than there are on OI. The result was a 6% increase in people making their way to 2nd locations. I imagine we’ll do even better when we act on the advise of mentors and testers who have suggested that the meaning of the word EXIT on OI/HI is often misunderstood.
Notably, In metrics associated with people returning to Second Life in subsequent days, the HI scored higher. And of course, we’d like people to come back, rather than giving up in frustration and leaving for good.
So the results of the A/B test are positive enough for us to go ahead and make a few more tweaks to HI and then TEMPORARILY replace the current OI. Your input has been valuable and much appreciated. I will certainly continue to watch discussions in Mentor channels and solicit your feedback at my office hour (http://wiki.secondlife.com/wiki/Office_Hours#Blue_Linden) as we continue to create and test a more effective alternative to the current Orientation experience.
xoBlue
By: Lisa Lowe 12:00, 25 July 2008 (PDT)
Blue also joined the Mental Mentors group (july 24, 2008). In here I tried to get some info. This is the transcript (filtered with from other non relevant comments):
Blue Linden appeared in Mental Mentors and answered a few questions about the tests.
[11:21] Blue Linden: A/B tests related to new user experience focus on a few metrics like those that give us $$.....% Premium upgrade, % L$ purchase, % Buy Land
[11:22] Blue Linden: and also retention numbers lik % returned after 1, 7, 30 days
[11:23] Blue Linden: one of the most interesting numbers is % of new users that visit 2nd location in SL.....very important when looking at the OI
[11:25] Blue Linden: for example, with the A/B test on Help Island as a first location, the numbers look really good [11:25] Blue Linden: better than the current OI
[11:26] Blue Linden: 66.9% go to a second location vs 61.3% on OI [11:26] Blue Linden: oh sorry, A/B test means you have two cohorts.....A is the default way it's done normally, and B is the test island that's different 11:27] Blue Linden: so in this case A = OI and B = HI [11:28] Lisa Lowe: might the 67% be because there are more exit signs on the HI's there?
[11:30] Blue Linden: yes, could be Lisa....we saw a jump in that number when we added a more obvious exit sign to the OI [11:30] Blue Linden: the old sign looked like an advert and people just ignored it [11:31] Blue Linden: one of the key goals is to help people find value in SL....some are not going to spend much time looking, so they aren't going to bother with tutorials until they know there's a reason to learn
[11:31] Blue Linden: yeah, or BEGIN your SL, or CONTINUE on to SL....
[11:32] Blue Linden: yes, live focus testing shows people ignoring almost every bit of text you put in front of them >_< [11:32] Lisa Lowe: Blue. You actually check all new residents if they start upgrading, buying etc?
[11:34] Blue Linden: Lisa, almost anything you do online leaves a papertrail....so the data people here just dig it out and find the patterns
[11:35] Blue Linden: yeah, people thinking the OI/HI is the whole of SL is a problem.....one of the reasons I like hubs as a 1st location....you can see the whole map from there
[11:37] Lisa Lowe: Blue, what are the latests retentionrate numbers? 12%? [11:38] Blue Linden: not sure Lisa, but I think those are published on the web somewhere
[11:40] Blue Linden: we will know better when we do next A/B test of hubs as 1st location [11:40] Blue Linden: the previos hubs test did well
[11:42] Lisa Lowe: Blue, on the HI 200+ lots go into Appearance right away. Where does that come from? a Tutorial like under Help? [11:43] Blue Linden: yes....appearance is sooner in the tutorial than in the OI HUD
[11:43] Blue Linden: new style hubs take that into account and provide quiet space early on for playing with av
[11:43] Lisa Lowe: They automatically have that Tutorial openend? [11:44] Blue Linden: yes [11:44] Blue Linden: although MOST of them close it right away >_< [11:44] Lisa Lowe: Isn't that tutorial only in the RC's?
[11:46] Blue Linden: Search is also much sooner in the Browser tutorial than on OIs
Note: He may not have seen everything due to heavy lag.