Beta Feedback Survey

This is an internal document

While making small fixes to this page like fixing typos and dead-links is encouraged, any changes which significantly modify the information of this page should be suggested on the discussion page instead, as this is an internal document.

This page contains an overview of the Beta retention, based on the user feedback survey and an analysis of the opt-in preference data. There is more detailed information on the following as well:

The data presented here was collected from August 8 to October 29, 2009.

Background

The Usability Beta is currently conducting two feedback surveys:

Feedback Survey: The feedback survey allows users to give general feedback about the Beta program. There are a series of multiple-choice questions as well as open ended questions around likes/dislikes about the Beta, usage of features, etc.
Opt-out Survey: When a user elects to opt-out of the Beta, there is an optional survey they can fill out upon exiting. The questions are the same as those in the Feedback Survey, but also include specific questions about reasons for leaving the Beta.

In addition to capturing the user feedback, the survey also logs the browser, OS, and screen resolution used by the participant.

For screenshots of the Beta surveys, click here.

Update: April 30, 2010

Note: There is a potential issue with the Commons data. We are investigating the data and will update once we know more.
April ended with a total of 698,018 users having tried the Beta.
The cumulative retention rate across all projects decreased slightly to 79.9% as of April 30.

Update: March 31, 2010

March ended with a total of 635,942 users having tried the Beta.

Approximately 64,400 additional users tried the Beta in March.
The cumulative retention rate across all projects increased slightly to 80.0% as of March 31.

Update: February 28, 2010

February ended with a total of 571,579 users having tried the Beta.

Approximately 59,100 additional users tried the Beta in February. This number is down slightly, even considering the fact that February is a shorter month.
The cumulative retention rate across all projects held steady at 79.8% as of February 28.

Individual Projects: February Retention and Trends

In Feb 2010, the incremental retention was 80.4%. This is down slightly from Jan 2009, where the incremental retention was 81.5%

enwiki retention was down slightly (83.2% in Feb vs. 84.3% in Jan). Cumulative retention for enwiki was 83.6% as of Jan.
dewiki retention was down slightly (72.6% in Feb vs. 75.1% in Jan). Cumlulative retention for dewiki was 72.2% as of Jan.
jawiki retention was down slightly (71.5% vs. 72.1% in Jan). Cumulative retention for jawiki was 62.9% as of Jan.
kowiki retention is still low (57.9% vs. 64.1% in Jan). Cumulative retention was 58.5% as of Jan.

Update: January 31, 2010

January ended with a total of 512,462 users having tried the Beta.

Approximately 70,500 additional users tried the Beta in January.
The cumulative retention rate across all projects was 79.8% as of January 31.

Individual Projects: January Retention and Trends

In Jan 2010, the incremental retention was 81.5%. This is down slightly from Dec 2009, where the incremental retention was 82.4%

enwiki retention was down slightly (84.3% in Jan vs 85.6% in Dec). Cumulative retention for enwiki was 83.6% as of Jan.
dewiki retention was up slightly (75.1% in Jan vs. 74.1% in Dec). Cumlulative retention for dewiki was 72.1% as of Jan.
jawiki retention in Jan 2010 appears to have increased (72.1% in Jan vs. 64.9% in Dec). Cumulative retention for jawiki was 62.0% as of Jan.
Kowiki retention is still low (64.1% in Jan vs 63.2% in Dec). Cumulative retention was 58.6% as of Jan.

End of Year Update (as of December 31, 2009)

The following is an year-end update for the Usability Initiative. Overall, usage of the the Beta has been continuing to increase and retention rate continues to inch upwards.

Snapshot as of December 31, 2009

A total of 441,756 users have tried the beta. In December, approximately 79,000 users joined the beta.
The cumulative retention rate, across all projects, is 79.5% (vs. 78% at the end of October)
Most of the beta users come from the top 20 projects, which account for approximately 89% of the beta users.

Cumulative Retention Rates:

Individual Projects: December Retention and Trends

enwiki, which has the largest beta usage, has a cumulative retention rate of 83.5%. Retention for enwiki has been rising steadily, with retention rate for the month of December at 85.5% (vs. 82.4% prior to Sep 12). Note: The Sep 12 date does not have any particular significance other than it's the date we ran the report. We will try to republish the data with more standard time periods (e.g., beginning of the month)
dewiki has stayed strong. The December retention rate for dewiki was 74.3%. Dewiki retention was 69.7% prior to Sep 12 and has been hovering in the mid-70s ever since).
jawiki is retention is still lagging at 64.9% for December. It looks like the retention may have improved over the last 2 months, but there is a lot of noise in the data, so we cannot be certain. While we would like to say that the font size change affected retention, we cannot conclude this because of the noise.
kowiki is lagging as well, with a cumulative 58.0% retention, though the number of beta users is still relatively low (3,371 have opted into the Beta as of Dec 31).
plwiki, which had an issue with gadget compatibility, has improved quite a bit. Its December retention was 75.8%. Prior to Sep 12, its retention rate was 65.3%.

December 2009 Retention Rates:

The detail behind the updated retention analysis, including both cumulative rates as well as incremental rates (e.g., rates for the month of December) may be found here.

Previous retention analysis may also be found here.

Previous Summary (as of November 30, 2009)

The following table lists the retention rates for the Usability Beta with the top 10 highly accessed language Wikipedias broken out:

The entire list of retention rates for all language wikis is located here.

A couple points:

Almost 380,000 users have opted into the Beta since the start of the program with an overall retention of about 79% (vector - the retention rates cited will all be for vector retention) as of November 30, 2009.
The retention rate for enwiki is strong (83%).
Several wikis in the top 10 have retentions below the average. Most notably, jawiki and plwiki have retention rates under 70%.

Analysis of Language Wikis with Low Retention

We did a review of the user comments submitted as part of the survey. Because of the volume of comments was very high, we were unable to do a quantitative review of the comments. Rather, we looked for general patterns that might help us understand why certain language wikis had a lower retention rate. Here’s what came out of the data:

jawiki: Many jawiki users noted that the font size in the Beta was too small. We confirmed this with Firefox and IE. There were also a number of comments on the layout (e.g., users not used to the location of the search box, users liked/did not like the new layout), but the most common comment by far was that the font size was too small.
dewiki: Unlike jawiki, there wasn’t one particular comment that seemed to appear significantly more frequently than others. In comparison with the users of other language wikis, it does appear that dewiki users are having a more difficult time adjusting to the new location of the search box. Other frequent comments included: like/dislike new layout and/or colors, beta seemed slow, spacing issues in tabs for long German words.

Another potential contributor to the lower retention rate for dewiki users is the number of IE6 users. These users had a much lower retention rate than average dewiki user (60% for IE6 vs. 70% average). The effect of IE6, however is limited since IE6 users comprised only about 3% of the dewiki users who completed the survey.

zhwiki: Users of zhwiki noted a few issues:

Traditional vs. Simplified Chinese: Some users mentioned that the Beta showed only Simplified characters, even when they tried to switch to Traditional characters.
Slow: Some users mentioned that the beta was "slow."
Font size too small: Like users of jawiki, some users of zhwiki complained that the font size was too small.

Regarding plwiki, we found that the Beta was not compatible with the refTools gadget. Starting September 29, this gadget was made compatible with the Beta, so we hope that the plwiki retention rate will be higher in the future.