03.31.2008
Introducing Test Pilot
Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments.
Presently, whenever Mozilla publishes a new experiment we rely upon the blogosphere and word-of-mouth to attract the attention of people interested in trying it out. We then hope that some number of those early adopters will come back and find the appropriate place to provide feedback.
The problem is that we are systemically biasing feedback towards only those who happen to hear about a particular experiment (on a given day) and in those cases skew to the technically-savvy early adopter. It is therefore very hard to derive conclusions representative of our much larger and increasingly mainstream user base.
We also have no structured way in which to request feedback, and in particular, positive feedback. People are much more likely to take the time to dig in and figure out how to provide feedback if they have a negative reaction. It’s also likely true that the less technical adopters are confused when faced with Mozilla’s panoply of feedback options.
And finally, at the end of this “testing process,” we effectively throw away the interested people who have taken the time to volunteer as testers and start over again from scratch the next time around.
Overview of Test Pilot
We can provide a much more satisfying experience all around by putting in place some basic infrastructure. Here’s the idea:
- We develop and promote a formal Test Pilot program with a Firefox add-on at its core.
- The first time the Test Pilot add-on is run, it asks a few simple non-personally-identifiable questions in order to put the user into a demographic bucket, e.g. technical level, locale, etc., and to let them opt in to additional anonymous instrumentation.
- Test Pilot will then notify its users when a new experiment is available for testing. If the user opts in, it will download the required software (if any) and load any information required to get started with the new experiment, e.g. overview, use cases, etc.
- After either a specified amount of time or upon completion of a specific action, Test Pilot will prompt the user for feedback. The feedback form will only ask a few questions selected from a much larger set. A link will be provided to provided more comprehensive unstructured feedback or bug reports.
- The set of questions posed for feedback will be randomly distributed within each demographic bucket to ensure statistical significance of the results.
- Anonymized aggregate results and analysis will then be posted automatically to the Test Pilot site.
- All participants will receive a “flight badge” displayed in their Test Pilot profile and available to embed on blogs, social networks, etc.
The idea is that by reducing the amount of required feedback to only a few clicks we can increase overall rates of participation.
While this will not take the place of our traditional testing and feedback mechanisms, Test Pilot should provide a new set of tools and capabilities to increase the efficiency and effectiveness of our user testing. It should also allow us to grow our community and mindshare with people who want to help but to whom we haven’t yet given the opportunity to participate.
Roadmap
We’re pulling together a basic framework now with plans to launch the first version within the next few weeks. Specific features and roadmap are being developed in the Test Pilot discussion forum on the Mozilla Labs site.
Getting Involved
- Discuss, debate, and add to the design in the Labs discussion forum
- Jump into #labs on irc.mozilla.org