Testing Program

Test Pilot

Presently, whenever Mozilla Labs publishes a new experiment we rely upon the blogosphere and word-of-mouth to attract the attention of people interested in trying it out. We then hope that some number of those early adopters will come back and find the appropriate place to provide feedback.

The problem is that we are systemically biasing feedback towards only those who happen to hear about a particular experiment (on a given day) and in those cases skew to the technically-savvy early adopter. It is therefore very hard to derive conclusions representative of our much larger and increasingly mainstream user base.

We also have no structured way in which to request feedback, and in particular, positive feedback. People are much more likely to take the time to dig in and figure out how to provide feedback if they have a negative reaction. It's also likely true that the less technical adopters are confused when faced with Mozilla's panoply of feedback options.

And finally, at the end of this "testing process," we effectively throw away the interested people who have taken the time to volunteer as testers and start over again from scratch the next time around.

Overview of Test Pilot

We can provide a much more satisfying experience all around by putting in place some basic infrastructure. Here's the idea:

• We develop and promote a formal Test Pilot program with a Firefox add-on at its core.
• The first time the Test Pilot add-on is run, it asks a few simple non-personally-identifiable questions in order to put the user into a demographic bucket, e.g. technical level, locale, etc., and to let them opt in to additional anonymous instrumentation.
• Test Pilot will then notify its users when a new experiment is available for testing. If the user opts in, it will download the required software (if any) and load any information required to get started with the new experiment, e.g. overview, use cases, etc.
• After either a specified amount of time or upon completion of a specific action, Test Pilot will prompt the user for feedback. The feedback form will only ask a few questions selected from a much larger set. A link will be provided to provided more comprehensive unstructured feedback or bug reports.
• The set of questions posed for feedback will be randomly distributed within each demographic bucket to ensure statistical significance of the results.
• Anonymized aggregate results and analysis will then be posted automatically to the Test Pilot site.
• All participants will receive a "flight badge" displayed in their Test Pilot profile and available to embed on blogs, social networks, etc.

The idea is that by reducing the amount of required feedback to only a few clicks we can increase overall rates of participation.

While this will not take the place of our traditional testing and feedback mechanisms, Test Pilot should provide a new set of tools and capabilities to increase the efficiency and effectiveness of our user testing. It should also allow us to grow our community and mindshare with people who want to help but to whom we haven’t yet given the opportunity to participate.