Introducing Test Pilot

cbeard

By: cbeard

On: March 31st, 2008 Posted In: Mozilla Labs

Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments.Test Pilot

Presently, whenever Mozilla publishes a new experiment we rely upon the blogosphere and word-of-mouth to attract the attention of people interested in trying it out. We then hope that some number of those early adopters will come back and find the appropriate place to provide feedback.

The problem is that we are systemically biasing feedback towards only those who happen to hear about a particular experiment (on a given day) and in those cases skew to the technically-savvy early adopter. It is therefore very hard to derive conclusions representative of our much larger and increasingly mainstream user base.

We also have no structured way in which to request feedback, and in particular, positive feedback. People are much more likely to take the time to dig in and figure out how to provide feedback if they have a negative reaction. It’s also likely true that the less technical adopters are confused when faced with Mozilla’s panoply of feedback options.

And finally, at the end of this “testing process,” we effectively throw away the interested people who have taken the time to volunteer as testers and start over again from scratch the next time around.

Overview of Test Pilot
We can provide a much more satisfying experience all around by putting in place some basic infrastructure. Here’s the idea:

  • We develop and promote a formal Test Pilot program with a Firefox add-on at its core.
  • The first time the Test Pilot add-on is run, it asks a few simple non-personally-identifiable questions in order to put the user into a demographic bucket, e.g. technical level, locale, etc., and to let them opt in to additional anonymous instrumentation.
  • Test Pilot will then notify its users when a new experiment is available for testing. If the user opts in, it will download the required software (if any) and load any information required to get started with the new experiment, e.g. overview, use cases, etc.
  • After either a specified amount of time or upon completion of a specific action, Test Pilot will prompt the user for feedback. The feedback form will only ask a few questions selected from a much larger set. A link will be provided to provided more comprehensive unstructured feedback or bug reports.
  • The set of questions posed for feedback will be randomly distributed within each demographic bucket to ensure statistical significance of the results.
  • Anonymized aggregate results and analysis will then be posted automatically to the Test Pilot site.
  • All participants will receive a “flight badge” displayed in their Test Pilot profile and available to embed on blogs, social networks, etc.

The idea is that by reducing the amount of required feedback to only a few clicks we can increase overall rates of participation.

While this will not take the place of our traditional testing and feedback mechanisms, Test Pilot should provide a new set of tools and capabilities to increase the efficiency and effectiveness of our user testing. It should also allow us to grow our community and mindshare with people who want to help but to whom we haven’t yet given the opportunity to participate.

Roadmap
We’re pulling together a basic framework now with plans to launch the first version within the next few weeks. Specific features and roadmap are being developed in the Test Pilot discussion forum on the Mozilla Labs site.

Getting Involved

  • Discuss, debate, and add to the design in the Labs discussion forum
  • Jump into #labs on irc.mozilla.org

This entry was posted on Monday, March 31st, 2008 at 12:09 pm and is filed under Mozilla Labs. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

22 Responses to “Introducing Test Pilot”

  1. lpalli

    The image of the Firefox Test Pilot, on the top of the post, is available in a bigger size or in vectorial version ? It’s Free, Open Source, Creative Commons or closed ?

    Thanks!

    Luca

    March 31st, 2008 at 1:27 pm
  2. Joey

    Love the idea,
    Would defenitely participate when the testpilot plugin is available.
    Keep up the good work!

    March 31st, 2008 at 1:43 pm
  3. Kurt (supernova_00)

    Introducing? Well where is the download link?

    March 31st, 2008 at 4:38 pm
  4. David Naylor

    Sounds like a great idea!

    But surely, if you don’t somehow include it randomly in the Firefox installs (like talkback used to be included) then you will still only be getting geeks involved?

    April 1st, 2008 at 2:26 am
  5. Tek Link Film

    Teşekkürler.. Very Good..

    April 1st, 2008 at 6:47 am
  6. Seattle John

    I would love to participate. I have plenty of time to play on ‘this new fangled machine’. Currently running FF 3.0 B5 with Weave 0.1.xx on MS Vista Home Prem.
    Let me know ASAP it is ready to dl.

    April 4th, 2008 at 10:07 am
  7. Natanael L

    I’ll participate in this when it’s ready!

    I have Ubuntu 7.10 (will install 8.04 when out of beta) and MS Vista Home Premium and are using both FF2 and FF3b5.

    April 7th, 2008 at 2:04 am

Trackbacks

  1. foxiewire.com

    Introducing Test Pilot…

    Test Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% representative sample of the Firefox user base for soliciting wide participation and structured feedback for Labs experiments….

    March 31st, 2008 at 12:57 pm
  2. 1207022002 at TweetLinked

    […] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p… Filed under Uncategorized […]

    March 31st, 2008 at 3:58 pm
  3. March 31st, 2008 23:53:22 at TweetLinked

    […] (reference) (more) I want to be a Mozilla Test Pilot! http://labs.mozilla.com/2008/03/introducing-test-p… Filed under Uncategorized […]

    March 31st, 2008 at 4:06 pm
  4. March 31st, 2008 23:53:22 -0400 at TweetLinked

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

    March 31st, 2008 at 4:46 pm
  5. March 31st, 2008 23:53GMT at TweetLinked

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

    March 31st, 2008 at 4:53 pm
  6. March 31st, 2008 23:53 GMT at TweetLinked

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

    March 31st, 2008 at 4:56 pm
  7. Mozilla Labs » Blog Archive » Introducing Test Pilot at TweetLinked

    […] Blog Archive » Introducing Test Pilot (reference) (more) I want to be a Mozilla Test Pilot! labs.mozilla.com/2008/03/introducing-test-pilot/ Filed under Uncategorized […]

    March 31st, 2008 at 5:00 pm
  8. l word

    […] site of the National Civil Rights Museum. Todahttp://www.centredaily.com/business/story/496981.htmlMozilla wants unbiased testing reports, creates Test Pilot&quotTest Pilot is an idea for a new user testing program for Mozilla Labs that aims to build a 1% […]

    March 31st, 2008 at 8:51 pm
  9. ¿Cómo conseguir retroalimentación por Internet para tu empresa? « Skylab

    […] gente de Mozilla Labs está perfilando una idea, bajo el nombre de Test Pilot, que tiene como fin obtener retroalimentación -selectiva- de una muestra representativa del 1% de […]

    April 1st, 2008 at 10:00 am
  10. Lancement de Test Pilot | FirefoxBlog

    […] Test Pilot (pilote d’essai, en français) est une extension destinée à permettre aux utilisateurs de Firefox 3 de donner leur avis sur différentes fonctionnalités du logiciel. […]

    April 8th, 2008 at 3:42 pm
  11. Online Poker Sites

    Online Poker Sites…

    Online Poker Sites
    Casino Games
    Play Poker
    Online Gambling
    Keno
    Free Slots
    Top Online Casinos…

    April 29th, 2008 at 7:16 pm
  12. Comment on Introducing Test Pilot by Online Poker Sites

    […] Marijan Stefanovic wrote an interesting post today onHere’s a quick excerptOnline Poker Sites… Online Poker Sites Casino Games Play Poker Online Gambling Keno Free Slots Top Online Casinos… […]

    April 29th, 2008 at 11:49 pm
  13. The Folding Fool » Blog Archive » Comment on Introducing Test Pilot by Online Poker Sites

    […] Talk to the author know. […]

    April 29th, 2008 at 11:56 pm
  14. Comment on Introducing Test Pilot by Online Poker Sites | Business Online

    […] Arjen Lentz wrote an interesting post today onHere’s a quick excerptOnline Poker Sites… Online Poker Sites Casino Games Play Poker Online Gambling Keno Free Slots Top Online Casinos… […]

    April 30th, 2008 at 12:10 am
  15. John’s Blog » Blog Archive » Mozilla, Firefox & Data

    […] used to improve the user interface of Firefox, and another is a project in Mozilla Labs called Test Pilot. They’re both early and very limited in […]

    May 13th, 2008 at 1:08 pm

Leave a Reply

About Mozilla Labs

Laboratories are where science and creativity meet to develop, research, and explore new ideas. Mozilla Labs embraces this great tradition - a virtual lab where people come together to create, experiment, and play with new Web innovations and technologies. Read More »

Archives

Subscribe


Left Shadow Right Shadow