A couple of weeks ago, Seth, Stas, and Pascal did a great FOSDEM presentation of the Community Survey project and the results from the first two surveys. For people who didn’t see the presentation, the Community Survey project was launched in November last year with the goal of learning more from our Mozilla community about e.g. decisions needed to be made, or ones that already may have been made.
Earlier this week, Seth, Stas, and I had a follow-up meeting where we went through the results of the survey about user-to-user support, both to summarize the insight we gained, and also to figure out what the next step should be moving forward. Designing this survey was very much a learning process for all of us, and we definitely learned about how hard it is to ask a question in a way that eliminates the risk of misinterpretation. We also learned (or rather got confirmed) that the results of a survey can be interpreted in many ways. Designing surveys falls somewhere between science and art, which made this both an intellectually challenging and genuinely fun experience.
We made a few important findings in the results of the survey (with more to come later):
- End users and active community members have different opinions on their current community support
- End users are more interested in “visuals” in support, e.g. screenshots and screencasts
- End users value interactive help higher than active community members
- The Mozilla community is incredibly vibrant and diverse
Let’s talk a little bit about each of the above findings.
End users and active community members have different opinions on their current community support
This graph shows how people rate the quality of their local support sites differently depending on what type of user they are:
As with any statistics, you have to be very careful about making assumptions. In this case, the classifications “End users,” “Community Members,” and “Active Members” are loosely based on the participant’s answer on the demographic question, which can be read about in detail on the Community Survey Blog shortly. As an example of why it’s dangerous to make any assumptions, one person could be actively following news on Mozilla and consider him/herself as an active community member, while another person could be doing the same but not consider him/herself as anything close to a community member. Everyone is different, and with that comes different perceptions, even on things like how to define a community.
That said, it’s pretty clear in the graph above that end users aren’t as enthusiastic about the support offerings as the hardcore community members are. The reason for that is non-trivial to explain without asking more detailed questions about the situation, which fortunately we did.
End users are more interested in “visuals” in support, e.g. screenshots and screencasts
This might come as no surprise; end users aren’t as interested in reading long instructions without illustrative screenshots, or even videos/screencasts. By adding these elements to documentation, the information is much easier for people to grasp. Getting this confirmed was important to us, as one of the important next steps for SUMO after the Firefox 3 release is to get screencasts to the Knowledge Base articles.
End users value interactive help higher than active community members
When asked what assets would make the support websites more useful, interactive help such as instant messaging was rated higher among end users, compared to community members. This isn’t surprising either; less technically inclined people want more hand-holding when it comes to problem solving. That’s not limited to computer problems, by the way. If you’re not very handy, you need more hand holding when e.g. assembling an IKEA kitchen table. In my case, the hand holding I request turns more into a demonstration, where my friends end up doing the work for me. Of course, I still pretend that I’m trying to learn how to do it myself, otherwise I don’t get the help I need.
When it comes to software documentation, interactive help could obviously mean IM or IRC chat, but it could also mean making the search experience more interactive, or, again, adding screencasts to the knowledge base articles. During our meeting, Seth played a bit with the idea that by extending knowledge base articles with screencasts, you would achieve the same effect as someone holding your hand while instructing you to fix the problem. I’d definitely like to explore this further.
The Mozilla community is incredibly vibrant and diverse
Lastly, we found that among the participants taking the English version of the survey, only 40.3% ofÂ them were actually from English speaking countries, despite the fact that our awesome l10n community helped us translating the survey into 17 languages. We’re just everywhere, it seems.
To conclude, the survey gave us more insight about our community, and we got a few assumptions about support confirmed which will help us move forward. We also learned more about survey design, which will help the Community Survey project to continually improve, just like every other Mozilla project.
I’d like to take this opportunity to thank Stas and Seth for letting me work with them on this. It’s been a really entertaining and educating ride. Oh, and if you guys have any free slots in the future, I’d love to do another survey as the SUMO project evolves!