Demystifying Moderated User Testing

One of my favourite activities at PurpleFire is user research, in particular moderated user testing. I’m a huge advocate of user testing to truly understand what users are doing on your site and why. I find sitting with real users and learning from them a truly eye opening and motivating experience. It is sometimes an underrated research technique which I feel is one of the most powerful tools in our repertoire.

Here I demystify some common misconceptions around moderated user testing.

Moderated user testing is clinical and unnatural

Due to the association of the term moderated user testing with a ‘testing lab’ there is the perception that this type of research is a scientific procedure with participants being interviewed, prodded and probed by the moderator wearing a lab coat, goggles and a clip board. One of the many myths I’d like to demystify.

Although moderated user testing is studying user behaviour, the way we approach this type of user research is more about the personal touches, setting the scene and making the participant feel at ease so they are able to give open and honest opinions.

To lessen the ‘observation effect’ we are very relaxed and personable in our approach. From the way we introduce ourselves, build a rapport, give our own opinion or anecdotes, sympathise with the participant and keep the conversation light, we get the most genuine and in-depth insight.

We often come out of the sessions with a full life story of the participant and many more personal details than we even asked due to the trusting and personable relationship built during the course of a session. A good sign that the participant feels relaxed and at ease!

Users will say what they think you want to hear

There is the opinion that participants are eager to please and will be extremely polite and positive about the site, irrelevant of their experience. In post session questions whilst recalling their experience we have seen this ‘rose-tinted view’, or users blaming themselves. We also find that users often refer back to positive and particularly negative experiences. It’s not uncommon for users to be almost hostile to the site.  However, observing users whilst they are actually on the site will tell you all you need to know; they can’t fake interactions.

Another way of dealing with this is comparisons with other sites, particularly when the users doesn’t know which site is under review. They will tell you which they prefer and why.

We encourage participants to provide feedback with positive and negative thoughts. In our experience participants fully immerse themselves into the scenario and browse the site, complete forms and talk to the computer (or whoever’s listening!) just as they would at home.

Users will not complete the scenario in the same way they would in their own home

Obviously users are not in their natural setting, sitting on their couch and casually browsing on their iPad, but -contrary to common belief – in moderated sessions they also aren’t sat with a camera in their face, a recording device pointing at them and a moderator breathing down their neck.

We use specialised facilities which are set up with soft furnishings to add a homely feel and flexible seating, which can be moved around depending on the participant and scenario.

We often test across multiple devices to gain extra valuable insights and also look at communication outside the standard website journey: text, email or postal communication, for example. As much as possible we try to mirror the end-to-end journey as a user would have completed it in ‘real life’ and it is certainly a skill of the moderator to probe for this extra insight around offline activity where appropriate.

The results are biased depending on the moderator’s agenda

We are very fortunate to have open and trusting relationships with our clients who don’t push us to ask questions about every single interaction and page, or to stick to a strict script.

We usually start by giving a fairly open scenario then sit back, watch and listen. The sessions are mostly lead by the participant, with the moderator adapting their questioning style depending on the type of participant. For example, a very reserved participant who is struggling to voice their thoughts may need a little more encouragement to explain what they were expecting or why. However, we never lead the participant. Instead we mirror them. If the participant asks a question we quite often just repeat it back to them.

Top Tip:

What participants say they would do and what they actually do is different, try and watch and listen as much as possible and only probe when necessary. This includes observing body language and facial expressions. Based on just looking at the session’s video with no sound you can tell an engaged user vs. a disinterested or hesitant user.

Don’t interrupt a participant whilst they are ‘in the zone’ and focussed on completing a form, looking for something in particular or explaining their thoughts. Use the post session questions to probe further so that the session is kept as seamless as possible.

When to use moderated user testing

Moderated testing can be used at any stage of a website lifecycle; before a re-design and development to inform the future requirements, when new features or site changes are released, using a prototype or test website or post launch of a new site to ensure it meets user’s needs.

Moderated research means you can be flexible and steer user sessions or clarify insights when necessary. Thus for more complex sites such as banking or utilities, when users may give up if not supported, in remote user testing for example, this type of research is extremely valuable. The intent is still captured yet the moderator can explore the users concerns and what their next steps would be.

If you want to know why users are interacting the way they are on your site, moderated user testing is the richest form of user research which will open your eyes to new challenges and concepts that may have not been previously considered and will certainly help you to put your users at the forefront of future business decisions.