Top 10 Mistakes People Make When Conducting User Testing

Susan WeinschenkAt yesterday’s Clean Up Your Bad User Testing Habits webinar, Dr. Susan Weinschenk highlighted the mistakes people make when planning, conducting, and analyzing user testing and shared tips on how to avoid those bad habits.

Dr. Weinschenk pointed out it’s easy to fall into bad habits when you’re conducting user testing. And those bad habits can sabotage your user research studies.

Don’t waste time and money with your user testing. Be more effective by avoiding these top ten mistakes:

  1. Skip the highlights video With all that video to review, you may be tempted to skip the highlights video. A picture is worth a thousand words; a video is worth more. Showing short clips of each participant having the same problem speaks volumes. Get in the habit of marking clips as you review so you can easily make a highlights video.
  2. Drawing conclusions that aren’t supported by your data: You may have ideas, but the data isn’t supporting that idea. We always want to solve the puzzle.
  3. Skip over surprises: There’s always surprises. Look for surprises. Be ready for surprises.
  4. Testing late: It’s a common problem to start testing late. Test early and often. Avoid waiting to test in the final stage in the product development process, where changes will be more time-consuming and expensive.
  5. Put too much stock in after-the-test surveys: Users don’t always answer them accurately.
  6. Do a data dump: Don’t just provide an Excel spreadsheet with data. You need to include analysis and interpretation. Provide a summary highlighting high, medium, low priority items with possible outcomes.
  7. Too many cells: Do you need to compare across gender, age, or other grouping? Identify what you need to compare. If you want to make sure you’re getting variability, you can do that within the cell.
  8. Testing too many people: You will spend money and time you might not have to spend. And you’ll have a lot of videos to watch and diminishing returns after a specific number of user tests. So, how many people should you test? Check Jeff Sauro’s How to Find the Right Sample Size for a Usability Test.
  9. Draw conclusions after two people: You don’t have enough data yet. Drawing conclusions early on will make it hard to let go of those conclusions later. People have a confirmation bias: we pay attention to data that confirms what we already believe.
  10. Skipping the pilot: Run a pilot with one person so you can ensure the wording on the instructions provides the information you are seeking. We don’t realize how the words will be understood.
  11. Tips Shared During Q & A

    1. Try not to do too many tasks in your user testing. People may not complete the last couple tasks. With unmoderated user testing, it’s more challenging to have a large number of tasks.
    2. Dr. Weinschenk’s recommendation for a great book for user testing: Steve Krug’s Rocket Surgery Made Easy

    Thank you to UserTesting for hosting the webinar. I always learn something new from their user experience webinars.

About Deborah Edwards-Onoro

Deborah is a web developer and user experience professional focused on usability and accessibility. She's active in local groups as a leader for Detroit User Experience, Refresh Detroit, and Metro Detroit WordPress. In her free time, you'll find her birdwatching, shooting photos, reading, or watching tennis.

%d bloggers like this: