Great story from an interview that Adapative Path’s Peter Merholz did with the founder of User Interface Engineering Jared Spool:
PM: Any other mistakes, in terms of how tests are structured?
JS: There are many problems I see in terms of people not
understanding the effects of testing affecting results.
Here’s a simple example that we discovered. We were working with a
client, testing a website that sold furniture. And the client had
already done some testing. They had created some simple tasks, and
one of the tasks was basically asking users to come to the site and buy
a bookcase they might want to buy.Every user in that study went to the search box, and typed in the word “bookcase,” immediately. So they were off trying to improve the hits on the word “bookcase,” and get the results better. We did a subsequent test, and instead of asking people to look for a bookcase, we gave them a slightly different task.
We said imagine you have 300 different paperback and hardcover books in
cardboard boxes, all strewn through your living room. You need a way to
organize this stuff such that you can find the stuff you like, and
people who come over can be impressed with your collection. What do you
do? People would go to the site, and then click on links. They’d click
on furniture links; they’d type into the search engine storage systems,
and then type in shelves. Not a single user in our phase of the study
typed in the word bookcase. So by changing the description of the task,
we got a completely different result.