A famous true story in Human Factors/Usability circles is the one where Rolf Molich ran a study called CUE-2, where nine teams did usability testing on Microsoft’s Hotmail email system. All nine came back with different types of reports, noting different problems. Few agreed on what these “problems” were and they didn’t all pick up on the same trouble spots.

The Molich study raised all kinds of questions, including how accurate usability testing is. Now they are evaluating data from a similar study called CUE-4. I’m interested in these studies and the opinions by Jakob Nielsen, Jared Spool, and many others. It is Nielsen who coined the phrase, “discount usability”, in which the practice of heuristic evaluations became popular because they are affordable and simple to execute.

Pin The Tail on The Problem

But, do they truly root out problems? An article written by Christine Perfetti called Usability Testing Best Practices: An Interview with Rolf Molich looks into this.

According to Molich, “Usability tests are spectacular. They are excellent for convincing skeptical colleagues that serious usability problems exist in an interface. But they are also inefficient and costly. We should use them mainly in an intermediate phase to establish trust with our colleagues, and then use much more cost-efficient preventive methods such as usable interface building blocks, reviews based on standards and proven guidelines, and contextual inquiry.”

There are some points I strongly agree with, such as heuristic reviews being a way to send “false alarms.” He also feels that these reports are often opinion-based. The latter is why I don’t provide metric scores in mine, other than noting obvious defects that would be critical to a launch (as in, causing failure of some sort.) But, how I may feel about colors, layout, and even my own personal emotional response to a site is irrelevant unless they are paying to get my opinion.

What matters is how the end user reacts, responds, and the actions or inactions they take.

Are Usability Evaluations Worth the Investment?

I was happy to see that the reports I developed for my own use with clients, which are based on a software QA background, met Molich’s suggestions for how reports should be presented. Things like no more than 30 pages, provide a summary and always, always, always accentuate the positive.

However, reading the interview had me cheering one minute and frustrated the next.

For example, when he says, “In my experience, usability testing is very effective for showing your colleagues what a usability problem looks like in their interface. But I think the study results indicate that usability testing is ineffective for finding all usability problems in an interface. Our results also indicate that it’s ineffective even for finding all the serious usability problems in an interface.”

This is because one visitor’s “problem” may not be a problem for another. So often a product launch is the time when actual user feedback comes back that wasn’t picked up in QA testing or wrestled out of user personas and task analysis beforehand.

It is because to get to every freaking possible serious problem, we first have to define “serious” and then convince stakeholders, who have different ideas of what a “serious” show stopper is. It’s no fun being an “anal QA Engineer”, but trust me. You really don’t want to not have one of them on your side.

Such As…

Some of your target market may not mind entering their private home phone number on your form, while another several hundred thousand of them simply can’t bear the thought and require an alternative. I would deem this “serious” if the form is a sales lead form and is the sole reason for building your web site and you’re paying $3000 a month for Google ads, and invested $100,000 for search engine optimizattion.

Consider, too, that some companies have installed policies that prevent employees from entering their work email addresses, in an effort to control Spam. Ordering a part online or signing up for training may require managerial permission to enter company information of any type. How many manufacturing web sites consider online customer situations such as this, and design alternatives?

No matter what Molich says, the SEO/M industry benefits from including heuristic and cognitive walk through evaluations as part of their services. For any company, site owner or application developer investing in marketing to search engines, it pays off in the long run to make sure the site or application meets usability, accessibility and web design standards.

More Bang For Your Buck

It pays even more when someone is brought in who understands conversions and the emotional connections between user interface and human beings. You can throw all the money you want at Pay Per Click marketing but if the site doesn’t convince the visitor within 30 seconds that the click is relevant and “worth it”, that money is no better than cow pasture decoration.

Consider landing pages. Conversions are known to drop off from landing pages. Why? Because the visitor could not complete a task. Because the landing page didn’t confirm the visitor arrived at the right place. Because it was hard to figure out what to do or where to go next. These are common problems that go untested because some SEO/M’s think any landing page is going to work miracles.

So, are heuristic evaluations useful? Yes. If you can find someone willing to “mind meld” with your project for awhile and help you think about how to make it work better, you’ve taken a huge step forward towards long-term, user centered success.

Is Google User Centered?

That could depend on what you hope to do once you go to Google. If you want to search, that’s easy. They provide an obvious field to begin your search. But, if you want to use their 15 sites, or find 15 Google topic blogs or play with their seven tools and download sites, you need to book an appointment with the site to look around and find the stuff.

Or, you can just go to Chris McEvoy’s redesigned user centered Simply Google and learn how to use the site before dinner.

Other Cool Stuff

Take your Vitamin. This is a new classy web dev site with an impressive list of contributors.

Loren Baker’s Umdum blog and Umdum Directory. (Where do people get these names?)

Yet even more blog stuff from the V7N side of the planet. The latest trick seems to be that they kidnapped Peter DaVanzo for Blog V7N.

And finally, may I close with a respectful nod to Professor Sullivan. I’m into my 10th year too, but to accomplish what you did in your 10 years, I would need a wife too. Heh.

It's Not All About You, Mr. Search Engine
Dedication is Priceless and Worth Millions


Kim Krause Berg’s long background in web design, SEO and usability includes software application functional and user interface testing, accessibility, information architecture and persuasive design. She shared her passion for Usability and SEO through her site and private consulting at Cre8pc for 17 years. Kim founded Cre8asiteforums in 1998. In the fall of 2012 she sold her forums to Internet Marketing Ninjas and retired from private consulting to join their Executive Management team where she continues her work in usability testing, customer experience and conversions design.

My Online Course: Web Site Usability 101


American Society for Information Science and Technology (ASIS&T)

Information Architecture Institute

Usability Professionals Association (UPA)