A startup’s guide to doing research on the cheap – usability testing

User research is an important part of the product design process. It can help you make sure you’re building the right thing for the right people and to continue to learn from what you’ve built.

I divide companies into two types: those that are driven by users and those driven by features. User-driven companies pay intense attention to the needs of their users and build products for them. Feature-driven companies focus on what they can do with technology and build those products. You don’t want to be the second.

Needs can be expressed or unexpressed. Some needs that users express may actually be counter to what they want. Important needs may go unexpressed because users don’t know exactly how to express what they want, or the research tools are poor.

There are many types of tools that you can use for research: usability testing, focus groups, ideation sessions, ethnographic research, user feedback, survey research, eye tracking, etc. Too often product managers confuse the roles of these tools and end up with bad interpretations.

Research can cost from nearly nothing to tens of thousands of dollars. In this series, I’ll go through research tools I think provide the biggest bang for the buck for startups.

Today’s topic: usability testing.

You need to assess whether people can understand what your product will do. Ideally, you’d do this shortly after the user flows have been built. Bring people in 1 by 1 and watch them accomplish a series of key tasks. Have the designer and product manager sit with the interview subject.

I like to just give the subject a task and see what happens. (After all, no one is going to be sitting with them while they are actually using the product.) “Order a large pepperoni-mushroom pizza and have it delivered to your house.” Then watch as he goes through the user flow. Have the interview subject articulate their thought process as they perform the task. Where did his eye go first? What was the overall flow?

Only when the subject gets stuck should the interviewer intervene. “Where did you expect to find it?” “Maybe try scrolling further down the page?”

Although ordering a pizza is a relatively simple task, there are a number of steps: finding the restaurant nearest your house, selecting your pizza, adding toppings, special instructions, delivery information, payment, etc.

Some people like to do the testing one step of the process at a time. I prefer to do it all in one pass. There are two primary reasons:

  • Time to completion is a key part of user success. If you stop the subject at every step, you aren’t going to be get a sense of how long he would have taken with your user interface. If it would have taken him 10 minutes to order a pizza, your interface has failed.
  • You get a sense of what other flows the user might see. That’s not necessarily a bad thing, but it’s important to get that sense.

Once the first pass is done, you can repeat the test step-by-test and ask for detail on what was confusing or not.

Usability testing can be done with static images linked together, but I really prefer doing it with live or close to live code.

You can also test a couple of different flows.

It’s important that the subject is told at the outset that he is not the one being tested; it is the user interface that’s being tested. There is no right or wrong answer.

Whom you choose to participate is key. On HBO’s Silicon Valley, the initial response to the compression tool was amazement. All of their friends loved it. Except for Monica, who hated it. Then, we saw the user interface:

0

The people who loved it, the initial test group, were friends of engineers who were also engineers. They loved this type of interface because it gave them control over every little detail of the compression. Most users don’t want this.

You want to get people who don’t do tech for a living, if at all possible. You want to get people who are as close to your target audience as possible. If your product is designed to be used by senior citizens, don’t have millennials do the testing. (In this example, you’ll miss things like font sizes.)

Usability testing doesn’t need to be a formal process and doesn’t require special equipment.

You can recruit test subjects via craigslist. $75 to $150 for 40 minutes is a reasonable amount to pay. If you sell a product, some of this could be in service credit. For example, a $75 AmEx gift card and $50 to use on your app. I’m also comfortable asking friends and family, if they would be in the target user group.

Usability testing isn’t survey research; you don’t need to interview 1,000 people. With ten people, you should be able to identify the biggest obstacles in your interface.

ux: Lists should be presented in an easy-to-understand order

How could this experience be improved?

Notifications.
Notifications screen in iOS.

This is so bizarre that I can’t believe Apple missed it. The notification settings screen is in no discernbile order.

If you want to turn off or change notifications, you have to scroll through the list until you find what you’re looking for.

Continue reading “ux: Lists should be presented in an easy-to-understand order”

CNN testing QR codes on TV

CNN began testing QR codes on air this weekend to direct people to a site where they can help Japanese earthquake and tsunami victims.

CNN QR code

The code was easy to scan, even without pausing the broadcast. It worked fine from across the room. Just launch a barcode scanner and it will decode the URL and give you the option to open it in the browser. If you have a scanner, you can scan it off the image above. If not, click to go to the Impact Your World mobile site.

This is a great implementation of the often over-hyped QR-code technology.┬áPrint ads have occasionally featured QR codes which take you to an advertiser’s URL.

Some other applications I’d like to see:

  • In movie trailers. Scan it and it gets added to your movies to see list, possibly with a calendar entry dropped on the release date. Or an option to add to your Netflix queue for movies that are less interesting.
  • In TV promos. Scan it and it gets added to your recordings list. (The better implementation would be that the DVR itself would recognize a tag and prompt you.)
  • In TV commercials and on billboards. Scan to go to the advertiser’s site.
  • On CNN. Scan to get more information on a story.

QR-codes have a number of advantages over other technologies. They are free to generate, don’t require any hardware beyond a camera, hold more data than a standard bar code, are easy to replicate, work across a distance and have a built-in call-to-action (scan me!). QR-codes can also hold structured data; scanning the QR code on Rakeshagrawal.com will load up my contact information.

But it’s not the ultimate technology for every application. As much as people in the technology industry like to claim that one technology will take everything, that rarely happens.

Artwork at MoMA scanned with Google Goggles
Artwork at MoMA scanned with Google Goggles. In this case, it was a scan of a picture of the painting on flickr.

Here are some other applications where other technologies work just as well or better.

  • Identifying artwork. Many paintings in the MoMA’s collection can be identified just by taking a picture of it with Google Goggles. Let’s face it, QR codes are ugly. They’re designed to be easily readable by machines, not to be pretty. I should point out that the wacky kids in Dubai are trying to turn them into architecture with a QR-code hotel. Still, it’s not my taste in architecture.
  • Payments. Because they are easy to reproduce, QR codes (and bar codes in general) aren’t well suited for payment applications. They only work when you don’t really care about security.
  • Scanning books or products. One discussion that came up recently was using QR codes in stores like Barnes & Noble to identify whether a book is available in nook format. That’s overkill — you can do this perfectly well with the bar code already printed on the book. Heck, you can take a picture of the cover and that’ll work.
  • Print ads. URLs can be detected with simple OCR software. No need to clutter your creative with an ugly QR code. The key here is to use a simple font against a high contrast background and leave space around it. That’s a good practice anyway to ensure that human eyes can read it.
  • Checking in to a business. WiFi and GPS positioning do a reasonably good job of this without requiring businesses to do any extra work. This could be improved, but it works OK.
Plain text URLs work just fine in Google Goggles.
Plain text URLs work just fine in Google Goggles.