Amazon Mechanical Turk: Crowdsourcing Predictability of User Choice

Usability is one of the hottest topics in product development. In the field of usability studies, there are myriad ways to go about making a product more user-friendly. Some possible methods include card-sorting, creating a number of note cards and rearranging them in a sort of brainstorming session, creating a paper craft prototype of the product, and having experts on the product evaluate problems using a set of heuristics. These are all really useful, but a paper presented at the 2011 Special Interest Group on Computer-Human Interaction (SIGCHI) asks an important question: how do you predict if a user will want to actually use an interface (



One answer to that question may lie in a source that usability experts hadn’t previously considered, Amazon’s Mechanical Turk. If you’re not familiar with the service, it’s a marketplace for Human Intelligence Tasks (HITs). Someone creates a HIT, and then a “worker” can sign up to accomplish the HIT. When they do, they get paid. The HITs can take just about any form the creator needs (see image). Best of all, HITs can be done from home in your own time. All you need is a computer and internet connection.



The paper, presented by the University of Washington’s DUB Group, suggests that the task of determining if users will actually want to use a computer interface could be crowdsourced to HIT workers on Mechanical Turk. According to the researchers, the method of quantifying user choice used by industry, called A/B testing, is “out of reach for most researchers and small developers because it requires a large up-front investment and an existing userbase to deploy.” Limitations in time and money are a big sticking point in developing useful interfaces. Could Mechanical Turk be the answer?

The authors suggest that it may indeed be the solution. They suggest that utilizing HITs could give researchers the same resources normally only available in expensive laboratories, which may simply be out of reach for many developers. They call this “a way to crowdsource utility, or the study of preference.” Not only that, but online marketplaces could provide researchers with a way of sharing data and recreating results, an important aspect of provability and the scientific process.

The DUB Group concludes that the methods they suggest could lead to a “science of the human motive in computer interactions.” Perhaps the future of interface design will start with the use of online marketplaces, and all of us will be able to participate.


–Joel Zuber