Learning Outcomes
Understand
- usability can only be defined meaningfully relative to context
Remember
- efficiency
- effectiveness
- user satisfaction
- freedom from risk
- learnability
Apply
- Means of characterising context
- Choosing usability metrics
Measuring Usability
We talked about different conceptions of usability in Week 1 (see also Useful definitions for Usability and User Experience). Here are some ways in which you can measure usability with actual numbers:
Counts of:
- commands used (efficiency)
- repetitions of failed commands (effectiveness, learnability)
- runs of successes and of failures (effectiveness, risk)
- good and bad features recalled by users (user satisfaction)
- available commands not invoked (learnability)
- users preferring your system (user satisfaction)
- Percentage of tasks completed in time period (combined efficiency and effectiveness)
Counts or percentages of:
- errors (risk, effectiveness)
- superior competitor products on a measure
Ratios of
- successes to failures (effectiveness, risk)
- favourable to unfavourable comments (user satisfaction)
Times
- to complete a task (efficiency)
- spent in errors (efficiency, risk)
- spent using help or documentation (efficiency, learnability)
Frequencies
- of help and documentation use (learnability)
- of interfaces misleading users (context coverage, efficiency)
- of errors (risk)
- users needing to work around a problem (efficiency, context coverage)
- users disrupted from a work task (context)
- users losing control of the system (effectiveness)
- users expressing frustration or satisfaction (user satisfaction)
Activity: Think about how you might go about assessing the usability of the various activities required for the Student Life study
The paper by Hornbaek and Law cited below surveys many studies that used quantitative methods to measure different aspects of usability. That measures for the three aspects of efficiency, effectiveness, and user satisfaction are not well correlated, is actually not a bad thing. If two measures are correlated, this means that scores on these measures may be affected by a common underlying concept. (Sometimes, they are not). The lack of correlation shows that in order to understand how usable a system is, you need to consider all three aspects, you can’t easily reduce usability to just one or two.
Activity: Discuss what Hornbaek and Law’s findings mean in terms of reliability and validity.
- K. Hornbaek and E. L.-C. Law, “Meta-analysis of correlations among usability measures,” in CHI ’07: Proceedings of the SIGCHI conference on Human factors in computing systems, 2007, pp. 617–626.
Discussion Questions
- How do we know in what context a product will be used?
- What determines the cost/benefit tradeoff of a design decision?
- How important is learnability, and do we need to embrace complexity?
- Why would unforeseen conditions of use / contexts occur?
- What is the difference between designing for the laziest people, and designing to minimise risk of malfunction?
- How does usability relate to standard Software Testing frameworks as discussed in Software Engineering?