When performing usability research, participants are not always particularly forthcoming with absolute facts about an artefact in question. For instance, their preference for a device may be biased by the brand because it is popular, rather than for good usability features; whereas something as mundane as the appearance of the device may be cause for someone to criticise it though its usability is sound. A person with years of experience with a particular system may find little at fault having become normalised to its quirks – for them “if the system ain’t broke, don’t fix it”.
Providing participants with a questionnaire can leave you asking more questions than you started out with: Why did they answer in such a way? How did they interpret the question? What was their understanding of the question? Is the answer a true representation of the facts? How thorough is the questionnaire process? And many more…
Filming the participant performing a task is useful but lacks context. For instance: What is the point to the task? Why are they doing what they’re doing? What short-cut key was pressed, what does it do, and why was it used at this particular time? Is this the most efficient way of performing the task?
The cognitive interview, proposed by Fisher and Geiselman in 1992, is a technique used in psychology and has been adopted for use in police investigations to enhance information retrieval of eye witnesses. The process follows a line of questioning that takes the eye witness through the events of the crime from different perspectives – what they themselves witnessed, what they think other witnesses would have seen or even the criminals. The questioning may require the witness to recount the events in a different order, coming in at varying time points. The interviewer attempts to recreate the scene: time of day, sounds, weather, feelings… the witness is encouraged to recall every detail no matter how unimportant they might think it is.
We used a modified cognitive interview that attempts to record a participant’s activities with a task and that seeks to understand the activity, the reason behind the activity and the thought process and any accompanying information of interest that relates to the activity. Screen and audio capture software was used to record the participant’s on-screen activity whilst the participant voiced the purpose of the overall task, the sub actions that were performed to complete the task and any usability frustrations/issues encountered. Additionally, the participant was observed performing the task by an experienced usability analyst who asked non-leading questions relating to the activity.
We found this technique to be of great value for:
- Identifying usability issues with the system.
- Understanding how experienced users of the system perform their duties and the differences in how the duties are carried out by different users to perform the same actions.
- Understanding workflows.
- Understanding the process of data to inform the design of a new system.
- Extracting information from users providing insights to otherwise hidden knowledge.
- Recording information for further data mining.