A clinical decision support system for chronic pain management in primary care: usability testing and its relevance

Kalpana Nair

Department of Family Medicine, McMaster University, Hamilton, Canada

Raheleh Malaeekeh

Department of Family Medicine, McMaster University, Hamilton, Canada

Inge Schabort

Department of Family Medicine, McMaster University, Hamilton, Canada

Paul Taenzer

Department of Psychiatry, University of Calgary, Calgary, Canada

Arun Radhakrishnan

Department of Family and Community Medicine, University of Toronto, Toronto, Canada

Dale Guenter

Department of Family Medicine, McMaster University, Hamilton, Canada

Cite this article: Nair K, Malaeekeh R, Schabort I, Taenzer P, Radhakrishnan A, Guenter D. A clinical decision support system for chronic pain management in primary care: usability testing and its relevance. J Innov Health Inform. 2015;22(3):329-332.

Copyright © 2015 The Author(s). Published by BCS, The Chartered Institute for IT under Creative Commons license http://creativecommons.org/licenses/by/4.0/

Author address for correspondence:

Kalpana Nair, PhD

Infant and Child Health Lab

Department of Family Medicine, McMaster University

David Braley Health Sciences Centre

Hamilton, ON L8P 1H6, Canada

Email: nairk@mcmaster.ca


ABSTRACT

Background Clinical decision support systems (CDSSs) that are integrated into electronic medical records may be useful for encouraging practice change compliant with clinical practice guidelines.

Objective To engage end users to inform early phase CDSS development through a process of usability testing.

Method A sequential exploratory mixed method approach was used. Interprofessional clinician participants (seven in iteration 1 and six in iteration 2) were asked to ‘think aloud’ while performing various tasks on the CDSS and then complete the System Usability Scale (SUS). Changes were made to the CDSS after each iteration.

Results Barriers and facilitators were identified: systemic; user interface (most numerous barriers); content (most numerous facilitators) and technical. The mean SUS score was 81.1 (SD = 12.02) in iteration 1 and 70.40 (SD = 6.78) in iteration 2 (p > 0.05).

Conclusions Qualitative data from usability testing were valuable in the CDSS development process. SUS scores were of limited value at this development stage.

Keywords: chronic pain, clinical decision support system (CDSS), clinical practice guideline, electronic medical record, primary care


INTRODUCTION

Clinical decision support systems (CDSSs) that are integrated into electronic medical records hold promise for facilitating better uptake of clinical practice guidelines in the practice setting. 1 Unfortunately, poor clinician usability has proven to be a barrier in the uptake and success of many health information technologies, including CDSSs.

Usability refers to ease of use of software technology and the user interface (UI).2 Attributes commonly described include learnability, efficiency, effectiveness, usefulness, accessibility and user satisfaction.2 Testing methods that include early involvement of current or potential users of the system and iterative evaluation may improve system usability.1,2

The objective of this small study was to engage end users in an iterative software development process to optimise the usability of a CDSS for chronic pain management. The purpose of the McMaster Pain Assistant (MPA) is to support inter-professional primary care management of chronic pain and standardise care by influencing clinical decisions for low back pain and neuropathic pain.


METHODS

MPA CDSS

The Open Source Clinical Application Resource (OSCAR) electronic medical record is the platform for the MPA CDSS. The MPA is a suite of electronic forms that are integrated with the workspace, workflow and databases contained within the electronic medical record. The MPA includes ‘encounter guides’ that actively and passively assist a clinician through a patient visit for low back pain, neuropathic pain and opioid management based on clinical practice guideline recommendations. The MPA provides data fields for recording assessment information during the encounter; automatic filling of data fields where information exists elsewhere in the patient’s record; automatic extraction from data fields (such as scores from questionnaires) into a tracking table and graphic presentation; web links to educational materials for both clinicians and patients; and brief video presentations to assist clinicians with difficult discussions with patients.

When this study was initiated, all of these functions were available at either partial or full level of function. The initial development requirements of the MPA functionality were based on focus group needs, assessments of inter-professional end-user clinicians, as well as an expert panel consisting of e-health academics, programmers, pain clinicians and knowledge translation experts.

Sample and Setting

An academic family medicine centre with 2 clinics, 30 physicians, 8 nurse practitioners, 70 family medicine residents and a diverse allied health group. A convenience sample of potential users of the MPA was used and included family physicians, family medicine residents and nurse practitioners. The target sample size was 5–8 participants in each iteration, based on the literature that suggests that 80% of deficiencies will be found in samples of 4–5 participants.3

Data Collection

Two iterations of usability testing were performed, 2 months apart, with an identical format and different participants. A demonstration version of the CDSS was used for testing, and participants received no training in its use prior to working through a scenario. Three different patient case scenarios were provided: medication renewal, diagnosis of low back pain and monitoring of neuropathic pain. Each study participant was observed separately, and sessions were facilitated by one of the authors (RM) and observed in person or on camera by a team that included co-investigators, research staff, business analyst and the system software developer. Screencast video and audio were recorded and transcribed, and all the observers wrote field notes about what they observed and what the participant reported during the exercise.

The System Usability Survey (SUS) was completed by each participant immediately following the exercise. The SUS consists of 10 items, each scored on a 5-point Likert scale that ranges from 1 – strongly disagree to 5 – strongly agree and provides an overall score of system usability.3,4 Scoring involves subtracting 1 from all odd items, and subtracting all even numbered item responses from 5, which scales each item from 0 to 4. The total is multiplied by 2.5 to provide a score out of 100, which is interpreted as a percentile ranking and not as a percentage. SUS can also provide sub-scores for learnability and usability.

Data Analysis and Use in Development

For each iteration, all the observer notes and SUS scores were discussed by observers and then by the research team. Directed content analysis was used, and the number of barriers and facilitators of usability was determined and used to generate a priority list for system revision. The mean, SD and p-value between the two iterations were calculated for SUS scores. The MPA was modified following round 1 of testing and the revised version used in iteration 2. Further modifications followed the second round of testing.


RESULTS

Participants

Seven clinicians (three family physicians, three nurse practitioners and one family medicine resident) participated in round 1 testing. Six new clinicians (three family physicians, two nurse practitioners and one family medicine resident) participated in round 2.

Qualitative Data

Transcribed audio recordings and observer field notes were analysed and resulted in the following categories of usability barriers and facilitators: system, UI, content and technical (see Table 1). Most identified barriers were related to the UI in both rounds, and this and technical were the areas that were modified most extensively between testing rounds. For example, titles of resources were identified as ambiguous in round 1. Between rounds, these were modified to be clearer, and round 2 demonstrated that this issue was largely resolved.

Most identified facilitators were related to content, and many were related to UI. For example, participants felt that the added feature of a timeline that correlated medication to pain screening scores could help track changes in pain scores based on medications taken. Changes between round 1 and 2 testing were not analysed for statistical significance.

SUS Score

The overall SUS score decreased from 81.10 (±12.02) out of a possible 100 points in round 1, to 70.40 (±6.78) in round 2. This decrease in score was not statistically significant (p = 0.86; Table 2). Subscale scores were calculated for learnability (SUS items 1 and 4) with round 1 mean score 3.57 (SD = 1.72) and round 2 3.16 (SD = 1.34 and p = 0.78). Subscale scores were also calculated for usability (SUS remaining items) with round 1 mean score 3.16 (SD = 1.40) and round 2 2.73 (SD = 1.06 and p = 0.69).

Table 1. Number of usability barriers of the MPA

Table 2. SUS scores in rounds 1 and 2


DISCUSSION

Although well established within the software and application development arenas, usability testing is less commonplace within the health care context. Trafton et al.5 carried out a process similar to ours, testing the usability of a CDSS for opioid therapy for chronic, non-cancer pain by primary care providers. ‘Think aloud’ protocols in combination with SUS, interviews and log files were used to improve the system design and the UI.4 Similarly, this process was useful in the early development of the MPA CDSS.

Our testing involved a small number of two different groups of participants over two iterations. This meant that SUS scores, although reassuringly situated in the acceptable range by industry standards (i.e. above 68), were not statistically useful. In addition, applying the SUS to a broad and complex clinical task made it difficult to interpret the meaning of the score. Conventions of usability testing support our small sample,3 but we were aware that ‘outliers’ were influencing our results (i.e. participants with less computer experience versus generally comfortable participants).

This being said that the overall testing process was highly beneficial to design and development for several reasons. Testing occurred early in development allowing for changes in design elements. The qualitative ‘think aloud’ data provided specific critique and suggestions. Observers and data reviewers were from diverse roles enabling a broadly informed analysis of the findings. The software system developer was present and engaged in all the steps of the testing, supporting an ‘in the moment’ integration of user observation with ideas for modification, especially as related to UI and technical aspects. The overall process strongly influenced the ongoing system design, which continued following the second round of testing, and will be evaluated in a larger study of the final and fully implemented CDSS.

Acknowledgement

The authors would like to thank all the participants for their time and thoughts.

Competing Interests: None

Funding: This work was supported by the Department of Family Medicine, McMaster University, Hamilton, Canada, and the Lawson Health Research Institute, London, Ontario, Canada.


REFERENCES

1. Johnson CM, Johnson TR and Zhang J. A user-centered framework for redesigning health care interfaces. Journal of Biomedical Informatics 2005;38(1):75–87. http://dx.doi.org/10.1016/j.jbi.2004.11.005. PMid:15694887.

2. Bastien J. Usability testing: a review of some methodological and technical aspects of the method. International Journal of Medical Informatics 2010;79(4):e18–23. http://dx.doi.org/10.1016/j.ijmedinf.2008.12.004. PMid:19345139.

3. Rubin J and Chisnell D (Eds). Handbook of Usability Testing, 2nd edition. Indianapolis: Wiley Publishing, 2008.

4. Brooke J. SUS: a retrospective. Journal of Usability Studies 2013;8(2):29–40.

5. Trafton J, Martins S, Michel M, Lewis E, Wang D, Combs A et al. Evaluation of the acceptability and usability of a decision support system to encourage safe and effective use of opioid therapy for chronic, noncancer pain by primary care providers. Pain Medicine (Malden, Mass) 2010;11(4):575–85. http://dx.doi.org/10.1111/j.1526-4637.2010.00818.x. PMid:20202142.

Refbacks

  • There are currently no refbacks.


This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT