Evaluating the usability of an interactive, bi-lingual, touchscreen-enabled breastfeeding educational programme: application of Nielson’s heuristics

Ashish Joshi

CUNY School of Public Health, City University of New York, NY, USA

Douglas M Puricelli Perin

Center for Global Health and Development, College of Public Health, UNMC, Omaha, NE, USA

Chioma Amadi

Center for Global Health and Development, College of Public Health, UNMC, Omaha, NE, USA

Kate Trout

Center for Global Health and Development, College of Public Health, UNMC, Omaha, NE, USA

Cite this article: Joshi A, Puricelli Perin DM, Amadi C, Trout K. Evaluating the usability of an interactive, bi-lingual, touchscreen-enabled breastfeeding educational programme: application of Nielson’s heuristics. J Innov Health Inform. 2015;22(2):265–274.

Copyright © 2015 The Author(s). Published by BCS, The Chartered Institute for IT under Creative Commons license http://creativecommons.org/licenses/by/4.0/

Author address for correspondence:

Ashish Joshi

Assistant Dean of Students and Associate Professor

CUNY School of Public Health

City University of New York,

New York, NY, USA

Email: ashish.joshi@sph.cuny.edu


ABSTRACT

The study purpose was to conduct heuristic evaluation of an interactive, bilingual touchscreen-enabled breastfeeding educational programme for Hispanic women living in rural settings in Nebraska. Three raters conducted the evaluation during May 2013 using principles of Nielson’s heuristics. A total of 271 screens were evaluated and included: interface (n = 5), programme sections (n = 223) and educational content (n = 43). A total of 97 heuristic violations were identified and were mostly related to interface (8 violations/5 screens) and programme components (89 violations/266 screens). The most common heuristic violations reported were recognition rather than recall (62%, n = 60), consistency and standards (14%, n = 14) and match between the system and real world (9%, n = 9). Majority of the heuristic violations had minor usability issues (73%, n = 71). The only grade 4 heuristic violation reported was due to the visibility of system status in the assessment modules. The results demonstrated that the system was more consistent with Nielsen’s usability heuristics. With Nielsen’s usability heuristics, it is possible to identify problems in a timely manner, and help facilitate the identification and prioritisation of problems needing urgent attention at an earlier stage before the final deployment of the system.

Keywords: breastfeeding, educational programme, heuristics, human–computer interaction


INTRODUCTION

Breastfeeding has been recognised as an optimal method to nourish an infant as it provides complete nutrition necessary for its full physical and mental development.13 Breastfeeding protects against common infections and diseases throughout childhood with effects lasting even into adulthood. Babies who are not fully breastfed for the first 34 months of age are at higher risk of health problems, such as gastroenteritis,1,4,5 respiratory infection,1 otitis media6 and urinary tract infection. 6 Benefits of breastfeeding for the mother include a rapid return of post-partum uterine tone and post-partum weight loss, higher bone mass density after menopause, delay of ovulation and decreased risk of breast and ovarian and endometrial cancers.79

Introduction of complementary foods should not happen before the sixth month and breastfeeding should continue at least until 12 months of age.3 According to the Centers of Disease Control and Prevention, 74.6% of breastfed babies are ever breastfed in the United States, 44.3% breastfed at 6 months and only 23.8% continue breastfeeding to 12 months.2 In Nebraska, where approximately 30% of the population lives in rural areas,10 the number of ever breastfed and exclusively breastfed babies at 6 months stay below the national average at 72.8% and 13.4%, respectively.2 Healthy People 2020 (HP2020) aim to increase the proportion of ever breastfed babies to 81.9%, 60.6% at 6 months and 34.1% at 12 months.4 The current outcomes are far behind public health goals.

Socio-economic, cultural, attitudinal and familial factors are associated with breastfeeding practices.11,12 Disparities in breastfeeding practices exist across different groups,7,8 and a recent review has indicated the need for more research on culturally tailored interventions to achieve the breastfeeding goals established by HP2020.13 Among Hispanic mothers, perceptions of infant rejection to breastfeeding and milk insufficiency are often reported,14 and formula supplementation is practised among one in three mothers.15 Previous research suggests that initiation of breastfeeding may be more frequent among urban women (59%) compared to that among rural women (49%).16 In urban settings, several longitudinal studies have prospectively examined breastfeeding initiation and discontinuation.1722 Among urban women, breastfeeding initiation and continuation may be influenced by several factors including participation in the WIC programme,17 support from the health system,1921 maternal depression21 and return to work or school.18,23

A number of health information technology (IT) solutions assist health care professionals in providing efficient quality care. Technology innovations facilitate presenting culturally relevant and tailored health information24,25 There has been increasing trend of using technologies for supporting health behaviour change such as enhancing physical activity,2631 healthy diet,3234 smoking35,36 and self-regulation of emotions.37

Health IT evaluation is complex as it intends to serve various functions. Evaluation involves hardware and information processes in a given environment.38,39 Lack of attention to health IT evaluation reflects an inability to achieve system efficiency, effectiveness and satisfaction. 38,39 There are unique methodological challenges in evaluating how populations use and navigate new technologies. Usability challenges have to be met in programme development and should meet all users’ needs. The errors generated due to complex nature and shortcomings of health care applications have been emphasised in various studies.38 Many health IT usability studies have been conducted to explore usability requirements, discover usability problems and design solutions.38,39 The poor design of interactive health care systems increase their complexity thereby hindering their utilisation and uptake by the users.38,39 Evaluating health technology interventions helps in identifying usability issues such as ease of navigation so as to design and develop solutions in a timely and an effective manner.40

Several methods are available to assess and improve the usability of interactive computer enabled applications. Among the expert-based methods, heuristic evaluation is commonly used.41 Nielson’s heuristics are a set of usability engineering principles developed to identify issues in user interface design and involves analysis of the interface. 42,43 Ten essential criteria constitute the Nielson’s heuristics and they include visibility of system status, match between system and the real world, user control and freedom, consistency and standards, error prevention, recognition rather than recall, flexibility and efficiency of use, aesthetic and minimalist design, recovery from errors and help and documentation. 42 Severity indexes ranging from negligible usability issues represented by zero (I don’t agree that this is a usability problem at all) to one (Cosmetic problem only: need not be fixed unless extra time is available on project), two (Minor usability problem: fixing this should be given low priority), three (Major usability problem: important to fix, so should be given high priority) and four (Usability catastrophe: imperative to fix this before product can be released) are utilised in assessing the extent to which such errors can hamper utilisation of the system.42 The majority of the errors can be identified in a systematic process, such as instructions and functionality problems, simplified representation and improved labeling.43 Heuristic evaluations require fewer resources than other methods to predict major usability problems.44,45 According to Nielson’s heuristics, a minimum of three and maximum of five raters are required to conduct a heuristic evaluation.45 Additional raters might not necessarily discover useful information.45

The objective of this study was to conduct a heuristic evaluation of an interactive, bilingual, touchscreen-enabled breastfeeding educational programme for Hispanic women living in rural settings.


METHODS

Conceptual framework of Patient Education and Motivation Tool

We modified Patient Education and Motivation Tool (PEMT) to design and develop an interactive, bilingual, touchscreen computer based, breastfeeding educational support programme integrating a variety of cognitive-behavioural theories. PEMT facilitates health information and messages to be adapted depending on the psychosocial elements including attitude, self-efficacy, expectations, personal norms and social influences.4649 The entire educational content is broken down into a series of modules, each module into sub-modules and each sub- module into a series of educational messages. Each message is then presented using various multimedia formats (such as audio, video, text and images). The primary objective of the PEMT is to present health information in an interactive tailored manner considering multiple factors influencing health status and health behaviours. The computer-based breastfeeding educational support programme aims to provide modular, culturally relevant, bi- lingual (English and Spanish) education tailored to the needs of the mothers (Figures 1 and 2). The details of the proposed breastfeeding educational support programme have been outlined in a prior study.

Figure 1 Breastfeeding programme interface

Figure 2 An interactive breastfeeding educational programme

The breastfeeding educational support programme is delivered with a kiosk containing a touchscreen-enabled computer with the breast-feeding educational support programme on a mobile cart in an outpatient clinic.

Evaluation method

Three usability raters (AJ, DP and CA) were involved in the evaluation of the breastfeeding educational support programme. Two usability experts (DP and CA) used Nielsen’s heuristics while reviewing the user interface in May 2013. One usability expert (DP) was a research assistant with over one year of experience in public health informatics and evaluation of health technologies. The other (CA) was a biological chemist and also a research assistant in the area of public health informatics and consumer health technologies. Both individuals worked at the Mobile Emerging Technologies and Population Health Outcomes Research laboratory in the College of Public Health at the University of Nebraska Medical Center. Both evaluators individually reviewed the user interface of the breastfeeding support programme and generated a list of heuristic violations. A severity scale ranging from 0 (no usability problem) to 4 (usability catastrophe) was applied to all violations [18]. A third rater (AJ), who is an expert in the area of evaluation of health technologies, reviewed and resolved any disagreements for any differences that occurred to identify the potential heuristic violations. The heuristic evaluations were carried out in the interface, programme sections and interactive educational modules. The interface included the navigational features and the various buttons that were implemented for the users to self-pace an interactive computer-based breastfeeding programme (Figure 1). The programme sections included the introductory screen, table of contents and four different modules that gathered information about the user’s socio-demographics, self-efficacy, knowledge and Breastfeeding Attrition Prediction Tool (BAPT). The bi-lingual interactive breastfeeding educational content (English and Spanish) presented in the form of text, audio and images was also evaluated (Figure 2). The study was approved by the University of Nebraska Medical Center Institutional Review Board (IRB protocol #430-12-EP).

Statistical analysis

Descriptive analysis was performed to report means and standard deviations for the continuous variables, while frequency analysis was performed for the categorical variables. All results of the analysis were performed using SAS version 9.1.


RESULTS

A total of 271 screens were evaluated, and 97 heuristics violations were observed. The various screen types included (a) interface (n = 5), (b) programme screens included socio- demographics (n = 9), self-efficacy assessment (n = 23), knowledge assessment (n = 90), BAPT (n = 60) and (c) educational content screens (n = 43). A total of 97 violations were identified, and majority of them were related to the interface (8 violations/5 screens) and programme modules (89 violations/266 screens). Other heuristic violations were related to socio-demographics (7 violations/9 screens), followed by knowledge component (64 violations/90 screens), self- efficacy (14 violations/23 screens) and BAPT (3 violations/60 screens). There was only violation reported for the interactive computer-based breastfeeding educational content (1 violation/43 screens; Table 1).

Table 1: Severity index categories for various heuristic evaluations

Overall heuristic violations

The most important heuristic violations included recognition rather than recall (62%, n = 60), followed by consistency and standards (14%, n = 14) and match between the system and real world (9%, n = 9; Figure 3). No violations were reported for ‘flexibility and efficiency of use’ and ‘aesthetic and minimalist design’.

Heuristic violations by programme components

The majority of the heuristic violations were reported for the assessment component (n = 88). For the assessment component, the most common heuristic violations reported were recognition rather than recall (67%; n = 60), while in the interface component, it was help and documentation (38%; n = 3), and for the educational component, it was user control and freedom (100%; n = 1). The number of heuristic violations for assessment, interface and education components is described in Figure 4.

Severity of heuristic violations

The average severity index of overall heuristic violations was 2 (SD = 0.88). Majority of the violations had minor usability issues (73%, n = 71). There were very few violations that were in the severity grade of 3 (n = 5) or 4 (n = 5). Majority of the violations were in the severity grade of 2 (n = 71; Table 1).

Figure 3 Overall number of heuristic violations

Figure 4 Number of heuristics violations by programme components

Severity of heuristic evaluations by programme components

The highest number of severe heuristic violations (grade 4) was seen among the assessment programme component (n = 5). No catastrophic issues were encountered for the interface. Sometimes clicking on the screen to receive more information would not achieve the desired result. The major usability violations were user control and freedom (60%, n = 3). Cosmetic issues represented 17% (n = 16) of the violations, and the most important categories included match between system and real world (44%, n = 7) and consistency and standards (31%, n = 5). Other suggestions included specifying a clearer indication in the title that more than one option can be selected. The programme was designed in English and Spanish, but the control panel does not offer the option to change languages to user. The assessment component had the majority of the catastrophic violations that were related to the visibility of system status (n = 5) compared to the educational module that had one catastrophic violation related to the user control and freedom (Table 2).

‘I’ represents interface, ‘A’ represents assessments and ‘E’ represents educational modules.

The common reasons of heuristic violations have been outlined in Table 3. The suggestions provided give an insight into the various possible changes that can be incorporated to the proposed interactive, bi-lingual touchscreen-enabled breastfeeding educational support programme for Hispanic women living in rural settings.

Table 2: Severity of heuristic evaluations by programme components

Table 3: Reasons for heuristic violations


DISCUSSION

The results of the heuristic evaluation reported a total of 97 violations based on the 271 different screen types evaluated in an interactive, bi-lingual touchscreen-enabled breastfeeding educational programme. The heuristic evaluation allows the software developers to identify and correct errors within each component of the IT system in a timely manner,40 and can be used to identify strengths in the design of the system. The violations identified in this study were related to the (a) interface (8 violations/5 screens), (b) socio-demographics (7 violations/9 screens), (c) self-efficacy (14 violations/23 screens), knowledge (64 violations/90 screens), (d) BAPT (3 violations/60 screens) and (e) educational content (1 violation/ 43 screens). The maximum number of heuristic violations included: recognition rather than recall (62%, n = 60) followed by consistency and standards (14%, n = 14), match between the system and real world (9%, n = 9), visibility of system status (5%, n = 5), help and documentation (4%, n = 4), user control and freedom (3%, n = 3), error prevention (2%, n = 2) and helping users recognise, diagnose and recover from errors (1%, n = 1). No violations were reported for ‘flexibility and efficiency of use’ and ‘aesthetic and minimalist design’. Identifying these heuristic violations will allow the conservation of resources compared to other methods to predict major usability problems.44,45 Additionally, heuristic evaluation is a better predictor of problems that are encountered by endusers and also identifies more severe usability problems.50,51

A severity rating ranging from 0 (no usability problem) to 4 (catastrophic) was utilised in classifying the degree of errors. By the identification of the severity rating, developers are able to prioritise problems needing urgent attention at an earlier stage before the final deployment of the system. The average severity index for the 97 violations identified was 2 (SD = 0.88). The violations due to recognition rather than recall were mainly restricted to the knowledge assessment module and related to the lack of specification of the expected answer to the questions asked before presenting the feedback. However, all the violations had a rating of 2 implying minor cosmetic problems. The raters felt that additional feedback should be provided on the same screen when a certain response was made to the question, as chosen by the user. This would help the user to make a clear distinction between the correct and the incorrect response. This is consistent with literature, where raters using Nielson’s heuristics are able to identify instructions and functionality problems, simplified representation and improved labeling.43 Majority of the violations due to consistency and standards were reported in the self-efficacy assessment module (8%, n = 8), followed by the knowledge module (4%, n = 4) and socio-demographics (1%, n = 1).

Identification of these violations in a systematic process using heuristic evaluation is suitable for a wide range of IT systems, and can be used in the implementation and development process. Some of the common heuristic violation findings included rephrasing of the words and the use of phrases or functions, which are more familiar to the user than system oriented and specifying a clearer indication in the title that more than one option can be selected. The violations due to match between the system and the real world were mainly based on the preference of users having a drop down menu for selecting age rather than having addition and subtraction symbols to alter their age. Violations due to visibility of system status were related to failure of the system to load the desired page when the suggested icon is clicked evaluation results encountered very few catastrophic (n = 5) and major usability violation (n = 1) in the breastfeeding educational support programme. The vast majority of the violations represented cosmetic issues, mainly regarding recognition than recall. In addition, some minor usability problems were identified related to match between system and real world, and consistency and standards. It would be important to correct these heuristic violations before the final deployment of the breastfeeding educational programme. Correcting the violations found in such evaluations can increase system efficiency, effectiveness and satisfaction among users.38,39

There are several limitations of using heuristic evaluation compared to other usability engineering methods. Evaluators detected different sets of problems, and exist for both novice and experienced evaluators.52 Therefore, it is beneficial to have an additional evaluator to resolve any disagreements between the raters, both in the detection of usability problems as well as the assessment of problem severity. The method heavily relies on the skills and expertise of the usability professionals, and these professionals might lack domain knowledge and may overlook domain related usability problems. One way to overcome this obstacle is to employ evaluators, known as double experts, who possess both usability and domain knowledge.53 It is essential to have a combination of these experts while evaluating computer-mediated patient education programmes in the health care environment, or else there is a risk of producing a mismatch between the system and the real world.


CONCLUSION

The results of the existing heuristic evaluation demonstrated that the system was more consistent with Nielsen’s usability heuristics. The ability to identify problems in a timely manner makes this method particularly well suited to the iterative design process. It is important to be considerate about the possible users of the system when conducting heuristic evaluations so that the system can speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms and information, and these words, phrases and concepts should appear in a natural and logical order. This will help us facilitate the identification and prioritisation of problems needing urgent attention at an earlier stage before the final deployment of the system.


ACKNOWLEDGEMENT

This study was funded by the Agency for Health care Research Quality # HS021321-01A1.


REFERENCES

1. World Health Organization. Global Strategy for Infant and Young Child Feeding. World Health Organization, 2003.

2. Gartner LM, Morton J, Lawrence RA, Naylor AJ, O’Hare D, Schanler RJ et al. Breastfeeding and the use of human milk. Pediatrics 2005;115:496–506. http://dx.doi.org/10.1542/ peds.2004-2491. PMid:15687461.

3. United States Department of Health and Human Services. Maternal, Infant and Child Health. MICH-21. http://healthypeople.gov/2020/topicsobjectives2020/overview.aspx?topicid=26. Accessed 31 March 2013.

4. Centers for Disease Control and Prevention. Breastfeeding Report Card 2011. Atlanta, GA: Division of Nutrition, Physical Activity, and Obesity, 2011. http://www.cdc.gov/breastfeeding/pdf/2011breastfeedingreportcard.pdf. Accessed 21 January 2013.

5. Wilhelm SL, Rodehorst TK, Stepans MBF, Hertzog M and Berens C. Influence of intention and self-efficacy levels on duration of breastfeeding for Midwest rural mothers. Applied Nursing Research 2008; 21(3):123–30. http://dx.doi.org/10.1016/j.apnr.2006.10.005. PMid:18684405.

6. Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. International Journal of Medical Informatics 2009; 78:340–53. http://dx.doi.org/10.1016/j.ijmedinf.2008.10.002. PMid:19046928.

7. Jones R, Pearson J, McGregor S, Cawsey AJ, Barret A, Craig N et al. Randomized trial of personalized computer based information for cancer patients. BMJ, 1999;319:1241–7.

8. Kreuter M, Farrell D, Olevitch L and Brennan L. Tailoring Health Messages. Customizing Communication with Computer Technology. London: Lawrence Earlbaum Associates, 2000. http://dx.doi.org/10.1136/bmj.319.7219.1241. PMid:10550090; PMCid:PMC28275

9. Kreuter MW and Wray RJ. Tailored and targeted health communication: strategies for enhancing information relevance. American Journal of Health Behavior 2003;27:S227–32.

10. Fogg BJ, Grudin J, Nielsen J and Card S. Persuasive Technology: Using Computers to Change What We Think and Do. Science & Technology Books, 2002.

11. Kientz, Julie A et al Heuristic evaluation of persuasive health technologies. Proceedings of the 1st ACM International Health Informatics Symposium, 2010, ACM.

12. Williams P, Nicholas D, Huntington P, and McLean F. Surfing for health: user evaluation of a health information website. Part one: background and literature review. Health Information & Libraries Journal 2002;19(2):98–108. http://dx.doi.org/10.1046/j.1471-1842.2002.00374_2.x. PMid:12389607.

13. Nielsen J. Heuristic evaluation. Nielsen J and Mack RL (Eds.) Usability Inspection Methods. New York, NY: John Wiley & Sons, 1994, pp. 25–62.

14. Nielsen J. Usability Engineering. San Diego, CA: Academic Press, 1993.

15. Turner-Bowker, Diane M et al Heuristic evaluation and usability testing of a computerized patient-reported outcomes survey for headache sufferers. Telemedicine and e-Health 2011;17:40–5. http://dx.doi.org/10.1089/tmj.2010.0114. PMid:21214341; PMCid:PMC3064878.

16. Joshi A, Arora M, Dai L, Price K, Vizer L and Sears A. Usability of a patient education and motivation tool using heuristic evaluation. Journal of Medical Internet Research 2009;11:e47.

17. Yen PY and Bakken S. Review of health information technology usability study methodologies. Journal of American Medical Informatics Association 2012;19:413–22. http://dx.doi.org/10.1136/amiajnl-2010-000020. PMid:21828224; PMCid:PMC3341772.

18. Nielsen J. How to conduct a heuristic evaluation. On the World Wide Web (1994).

19. Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review 1956;63(2):81–97. doi: 10.1037/h0043158. http://dx.doi.org/10.1037/h0043158.

20. Duffy T and Jonassen DH. Constructivism and the Technology of Instruction: A Conversation. Hillsdale, N.J: Lawrence Erlbaum Associates Publishers, 1992.

21. Spiro R. The Library of the University of Illinois-Urbana Champaign. 1988. [2013-10-29]. Cognitive flexibility theory: advanced knowledge acquisition in ill-structured domains. https://www.ideals.illinois.edu/bitstream/handle/2142/18011/ctrstreadtechrepv01988i00441_opt.pdf?sequence=1. Accessed 12 February 2013.

22. Sweller J. Online library Wiley. 1988. Cognitive load during problem solving: effects on learning http://onlinelibrary.wiley.com/store/10.1207/s15516709cog1202_4/asset/s15516709cog1202_4.pdf?v=1&t=hnd7f8uy&s=fe685a8b2c5fb070c8097c15693a7fa1d29a1f92. Accessed 12 February 2013.

23. Boeree C. Personality theories. 1998. [2013-10-21]. B.F. Skinner. http://webspace.ship.edu/cgboer/skinner.html. Accessed 12 February 2013.

24. McDonald EM, Solomon B, Shields W, Serwint JR, Jacobsen H, Weaver NL et al. Evaluation of kiosk-based tailoring to promote household safety behaviors in an urban pediatric primary care practice. Patient Education and Counseling. 2005;58(2):168–81.

25. Petty RT and Cacioppo JT. Attitudes and Persuasion: Classic and Contemporary Approaches. Dubuque, Iowa:W. C. Brown, 1981.

26. Anderson I, Maitland J, Sherwood S et al. Shakra: tracking and sharing daily activity levels with unaugmented mobile phones. Mobile networks and applications 2007;12(2–3):185–9. http://dx.doi.org/10.1007/s11036-007-0011-7.

27. Consolvo S, Everitt KM, Smith I and Landay JA. Design requirements for technologies that encourage physical activity. Proceedings of CHI, 2006, ACM, pp. 457–66.

28. Consolvo S, Klasnja P, McDonald DW et al. Flowers or a robot army?: encouraging awareness and activity with personal, mobile displays. Proceedings of the 10th International conference on Ubiquitous computing, 2008, ACM, pp. 54–63.

29. Lin JL, Mamykina L, Lindtner S, Delajoux G and Strub HB. Fish ‘n’ steps: encouraging physical activity with an interactive computer game. Proceedings of Ubicomp, 2006, Springer, pp. 261–78.

30. King AC, Ahn DK, Oliveira BM, Atienza AA, Castro CM and Gardner CD. Promoting physical activity through hand-held computer technology. American Journal of Preventive Medicine 2008;34(2):138–42. http://dx.doi.org/10.1016/j.amepre.2007.09.025. PMid:18201644; PMCid:PMC2715220.

31. Wieland LS, Falzon L, Sciamanna CN, Trudeau KJ, Brodney S, Schwartz JE et al. Interactive computer-based interventions for weight loss or weight maintenance in overweight or obese people. Cochrane Database of Systematic Reviews 2012;8: CD007675.

32. Denning T, Andrew A, Chaudhri R et al. BALANCE: towards a usable pervasive wellness application with accurate activity inference. Proceedings of the 10th Workshop on Mobile Computing Systems & Applications 2009, ACM, pp. 1–6.

33. Gasser R, Brodbeck D, Degen M, Luthinger J, Wyss R and Reichlin S. Persuasiveness of a mobile lifestyle coaching application using social facilitation. Proceedings of the 1st International Conference on Persuasive Technology, 2006, Springer Verlag, pp. 27–38.

34. Lee G, Tsai C, Grisworld WG, Raab F and Patrick K. PmEB: a mobile phone application for monitoring caloric balance. CHI ’06 extended abstracts on Human Factors in computing systems 2006, ACM, pp. 1013–8.

35. Devries KM, Kenward MG and Free CJ. Preventing smoking relapse using text messages: analysis of data from the txt2stop trial. Nicotine & Tobacco Research 2013;15(1):77–82. http://dx.doi.org/10.1093/ntr/nts086. PMid:22523120.

36. Michie S, Brown J, Geraghty AW, Miller S, Yardley L, Gardner B et al. Development of StopAdvisor: a theory-based interactive internet-based smoking cessation intervention. Translational Behavioral Medicine 2012;2(3):263–75. http://dx.doi.org/10.1007/s13142-012-0135-6. PMid:24073123; PMCid:PMC3717907.

37. Morris M, Kathawala Q, Leen TK et al. Mobile therapy: case study evaluations of a cell phone application for emotional self awareness. Journal of Medical Internet Research 2010;12(2):e10.

38. Yen PY and Bakken S. Review of health information technology usability study methodologies. Journal of American Medical Informatics Association 2012;19(3):413–22. http://dx.doi.org/10.1136/amiajnl-2010-000020. PMid:21828224; PMCid:PMC3341772.

39. Minshall S. A review of health care information system usability and safety. Studies in Health Technology and Informatics 2013;183:151–6. PMid:23388273.

40. Segall N, Saville JG, L’Engle P, Carlson B, Wright MC, Schulman K et al. Usability evaluation of a personal health record. Proceedings of the AMIA Annual Symposium, 2011, pp. 1233–42.

41. Nielsen J. Usability Engineering. Boston: Academic Press, 1993.

42. Tang Z, Johnson TR, Tindall RD and Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemedicine Journal and E-Health 2006;12(1):24–34. http://dx.doi.org/10.1089/tmj.2006.12.24. PMid:16478410.

43. Joshi A, Arora M, Dai L, Price K, Vizer L and Sears A. Usability of a patient education and motivation tool using heuristic evaluation. Journal of Medical Internet Research 2009;11(4):e47. http://dx.doi.org/10.2196/jmir.1244. PMid:19897458; PMCid:PMC2802560.

44. Jaspers MWM. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. International Journal of Medical Informatics 2009;78(5):340–53. http://dx.doi.org/10.1016/j.ijmedinf.2008.10.002. PMid:19046928.

45. Joshi A, Lichenstein R, King J, Arora M and Khan S. Evaluation of a computer-based patient education and motivation tool on knowledge, attitudes, and practice towards influenza vaccination. IEJHE 2009;12:1–15.

46. Smith MK. The Encyclopedia of Informal Education. 2003. Learning Theory. Available from: http://www.infed.org/biblio/b-learn.htm. Accessed 26 Oct 2009.

47. Jezierski J. Learnativity.com. 2003. Learning Styles. Available from: http://www.learnativity.com/learningstyles.html. Accessed 30 Oct 2009.

48. Russell SS. An overview of adult learning processes. Urologic Nursing 2006;26(5):349–52, 370. PMid:17078322.

49. Shneiderman B. Designing the User Interface: Strategies for Effective Human–Computer Interaction. Reading, MA: Addison–Wesley, 1998. PMid:9929185; PMCid:PMC2232192.

50. Jeffries R, Miller JR, Wharton C and Uyeda K. User interface evaluation in the real world: a comparison of four techniques. Proceedings of the 1991 CHI Conference on Human Factors in Computing Systems, 1991, New Orleans, LA: ACM, pp. 119–24.

51. Nielsen J. Finding usability problems through heuristic evaluaion. Proceedings of the 1992 SIGCHI Conference on Human Factors in Computing Systems,1992, Monterey, CA: ACM, pp. 373–80.

52. Hertzum M and Jacobsen NE. The evaluator effect: a chilling fact about usability evaluation methods. International Journal Human–Computer Interacttion 2001;13:421–43. http://dx.doi.org/10.1207/S15327590IJHC1304_05.

53. Tang Z, Johnson TR, Tindall RD and Zhang J. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemedicine Journal and E-Health 2006;12(1):24–34. http://dx.doi.org/10.1089/tmj.2006.12.24. PMid:16478410.

Refbacks

  • There are currently no refbacks.


This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT