Development of the quality assessment model of EHR software in family medicine practices: research based on user satisfaction

Damir Kralj, Josipa Kern, Stanko Tonkovic, Miroslav Koncar


Background Family medicine practices (FMPs) make the basis for the Croatian health care system. Use of electronic health record (EHR) software is mandatory and it plays an important role in running these practices, but important functional features still remain uneven and largely left to the will of the software developers.

Objective The objective of this study was to develop a novel and comprehensive model for functional evaluation of the EHR software in FMPs, based on current world standards, models and projects, as well as on actual user satisfaction and requirements.

Methods Based on previous theoretical and experimental research in this area, we made the initial framework model consisting of six basic categories as a base for online survey questionnaire. Family doctors assessed perceived software quality by using a five-point Likert-type scale. Using exploratory factor analysis and appropriate statistical methods over the collected data, the final optimal structure of the novel model was formed. Special attention was focused on the validity and quality of the novel model.

Results The online survey collected a total of 384 cases. The obtained results indicate both the quality of the assessed software and the quality in use of the novel model. The intense ergonomic orientation of the novel measurement model was particularly emphasised.

Conclusions The resulting novel model is multiple validated, comprehensive and universal. It could be used to assess the user-perceived quality of almost all forms of the ambulatory EHR software and therefore useful to all stakeholders in this area of the health care informatisation.



electronic health records (EHRs), ergonomics, family medicine practices (FMPs), quality evaluation, user satisfaction

Full Text:



Rakel RE and Rakel DP. Textbook of Family Medicine, 8th edition. Philadelphia: Elsevier Saunders, 2011.

Croatian Health Insurance Fund. CEZIH PZZ. Available from: Accessed on 6 April 2014.

Končar M and Gvozdanović D. Primary health care information system – the cornerstone for the next generation health care sector in Republic of Croatia. International Journal of Medical Informatics 2006;75:306–14.

Kralj D. Quality assessment of e-Health solutions in primary health care – approach based on user experience. In: Ganesh RN (Ed). Applied Biological Engineering – Principles and Practice. Rijeka: InTech, 2012, 509–28.

Certification Commission for Healthcare Information Technology. CCHIT Certified 2011 Program. Available from: Accessed on 12 April 2014. Meaningful Use. Available from: Accessed on 4 March 2014.

EuroRec. European Institute for Health Records. Available from: Accessed on 6 April 2014.

Framework for Rural and Remote Readiness in Telehealth, Written by the Alliance of Building Capacity, CANARIE Final Report. 2002. Available from: Accessed on 12 October 2013.

World Health Organization. Strategy 2004–2007: eHealth for Health Care Delivery. Available from: Accessed on 6 November 2013.

Khoja S, Scott RE, Mohsin M, Ishaq AFM and Casebeer A. Developing a conceptual-framework for e-Health readiness assessment tools for developing countries. In: Svensson PG (Ed). ICT Developments: e-Health in Developing Countries (International Hospital Federation Reference Book). London: Pro-Brook, 2007–2008, 79–81.

Li JH. e-Health Readiness Framework from Electronics Health Records Perspective. M.S. Thesis. Sydney: University of New South Wales, 2008.

Dobrev A, Haesner M, Hüsing T, Korte WB and Meyer I. Benchmarking ICT Use Among General Practitioners in Europe. Bonn/Brussels: Empirica, 2008.

Bjomberg A, Garrofe BC and Lindblad S. Euro Health Consumer Index 2009 – Report. Brussels: Health Consumer Powerhouse, 2009.

National Institute for Health and Clinical Excellence. Quality and Outcome Framework. Available from: Accessed on 6 November 2013.

OECD. SMART 2011/0033 – Benchmarking Information and Communication Technologies in Health Systems: Benchmarking Deployment of eHealth Among General Practitioners II – Background Document. Available from Accessed on 12 April 2013.

HL7 International. Introduction to HL7 Standards. Available from: Accessed on 26 March 2014.

National Institute of Standards and Technology. NISTIR 7741: NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. 2010.

National Institute of Standards and Technology. NISTIR 7804: Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records. 2012.

Integrating the Healthcare Enterprise. Changing the Way Healthcare Connects. Available from: Accessed on 26 March 2014.

Agency for Healthcare Research and Quality. Medical Liability Reform & Patient Safety. Available from: Accessed on 8 December 2013.

Ammenwerth E, Iller C and Mahler C. IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Medical Informatics and Decision Making 2006;6(3). PubMed PMID: 16401336; PubMed Central PMCID: PMC1352353.

de Keizer N and Ammenwerth E. The Effects and Quality of IT Evaluation Studies: Trends in 1982–2002, In: AMIA Annual Symposium Proceedings 2005, pp. 186–90. Review. PubMed PMID: 16779027; PubMed Central PMCID: PMC1560570.

Hoerbst A and Ammenwerth E. Quality and certification of electronic health records: an overview of current approaches from the US and Europe. Applied Clinical Informatics 2010;1(2):149–64.

Riedmann D, Jung M, Hackl WO and Ammenwerth E. How to improve the delivery of medication alerts within computerised physician order entry systems: an international Delphi study. Journal of the American Medical Informatics Association 2011;18(6):760–6. Epub 2011 Jun 22. PubMed PMID: 21697293; PubMed Central PMCID: PMC3197982.

Darmon D, Sauvant R, Staccini P and Letrilliart L. Which functionalities are available in the electronic health record systems used by French general practitioners? An assessment study of 15 systems. International Journal of Medical Informatics 2014;83:484–94.

McAlearney AS, Hefner JL, Sieck C, Rizer M and Huerta TR. Evidence-based management of ambulatory electronic health record system implementation: an assessment of conceptual support and qualitative evidence. International Journal of Medical Informatics 2014;83:37–46.

Cabitza F, Del Zotti F and Misericordia P. Electronic Records for General Practice – Where we are, Where we Should Head to Improve Them. In: Healthinf 2014. Proceedings of the 7th International Conference on Health Informatics (France, 3–6 March), 2014: INSTICC Press.

Jung M, Riedmann D, Hackl WO, Hoerbst A, Jaspers MW, Ferret L et al. Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerised physician order entry systems. BMC Medical Informatics and Decision Making 2012;12(111). PubMed PMID: 23031275; PubMed Central PMCID: PMC3522054.

Hyppönen H, Reponen J, Lääveri T and Kaipio J. User experiences with different regional health information exchange systems in Finland. International Journal of Medical Informatics 2014;83:1–18.

Pandhi N, Yang W-L, Karp Z, Young A, Beasley JW, Kraft S et al. Approaches and challenges to optimising primary care teams’ electronic health record usage. Informatics in Primary Care 2014;21(3):142–51.

Kalpić D, Mornar V, Kovač M, Fertalj K and Kos M. An Insight into Efforts to Establish Computerisation and E-Services for Public Health in Croatia. In: Gavish B (Ed). Proceedings of the 2005 Networking and Electronic Commerce Research Conference. Dallas: Gavish, Bezalel;2005.

Kern J and Polašek O. Evaluation of the GP Information System – Pilot Study. In: Proceedings of the European Federation of Medical Informatics. Special Topic Conference. Athens, Greece:IOS Press;2005, pp. 113–5.

Kern J and Polašek O. Information and communication technology in family practice in Croatia. European Journal for Biomedical Informatics. 2007;1:7–14.

Kralj D, Končar M and Tonković S. Use of Guidelines and Decision Support Systems Within EHR Applications in Family Practice – Croatian Experience. In: Pallikarakis B (Ed). Proceedings of the 12th Mediterranean Conference on Medical and Biological Engineering and Computing: MEDICON 2010 (Chalkidiki, Greece 27–30 May). Heidelberg, Germany: Springer; 2010, pp. 928–31.

Kralj D, Končar M and Tonković S. A methodology to assess experiences in implementing e-Health solutions in Croatian family medicine. Studies in Health Technology and Informatics 2011;165:129–34.

Andrijašević L, Angebrandt P and Kern J. Users’ satisfaction with the primary health care information system in Croatia: a cross-sectional study. Croatian Medical Journal 2012;53(1):60–5.

Kralj D, Končar M and Tonković S. Quality Assessment Model of Software for Managing Family Medicine Practice – Methodology and Basic Results. In: Barrionuevo SG (Ed). Proceedings of the BHI 2014: 2nd IEEE EMBS International Conference on Biomedical and Health Informatics: BHI 2014; (Valencia, Spain, Jun 1–4). IEEE; 2014, pp. 334–7.

Nykanen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC et al. Guideline for Good Evaluation Practice in Health Informatics (GEP-HI). International Journal of Medical Informatics 2011;80:815–27.

Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykanen P and Rigby M. STARE-HI – Statement on Reporting of Evaluation Studies in Health Informatics. International Journal of Medical Informatics 2009;78:1–9.

Anastasi A. Psychological Testing – 6th edition. New York: Macmillan Publishing Company, 1988.

Urbina S. Essentials of Psychological Testing. Hoboken, New Jersey: John Wiley & Sons Inc, 2004.

Nunnally JC and Bernstein IH. Psychometric Theory – 3rd edition. New York: McGraw-Hill Inc, 1994.

SurveyMonkey. Survey UMKPPVOOM-01. Available from: Accessed on 31 January 2013.

Tabachnick BG and Fidell LS. Using Multivariate Statistics, 5th edition. Boston: Pearson Education, 2007.

Sauro J and Lewis JR. Quantifying the User Experience. Waltham, MA: Elsevier Morgan Kaufmann, 2012.

Wittkowski KM, Lee E, Nussbaum R, Charnian FN and Krueger JG. Combining several ordinal measures in clinical studies. Statistics in Medicine 2004;23:1579–92.

Stevens SS. On the theory of scales of measurement. Science. 1946;103(2684):677–80.

Pallant J. SPSS Survival Manual. New York: McGraw-Hill Inc, 2007.

Garth A. Analysing data using SPSS. Shefield Hallam University, 2008.

Zar JH. Biostatistical Analysis – 5th edition. New Jersey: Pearson Prentice Hall, 2010.

Watkins MW. Determining parallel analysis criteria. Journal of Modern Applied Statistical Methods 2006;5(2):344–6.

Cook DA and Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. American Journal of Medicine 2006;119(2):e7–16.

Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297–334.

Certification Commission for Healthcare Information Technology. CCHIT 2011 Usability Testing Guide for Ambulatory EHR’s. Available from: Accessed on 5 January 2014.



  • There are currently no refbacks.

This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT