The Heimdall Framework for Supporting Characterisation of Learning Health Systems

Scott McLachlan, Henry W.W. Potts, Kudakwashe Dube, Derek Buchanan, Stephen Lean, Thomas Gallagher, Owen Johnson, Bridget Daley, William Marsh, Norman Fenton


Background: Learning Health Systems (LHS) can focus population medicine and Evidence Based Practice; smart technology delivering the next generation of improved healthcare described as Precision Medicine, and yet researchers in the LHS domain presently lack the ability to recognise their relevant works as falling within this domain.

Objective: To review LHS literature and develop a framework describing the domain that can be used as a tool to analyse the literature and support researchers to identify health informatics investigations as falling with the domain of LHS.

Method: A scoping review is used to identify literature on which analysis was performed. This resolved the ontology and framework. The ontology was applied to quantify the distribution of classifications of LHS solutions. The framework was used to analyse and characterise the various works within the body of LHS literature.

Results: The ontology and framework developed was shown to be easily applicable to the literature, consistently describing and representing the goals, intentions and solutions of each LHS investigation in the literature. More proposed or potential solutions are described in the literature than implemented LHS. This suggests immaturity in the domain and points to the existence of barriers preventing LHS realisation.

Conclusion: The lack of an ontology and framework may have been one of the causes for the failure to describe research works as falling within the LHS domain. Using our ontology and framework, LHS research works could be easily classified, demonstrating the comprehensiveness of our approach in contrast to earlier efforts.


Learning Health Systems, Electronic Health Records, ontology, framework

Full Text:



IoM. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care.Washington,DC: Institute of Medicine, 2011. Available from:

Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a science of learning systems: a research agenda for the high-functioning learning health system. Journal of American Medical Informatics Association 2015;22:43–50.

OECD. Data-Driven Innovation: Big Data for Growth and Well-Being. Paris, France: OECD Publishing, 2015.

Hanna N. Mastering Digital Transformation: Towards a Smarter Society, Economy, City and Nation. Bingley, UK: Emerald Group Publishing, 2016.

Rubin J and Friedman C. Weaving together a healthcare improvement tapestry: learning health system brings together health IT data stakeholders to share knowledge and improve health. Journal of AHIMA 2014;85(5):38–43.

Lim S, Keung C, Zhao L, Curcin V, Ethier J, Burgun A, et al. TRANSFoRm: Implementing a Learning Healthcare System in Europe through embedding clinical research into clinical practice. International Conference on System Sciences (HICSS) (Hawaii), 2015.

IoM. Foreward. In: Olsen L, Aisner D and McGinnis M (Eds.), The Learning Healthcare System: Workshop Summary. Washington, DC: Institute of Medicine, 2007.

Wiley. Learning health systems. 2017. Available from: Accessed 13 September 2017.

UCLPartners. Learning health systems. 2016. Available from: Accessed 13 September 2017.

UCL. Health informatics. 2017. Available from: Accessed 13 September 2017.

Basden A and Clark E. Data integrity in a general practice computer system (CLINICS). International Journal Bio-Medical Computing 1980;11:511–9. Available from:

Yucha C and Reigeluth C. The use of computers in nursing education, practice and administration. Computer Education 1983;7(4):223–6. Available from:

Sengstack P and Boicey C. Mastering Informatics: A Healthcare Handbook for Success. Indianapolis, IN: Sigma Theta Tau, 2015. PMid:25921891.

Sur R and Dahm P. History of evidence based medicine. Indian Journal of Urology 2011;27(4):487–9. Available from: PMid:22279315; PMCid:PMC3263217.

Smith G. Electronic Health Record. Los Angeles, CA: California Department of Alcohol and Drug Programs, 1980.

Gans D, Kralewski J, Hammons T and Dowd B. Medical groups’ adoption of of electronic health records and information systems. Health Affairs 2005;24(5):1323–33. Available from: PMid:16162580.

Rumball-Smith J, Shekelle P and Bates D. Using the Electronic Health record to understand and minimise overuse. Journal of the American Medical Association 2017;317(3):257–8. Available from: PMid:28114561.

Hudis C. Comment on Health Care Competition. Alexandria, VA: American Society of Clinical Oncology, ASCO Website, 2014.

Klimt C, Becker S, Fox BS and Ensminger F. A transcribed emergency record at minimum cost. Annals of Emergency Medicine 1983;12(9):556–8. Available from:

Wyatt J. Clinical data systems, part 1: data and medical records. The Lancet 1994;344:1543–7. Available from:

Menachemi N and Collum T. Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy 2011;4:47–55. Available from: PMid:22312227; PMCid:PMC3270933.

Cresswell K, Mozaffar H, Lee L, Williams R and Sheikh A. Safety risks associated with the lack of integration and interfacing of hospital health information technologies: a qualitative study of hospital electronic prescribing systems in England. British Medical Journal of Quality and Safety 2017;26:530–41.

Furukawa M, King J, Patel V, Hsiao CJ, Adler-Milstein J and Jha AK. Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Affairs 2014;33(9):1672–9. Available from: PMid:25104827.

Tange H. How to approach the structuring of the medical record? Towards a model for flexible access to free text medical data. International Journal of Biomedical Computing 1996;42:27–34. Available from:

Roos L, Sharp S and Wadja A. Assessing data quality: a computerised approach. Social Science and Medicine 1989;28(2):175–82. Available from:

Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. Journal of the American Medical Association 2005;293(10):1197–203. Available from: PMid:15755942.

Greenhalgh T, Potts HW, Wong G, Bark P and Swinglehurst D. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. The Millbank Quarterly 2009;87(4):729–88. Available from: PMid:20021585; PMCid:PMC2888022.

Grumbach K, Lucey C and Johnston C. Transforming from centres of learning to learning health systems: the challenge for academic health centres. Journal of the American Medical Association 2014;311(11):1109–10. Available from: PMid:24643597.

Bhandari R, Feinstein AB, Huestis SE, Krane EJ, Dunn AL, Cohen LL, et al. Pediatric-collaborative health outcomes information registry (Peds-CHOIR): a learning health system to guide pediatric pain research and treatment. PAIN 2017;157(9):2033–44. Available from: PMid:27280328; PMCid:PMC4988911.

Crawford L, Matczak GJ, Moore EM, Haydar RA and Coderre PT. Patient-centered drug development and the Learning Health System. Learning Health Systems 2017;1(3):1–7.

Faden R, Kas NE, Goodman SN, Provost P, Tunis S and Beuchamp TL. An ethics framework for a Learning Health Care System: a departure from traditional research ethics and clinical ethics. Ethical Oversight of Learning Health Care Systems, Hastings Centre Report Special Report. 2013.

SCOAP Collaborative; Writing Group for the SCOAP Collaborative; Kwon S, Florence M, Grigas P, Horton M, Horvath K, Johnson M, et al. Creating a learning healthcare system in surgery: Washington State’s Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery 2011;151(2):146–52. PMCid:PMC4208432.

Gold M, Hossain M and Mangum A. Consumer engagement in Health IT: distinguishing rhetoric from reality. eGEMS 2015;3(1):1190. Available from: PMid:26665120; PMCid:PMC4672873.

Flores M, Glusman G, Brogaard K, Price ND and Hood L. P4 Medicine: how systems medicine will transform the healthcare sector and society. Personalised Medicine 2013;10(6):565–76. Available from: PMid:25342952; PMCid:PMC4204402.

Friedman C, Allee NJ, Delaney BC, Flynn AJ, Silverstein JC, Sullivan K, et al. The science of learning health systems: foundations for a new journal. Learning Health Systems 2016;1.

Foley T and Vale L. What role for learning health systems in quality improvement within healthcare providers? Learning Health Systems 2017;1(4).

Forrest C, Margolis P, Seid M and Colletti RB. PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Affairs 2014;33(7):1171–7. Available from: PMid:25006143.

Everson J. The implications and impact of 3 approaches to health information exchange: community, enterprise, and vendor-mediated health information exchange. Learning Health Systems 2016;1:e10021.

Etheredge L. A rapid-learning health system. Health Affairs 2007;26(2):107–18. Available from: PMid:17259191.

Curcin V. Embedding data provenance into the learning health system to facilitate reproducible research. Learning Health Systems 2017;1(2):e10019.

Faden R, Kass N, Whicher D, Stewart W and Tunis S. Ethics and Informed Consent for Comparative Effectiveness Research with prospective electronic clinical data. Medical Care 2013;51(8):S53–7. Available from:

Kitson A and Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Quality in Health Care 1998;7:149–58. Available from: PMid:10185141; PMCid:PMC2483604.

Cherry B, Carter M, Owen D and Lockhart C. Factors affecting electronic health record adoption in long-term care facilities. Journal for Healthcare Quality 2008;30(2):37–47. Available from: PMid:18411891.

Jamoom E, Patel V, Furukawa MF and King J. EHR adopters vs. non-adopters: Impacts of, barriers to, and federal initiatives for EHR adoption. Healthcare Amsterdam 2014;2(1):33–9. Available from: PMid:26250087; PMCid:PMC4878018.

Nguyen L, Bellucci E and Nguyen L. Electronic health records implementation: an evaluation of information system impact and contingency factors. International Journal of Medical Informatics 2014;83:779–96. Available from:

Tang P, Ash JS, Bates DW, Overhage JM and Sands DZ. Personal health records: definitions, benefits and strategies for overcoming barriers to adoption. Journal of the American Medical Informatics Association 2006;13(2):121–6. Available from: PMid:16357345; PMCid:PMC1447551.

Schumaker R and Reganti K. Implementation of Electronic Health Record (EHR) system in the heathcare industry. International Journal of Privacy and Health Information Management 2014;2(2):57–71. Available from:

Ludwick D and Doucette J. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics 2009;78:22–31. Available from:

Kaye R, Kokia E, Shalev V, Idar D and Chinitz D. Barriers and success factors in health information technology: a practitioners perspective. Journal of Management and Marketing in Healthcare 2010;3(2):163–75. Available from:

McGinn C, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, et al. Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Medicine 2011;9(46):46. Available from:

Lluch M. Healthcare professionals’ organisational barriers to health information technologies: a literature review. International Journal of Medical Informatics 2011;80:849–62. Available from:

Mclachlan S, Dube K, Buchanan D, Lean S, Johnson O, Potts H, et al. Learning Health Systems: the research community awareness challenge. BCS Journal of Innovation in Health Informatics 2018;25(1):38–40. Available from:

NASEM. The learning healthcare system in America. 2011. Available from: Accessed January 2011.

NASEM. The learning healthcare system in America. 2011. Available from: Accessed May 2011.

Wong G, Greenhalgh T, Westhorp G, Buckingham J and Pawson R. RAMESES publication standards: realist syntheses. BMC Medicine 2013;11(1):21. Available from: Available from:

Bellack J. Creating a continuously learning health system through technology: a call to action. Journal of Nursing Education 2016;55(1):3–5. Available from: PMid:26812374.

Yoder-Wise P. The continuously learning health system: recommendations for the Josiah Macy Jr. Foundation. Journal of Continuing Education in Nursing 2015;46(9):379–80. Available from: PMid:26352036.

Bernstein J, Friedman C, Jacobsen P and Rubin J. Ensuring public health’s future in a national-scale learning health system. American Journal of Preventative Medicine 2015;48(4):480–7. Available from: PMid:25700654.

Brody H and Miller F. The research-clinical practice distinction, learning health system and relationships. Hastings Centre Report 2013;43(5):41–7. Available from: PMid:24092591.

Vaismoradi M, Turunen H and Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nursing and Health Sciences 2013;15(3):398–405. Available from: PMid:23480423.

Stumme G. Formal concept analysis. In: Staab S and Studer R (Eds.), Handbook on Ontologies. Berlin, Germany: Springer, 2009, pp. 177–99. Available from:

Friedman C, Wong A and Blumenthal D. Achieving a nationwide learning health system. Science Transitional Medicine 2010;2(57):1–3. Available from:

Deeny S and Steventon A. Making sense of the shadows: Priorities for creating a learning healthcare system based on routinely collected data. BMJ Quality Safety 2015;24:505–15. Available from: PMid:26065466; PMCid:PMC4515981.

Bradley E, Curry LA, Ramanadhan S, Rowe L, Nembhard IM and Krumholz HM. Research in action: using positive deviance to improve quality of health care. BMC Implementation Science 2009;4:25.

Lewis G, Kirkham H and Vaithianathan R. How health systems could avert ‘triple fail’ events that are harmful, are costly, and result in poor patient satisfaction. Health Affairs 2013;32(4):669–76. Available from: PMid:23569046.

Foley T and Fairmichael F. Site visit to Geisinger Health Systems. The Learning Healthcare Project 2015. Available from: Accessed August 2017.

Garg A, Adhikari NK, Mcdonald H, Rosas-Arellano M, Devereaux PJ, Beyene J and Hayes RB. Effects of computerised clinical decision suport systems on practitioner performance and patient outcomes. Journal of the American Medical Association 2005;293(10):1223–38. Available from:

Wyatt J and Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Fifteenth Annual Symposium on Computer Applications in Medical Care. Washington, DC: American Medical Informatics Association, 1991.

Kawamoto K, Houlihan CA, Balas EA and Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. British Medical Journal 2005;330:765–8. Available from: PMid:15767266; PMCid:PMC555881.

Abernethy A, Ahmad A, Zafar SY, Wheeler JL, Reese JB and Lyerly HK. Electronic patient-reported data capture as a foundation of rapid learning cancer care. Medical Care 2010;48(6):S32–8. Available from: PMid:20473201.

Ye Y, Wamukoya M, Ezeh A, Emina JB and Sankoh O. Health and Demographic Surveillance Systems: a step towards full civil registration and vital statistics in sub-Saharan Africa? BMC Public Health 2012;12:741. Available from:

Ethier J. Integrating Resources for Translational Research: A Unified Approach for Learning Health Systems. Paris, France: Universite Pierre et Marie Curie, 2016.

Sacristan J. Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients. BMC Medical Informatics and Decision Making 2013;13:6. Available from:

Mirnezami R, Nicholson J and Darzi A. Preparing for precision medicine. New England Journal of Medicine 2012;366(6):489–91. Available from: PMid:22256780.

Jameson J and Longo D. Precision medicine—personalized, problematic and promising. New England Journal of Medicine 2015;372(23):2229–34. Available from: PMid:26014593.

Harwood J, Butler C and Page A. Patient-centred care and population health: establishing their role in orthapaedic practice. The Journal of Bone and Joint Surgery 2016;98(10):e40. Available from: PMid:27194502.

Yu P, Artz D and Warner J. Electronic health records (EHRs): supporting ASCO’s vision of cancer care. ASCO Educational Book 2014;2014:225–31.

Cresswell, Bates D and Sheikh A. Ten key considerations for the successful optimisation of large-scale health information technology. Journal of the American Medical Informatics Association 2017;24(1):182–7.

Rioth M, Warner J, Savani BN and Jagasia M Next generation long term transplant clinics: improving resource utilisation and the wuality of care through health information technology. Bone Marrow Transplantation 2016;51(1):34–40. Available from: PMid:26367235; PMCid:PMC4915825.

Pritchard D, Moeckel F, Villa MS, Housman LT, McCarty CA and McLeod HL. Strategies for integrating personalised medicine into healthcare practice. Personalised Medicine 2017;14(2):141–52.

Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency and cost of medical care. Annals of Internal Medicine 2006;144(10):742–52. Available from: PMid:16702590.

Buntin M, Burke MF, Hoaglin MC and Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominately positive results. Health Affairs 2011;30(3):464–71. Available from: PMid:21383365.

Yu P. Knowledge bases, clinical decision support systems, and rapid learning in oncology. Journal of Oncology Practice 2015;11(2):e206–11. Available from: PMid:25715002.

Zalon M. Technology and continuously learning health systems. The Journal of Continuing Education in Nursing 2016;47(6):243–5. Available from: PMid:27232218.

Chorpita B, Daleiden E and Bernstein A. At the intersection of Health Information Technology and Decision Support: measurement feedback systems... and beyond. Administration and Policy in Mental Health and Mental Health Services Research 2016;43(3):471–7. Available from: PMCid:PMC5300738.

Etheredge L. Rapid learning: a breakthrough Agenda. Health Affairs 2014;33(7):1155–62. Available from: PMid:25006141.

Fihn S, Francis J, Clancy C, Nielson C, Nelson K, Rumsfeld J, et al. Insights fron advanced analytics at the Veterans Affairs Health Administration. Health Affairs 2014;33(7):1203–11. Available from: PMid:25006147.

Angus D. Fusing randomised trials with big data: the key to self-learning health care systems. Journal of the American Medical Association 2015;314(8):767–8. Available from: PMid:26305643.

Dreyer N. Making observational studies count: shaping the future of comparitive effectiveness research. Epidemiology 2011;22(3):295–7. Available from: PMid:21464648.

Muir Gray J. The shift in personalised and population medicine. The Lancet 2013;382(9888):200–1.

LeFevre M. From authority- to evidence-based medicine: are clinical practice guidelines moving us forwards or backwards? Annals of Family Medicine 2017;15(5):410–12. Available from: PMid:28893809.

Boonstra A and Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research 2010;10(1):231.

Hillstead R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, et al. Can electronic medical record systems transform health care? Potential health benefits, savings and costs. Health Affairs 2005;24(5):1103–17. Available from: PMid:16162551.

AMA. Shared electronic medical records. 2016. Available from: Accessed 3 September 2017.

Davidson A. Creating value: unifying silos into public health business intelligence. eGEMS 2015;2(4):1172.

Denaxas S, Friedman CP, Geissbuhler A, Hemingway H, Kalra D, Kimura M, et al. Discussion of “combining health data uses to ignite health system learning.” Methods of Information in Medicine 2015;54(6):488–99. Available from: PMid:26538343.

Wilk A and Platt P. Measuring physicians’ trust: a scoping review with implications for public policy. Social Science and Medicine 2016;165:75–81. Available from: PMid:27497858.

Lean S, Guesgen HW, Hunter I and Tretiakov A. Working toward automated coding in general practice. Palmerston North, New Zealand: Massey University, 2016.

Faden R, Beauchamp T and Kass N. Informed consent, comparative effectiveness, and learning health care. New England Medical Journal 2014;370(8):766–8.

Marsolo K, Margolis PA, Forrest CB, Colletti RB and Hutton JJ. A digital architecture for a network-based learning health system - Integrating chronic care management, quality improvement and research. eGEMS 2015;3(1):1168. Available from: PMid:26357665; PMCid:PMC4562738.

Nishimura A, Carey J, Erwin PJ, Tilburt JC, Murad MH and McCormick JB. Improving understanding in the research informed consent process: a systematic review of 54 interventions tested in randomized control trials. BMC Medical Ethics 2013;14(28):28. Available from:

Feeley T, Sledge GW, Levit L and Ganz PA. Improving the quality of cancer care in America through health information technology. Journal of the American Medical Informatics Association 2014;21:772–5. Available from: PMid:24352553; PMCid:PMC4147622.

Hughes K, Ambinder EP, Hess GP, Yu PP, Bernstam EV, Routbort MJ, et al. Identifying health information technology needs of oncologists to facilitate the adoption of genomic medicine: recommendations from the 2016 American society of clinical oncology omics and precision oncology workshop. Journal of Clinical Oncology 2017;35:3153–9.



  • There are currently no refbacks.

This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT