The Heimdall framework for supporting characterisation of learning health systems

Scott McLachlan

Queen Mary University of London, London, UK

Henry W. W. Potts

University College London, London, UK

Kudakwashe Dube

Massey University, Auckland, New Zealand

Derek Buchanan

Massey University, Auckland, New Zealand

Stephen Lean

Massey University, Auckland, New Zealand

Thomas Gallagher

University of Montana, Missoula, MT

Owen Johnson

University of Leeds, Leeds, UK

Bridget Daley

Blizzard Institute, Queen Mary University of London, London, UK

William Marsh

Queen Mary University of London, London, UK

Norman Fenton

Queen Mary University of London, London, UK

Author address for correspondence:

Scott McLachlan

Queen Mary University of London (QMUL)

London E1 4FZ, UK


Cite this article: McLachlan S, Potts H WW, Dube K, Buchanan D, Lean S, Gallagher T, Johnson O, Daley B, Marsh W, Fenton N. The Heimdall framework for supporting characterisation of learning health systems. J Innov Health Inform. 2018;25(2):77–87.

Copyright © 2018 The Author(s). Published by BCS, The Chartered Institute for IT under Creative Commons license


Background There are many proposed benefits of using learning health systems (LHSs), including improved patient outcomes. There has been little adoption of LHS in practice due to challenges and barriers that limit adoption of new data-driven technologies in healthcare. We have identified a more fundamental explanation: the majority of developments in LHS are not identified as LHS. The absence of a unifying namespace and framework brings a lack of consistency in how LHS is identified and classified. As a result, the LHS ‘community’ is fragmented, with groups working on similar systems being unaware of each other’s work. This leads to duplication and the lack of a critical mass of researchers necessary to address barriers to adoption.

Objective To find a way to support easy identification and classification of research works within the domain of LHS.

Method A qualitative meta-narrative study focusing on works that self-identified as LHS was used for two purposes. First, to find existing standard definitions and frameworks using these to create a new unifying framework. Second, seeking whether it was possible to classify those LHS solutions within the new framework.

Results The study found that with apparently limited awareness, all current LHS works fall within nine primary archetypes. These findings were used to develop a unifying framework for LHS to classify works as LHS, and reduce diversity and fragmentation within the domain.

Conclusions Our finding brings clarification where there has been limited awareness for LHS among researchers. We believe our framework is simple and may help researchers to classify works in the LHS domain. This framework may enable realisation of the critical mass necessary to bring more substantial collaboration and funding to LHS. Ongoing research will seek to establish the framework’s effect on the LHS domain.

Keywords: electronic health records, learning health systems, learning healthcare systems, precision medicine


Learning health systems (subsequently referred to as LHS) are defined by the Institute of Medicine (IoM) as systems in which alignment of scientific and cultural tools lead to knowledge generation to improve healthcare as a result of daily practice.1 Since LHS was conceptualised in 2007, they have been the focus of increasing research attention.26 The opportunity and promise of LHS have resulted in texts presenting collections of LHS-specific research1,7 creation of a new journal, Learning Health Systems,8 and new courses of academic study.9,10 We believe it could be the most significant development in healthcare since the advent of evidence based medicine (EBM) and electronic health records (EHRs) that support EBM.1416 EHR has existed for more than 40 years1113 and organisations that implemented EHR discovered reductions in costs, clinical testing and patterns of repeated and sometimes unneeded prescriptions. Enhanced co-ordination and communication between clinicians were seen to improve the quality of patient care.12,1720

Despite the benefits, early EHR systems were considered expensive, focused on information gathering rather than improving healthcare.20 Development lacked clinical input, existed as multiple stand-alone systems, experienced slow adoption, suffered from trust and data quality issues, claims that systems increase or exacerbate risk for errors, and concerns over patient privacy and security.18,20 All of these issues are still seen as unresolved barriers to adoption of EHR.18,2127 Despite this, EHRs are the foundation for LHS. Efforts towards LHS, coupled with proposed changes to legislation, policy and the ethics of how clinicians engage with clinical datasets suggest an entirely new dimension to EHR. One in which they are used collectively as ‘big data’ and focused using individual patient’s attributes to identify causes and optimal treatments strategies for disease.

Descriptions developed in IoM reports are the basis of most author definitions and descriptions of LHS.2833 Medical information systems that can be predictive, preventative, personalised and participatory represent the core principles of 4P medicine.34 According to the IoM, these systems have the potential to identify groups at greatest risk of complications for purposes of targeting interventions.7 In parallel, maturing technologies such as large datasets, machine learning, and enhanced processing power further enable the concept of LHS.2,34,35

The IoM promoted EBM as the primary driver for LHS,7 yet their definition fails to describe LHS attributes which contribute to quality, safety, efficiency and effectiveness of patient care.36 The four fundamental attributes37 listed in Figure 1 provide tangible metrics to compare and contrast LHS efforts. These attributes were not included in the IoM definition, but are widely found in LHS. They clarify the use of large EHR datasets as the source of knowledge for LHS achieving the goal of improving quality in individual patient care.

There are numerous examples of proposed benefits of LHS. For clinicians, these include assessing which laboratory or imaging tests may be more diagnostic given a patient’s presenting symptomology38; reducing risk from prescribing errors31 and increasing awareness of pharmacogenetics.39 It is claimed that patients would benefit from advanced knowledge developed from the experiences and diagnoses of past patients, which saves time and reduces costs.38,40,41 A learning healthcare organisation culture supports EBM, while successful integration of research into practice is what enables it.42 The ability to use technology to record, compare, contrast and present information in almost real-time enhances the input, analysis and decision phases of the learning lifecycle. Alternatively, it is said that the financial burden to implement and support health technology4345 along with a persistent need for data and systems standardisation,4548 interoperability46,49,50 and integration49,51 have all acted as barriers to broad LHS adoption.

Figure 1 Four elements of an LHS

In our group’s recent letter to the editor of this journal,52 we demonstrated the lack of awareness and barriers for researchers to appropriately identify their efforts as LHS solutions. We believe that this results from a number of significant problems in the domain. The lack of adequate classification and standardisation results in groups working on comparable systems not identifying their efforts as LHS, and may be the cause of unnecessary duplication of efforts and the observable lack of collaboration. The absence of a unifying framework means the domain is yet to generate a necessary critical mass, limiting efforts to resolve barriers and challenges to the adoption of LHS and constraining funding availability. We were only able to identify one primary article that attempted to consolidate and analyse the current state of knowledge in LHS.36 We extend that effort drawing on a larger collection of works to establish a unifying framework for LHS.


Our study sought first to define a comprehensive framework and taxonomy for LHS, and then to demonstrate its application to self-identified literature from the LHS domain. The literature search used identical plain language search terms as Foley and Vale36: ‘LHS’ and ‘learning healthcare system’ drawing articles from Scopia, Science Direct, PubMed, EBSCOhost, DOAJ and Elsevier (n = 1083). These works were all authored in the decade since the IoM’s initial LHS report.7 Figure 2 shows this literature, by year, in orange (the drop in 2017 is due to reporting only up to July); contrasted with literature identified by Foley and Vale36 shown in grey. The leap in publications in 2011 followed the first53 and second54 meetings of the Committee on the Learning Healthcare System in America, both proceedings published during 2011. More than 50% of LHS publications were generated since 2013.

We undertook this review following the identification of seminal sources approach of meta-narrative reviewing.27,55 An initial read of abstracts was used to reject duplicates and papers not related to the central topic, including those using the search term in context of learning in the academic or education sense.56,57 This reduced the collection source pool (n = 542). Conclusions and methods were reviewed, seeking to reject papers that did not present or propose an LHS; for example, those exploring the medicolegal, ethical or societal aspects of LHS.31,58,59 The resulting core pool (n = 230) was then comprehensively reviewed. Of these, 53% proposed a potential solution compared to 47% that presented an existing solution.

We used content and thematic analysis60 to recognise and classify LHS uses while also identifying common barriers and thematic concepts for investigation. Formal concept analysis61 was used to identify the frequency and interrelationships between the identified concepts. The elements of both analysis methods were identified inductively during the first full reading of the core pool of literature and used to develop spreadsheets for analysis. A second full reading was performed to data mine the literature and populate spreadsheets for further analysis during framework development. Table 1 provides examples of attributes identified for the formal concept analysis.

Figure 2 LHS publications by year

Table 1 LHS formal concept analysis attributes


Taxonomy for LHS

We found only three papers36,62,63 that proposed classification systems for LHS. Surveillance and Comparative Effectiveness Research were the only types common to all three. Figure 3 unifies the knowledge identified from all three papers into a taxonomy of nine LHS classification types, indicating abbreviations (initials) and the primary reference for each.

Cohort identification (CI) seeks patients with similar attributes, used to determine the feasibility of studies and quantify numbers of potential patients that may be helped.62 CI is also the first operational step of most other LHS types.

Positive deviance (PD) uses outcome data to benchmark clinical care. PD identifies elements of safer, more effective, timely and patient-centred care, recognising beneficial behaviours for incorporation into another clinician’s practice.64 PD can also identify common traits of patients benefiting from a treatment, using these to identify others who may benefit from the same intervention.62

Negative deviance (ND) identifies instances of sub-optimal care outcomes. ND presupposes some particular clinical behaviour negatively impacted patient care and the resulting outcome. The clinician critically evaluates the care provided, investigating causes for sub-optimal results.63

Predictive patient risk modelling (PPRM) uses patterns discovered in patient datasets to identify cohorts at higher risk for future adverse events. PPRM can use routine health data to identify ‘triple fail’ events; where treatment fails to improve patient care experiences, population health, or lower healthcare costs.65

Predictive care risk and outcome model (PCROM) algorithms identify situations of high risk for ‘unsafe’, ‘delayed’ or ‘inefficient’ care, providing estimates of the effectiveness of different interventions.36 Geisinger Health Systems have incorporated PCROM approaches into clinical software, identifying when spikes in hospital activity or patient non-attendance may occur.66

Clinical decision support systems (CDSSs) are active knowledge systems where two or more characteristics of the patient are matched to computerised knowledge bases with algorithms generating patient-specific treatment recommendations.6769

Comparative effectiveness research (CER) compares interventions and outcomes within an EHR dataset to determine the most effective treatment, using a method considered more efficient than randomised control trials.36 CER isolates patients with similar attributes to the current patient, returning knowledge on treatments that deliver optimum health outcomes.70

Figure 3 LHS taxonomy

Intelligent assistance (IA) uses data sources to automate routine processes such as prepopulating pathology orders and clinical notes, or summarising patient case notes prior to consultations.36

Surveillance (S) monitors EHR data for outbreaks of disease (e.g., measles) or treatment issues (e.g., contaminated medicines or increased frequency for post-surgical infections). Examples observed include health and demographic surveillance systems used in sub-Saharan Africa.71

The Heimdall-integrated LHS framework

Just as the Norse God Heimdall was said to be the son of nine mothers, we started from our nine LHS classifications to develop the integrated LHS framework in Figure 4. The diagram’s conical structure demonstrates the use of technology (large datasets and processing systems) to record, store, index and present information that flows into and improves the learning processes used in EBM, focusing clinical practice towards delivery of precision medicine (PM). This enables the learning healthcare organisation to engage in decisions individualised to match unique patient characteristics.

PM results from enhancing the generalised population health approach using attributes in the EHR to constrain analysis for diagnosis and treatment options to cohort predominately matching the presenting patient’s profile. As the clinician enters attributes about the current patient, the speed and accuracy of decisions increase as illustrated by arrows in Figure 4. LHS draws knowledge from a reducing cohort whose attributes predominately match the current patient as illustrated by the circular design. In examining a cross-section of Figure 4, the larger white circle represents the entire population used to select the most effective common treatment. The light grey circle reduces that population to those who share some basic attributes with the patient that would normally be identified in the slower learning organisation approach of EBM. The inner dark circle further reduces the population to a significant cohort with clinical, genetic and socioeconomic attributes predominately matching the patient at the centre. The interrelationship between the Heimdall framework and our taxonomy is shown down the right side of the conical portion of the diagram. While LHS is technology solutions, the majority operate in the context of either the treatment provider’s learning organisation, or the clinician’s primary patient-facing role.

Figure 4 The LHS unifying framework

Within this framework, we incorporate the concept of a clinical lifecycle as shown in Figure 5 and adapted from multiple works in this review.63,66,70 The right side of the diagram represents largely clinician-driven aspects, while the left side identifies those aspects where LHS technology delivers improvements. The more challenging barriers to LHS regarding data quality, interoperability and standardisation all result from activities on the right side of the lifecycle.

This cycle is repeated, both for surveilling the proposed transformation and to seek further items of knowledge.63,66,70 LHS engenders a close relationship between care, research and knowledge translation, aimed at providing a platform for integrating various data to better understand patients.72 LHS is commonly described using an iterative lifecycle similar to many other EBM processes where: (a) patient data is collected by clinicians; (b) aligned, transformed and amalgamated into larger data sets; (c) a problem is defined and (d) analysis performed; (e) with evidence data returned and (f) made into new knowledge, that is, (g) used to transform clinical practice.

The taxonomy and Heimdall framework provided a toolkit for characterising LHS literature in terms of the following thematic and analytical aspects to assess whether the LHS demonstrates:

• Taxonomic consistency – conforms to the taxonomy.

• Patient focused – ensures personalised health care, known as PM.

• Technology usage – uses health IT with big data, machine learning algorithms and automation.

• Decision support – near real-time support to clinicians, bringing recent scientific advances, machine learning and EBM together at point-of-care.

• Application of LHS – goes beyond selecting ‘most likely treatment for a population’, to selecting the ‘most applicable treatment for an individual’.

• Barriers and further observations – challenges limiting implementation.

What follows is a survey and discussion using the Heimdall framework to identify and resolve key questions from within the domain of LHS. Answers to these questions were resolved through applying the framework to the broader literature cohort.

Figure 5 An example of the clinical lifecycle

Validation of the LHS taxonomy

To validate our taxonomy, proposed or presented solutions were reviewed and classified using the taxonomic descriptions. Some proposed solutions were incompletely described, but we found that the intention of the authors was always clear from the information presented. All LHS solutions conformed easily to one of our identified taxonomic types. We believe this validates the taxonomy as we have presented. Our validation of the taxonomy also found that CER were the most prevalent type of LHS presented in the literature, as identified in Table 2. CER was followed at some distance by stand-alone CI, although it should be noted that this classification type had not been identified by either Deeny and Steventon63 or Foley and Vale36.


A number of themes emerged during this analysis and are discussed in the following section, which includes topical analysis to investigate their effect on the LHS domain.

Patient focused

While clinicians argue they always practiced patient-focused medicine, normal clinical practice follows population medicine-based EBM.73 PM extends diagnostic practices with profiling techniques and therapies tailored to the individual.74,75 Patient focus is a key dimension that LHS improve.36 PM approaches can be retrospective, as in CER, and prospective, when genotyping for treatment selection.74,75 Patient-centred care encourages data use in optimising care for individuals.76 Aggregated patient records enable LHS to identify cohorts similar to the patient.77 LHS is an efficient tool for integrating PM into practice. As the clinician enters attributes about the patient, the LHS refines a cohort of prior similar patients. It assesses the treatments they received to recommend one most likely to produce an optimal outcome.

Technology usage

The focus for health IT has shifted from issues of adoption to identifying how to best use technology to improve healthcare delivery and outcomes.78 This shift is significant in creating LHS and elevates issues in EMR/EHR interoperability, data standardisation and quality that must be resolved if LHS is to be truly practical and ubiquitous.7880 Health IT’s ability to improve healthcare service delivery quality and efficiency is recognised.81,82 Enabling necessary data flow and integration of data sources are key abilities technology can deliver, representing core requirements to enabling LHS.72,83 Integration of learning into technology is observed in every LHS solution reviewed. Technology is fundamental to LHS. As EBM evolves from paper-based roots, clinicians and healthcare providers will realise benefits from coupling technology to Learning Health Organisations, thus realising LHS.79,84

Table 2 Distribution of LHS solutions (per 100 publications)

Decision support

Healthcare providers evolved from considering health IT as a billing and documentation facilitator, to contemplating its active participation and capabilities to answer complex questions in care delivery.85 LHS brings opportunities for improving speed and efficiency of clinical decision support.83,85,86 LHS solutions are context-sensitive, incorporating CI and risk modelling in real time to identify interventions for improving individual patient outcomes.87,88 LHS has potential to rapidly perform retrospective comparative effectiveness trials, evaluating treatment options against each other where they have been provided to similar patients.88,89 In contrast with randomised clinical trials, LHS is considered safer, and engenders greater confidence in accuracy of the treatment choice.88,89

Application of LHS

Clinical epidemiology is an example of learning healthcare. EBM evolved from clinical epidemiology: statistically identifying the optimal treatment which becomes best practice for that condition.90,91 Conversely, the focus for PM is selecting from available interventions the treatment that will best serve the individual. The primary driver towards population medicine was economic: maximising benefit while minimising cost, harm and waste.90 While meant for benefiting the individual patient, population medicine has disadvantages in that the individual’s best interests may conflict with those of the population and it is difficult to reconcile the two.76,90 Individual patients may be denied higher priced precision interventions in favour of lower cost population-optimised interventions.90 The Heimdall framework demonstrates that LHS focus healthcare using population medicine and EBM directly onto the presenting patient. While EBM selects one treatment for all patients, LHS produces a cohort with attributes similar to the presenting patient, identifying the treatment most likely to be effective for this individual patient. LHS in this way delivers PM.

Barriers and further observations

Most authors discuss barriers to implementation. The most common are cost,32,92,93 data interoperability and standardisation,9496 poor data quality and integrity,63,97,98 informed consent and ethics review complications,99101 privacy and security issues70,95 and slow technology adoption.95,102,103 These issues are seen in the same context for adopting EHR/EMR. This suggests LHS is inheriting problems from the EHR/EMR on which they depend.


LHS represents a significant improvement on the present learning organisation, evidence-based practice and population medicine approaches. LHS improves the focus on the individual patient, bringing efficient and expedient PM solutions. LHS is a significant evolution to EBM, and the natural next step in realising the benefits that were expected from implementing EHR/EMR. However, the lack of taxonomy for classifying and describing LHS may be a significant reason for fragmented and duplicated research and solutions in LHS that has impeded widespread adoption. Many authors presenting solutions fail to identify them as LHS. Our research has presented a taxonomy and framework to address this problem, and may help address the challenges in realising all that LHS promise.


Scott McLachlan, William Marsh and Norman Fenton acknowledge support from the EPSRC under project EP/P009964/1: PAMBAYESIAN: Patient Managed decision-support using Bayes Networks. Henry W. W. Potts has received consultancy fees from Crystallise and The HELP Trust, and funding from myownteam and Owen Johnson was supported by ClearPath Connected Health Cities Project.

Competing Interests

None of the authors identified a competing interest relevant to this research.


William Marsh raised the topic, leading to Scott McLachlan performing the primary research and preparing the first draft of this paper. Kudakwashe Dube proposed formation of the framework, Owen Johnson developed the initial framework which was further enhanced by Derek Buchanan from the clinician’s perspective. Derek Buchanan and Bridget Daley provided review and input based on clinical experience. The initial method was proposed by Scott McLachlan, extended and significantly improved by Henry W. W. Potts, with Stephen Lean, Kudakwashe Dube and Thomas Gallagher providing input to the survey structure and framework. Thomas Gallagher, Kudakwashe Dube, Norman Fenton, Bridget Daley, Stephen Lean and Henry W. W. Potts provided editorial review. Norman Fenton and William Marsh supervised the research. All authors commented on and approved the final draft.


1. IoM. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care.Washington,DC: Institute of Medicine, 2011. Available from:

2. Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a science of learning systems: a research agenda for the high-functioning learning health system. Journal of American Medical Informatics Association 2015;22:43–50.

3. OECD. Data-Driven Innovation: Big Data for Growth and Well-Being. Paris, France: OECD Publishing, 2015.

4. Hanna N. Mastering Digital Transformation: Towards a Smarter Society, Economy, City and Nation. Bingley, UK: Emerald Group Publishing, 2016.

5. Rubin J and Friedman C. Weaving together a healthcare improvement tapestry: learning health system brings together health IT data stakeholders to share knowledge and improve health. Journal of AHIMA 2014;85(5):38–43.

6. Lim S, Keung C, Zhao L, Curcin V, Ethier J, Burgun A, et al. TRANSFoRm: Implementing a Learning Healthcare System in Europe through embedding clinical research into clinical practice. International Conference on System Sciences (HICSS) (Hawaii), 2015.

7. IoM. Foreward. In: Olsen L, Aisner D and McGinnis M (Eds.), The Learning Healthcare System: Workshop Summary. Washington, DC: Institute of Medicine, 2007.

8. Wiley. Learning health systems. 2017. Available from: Accessed 13 September 2017.

9. UCLPartners. Learning health systems. 2016. Available from: Accessed 13 September 2017.

10. UCL. Health informatics. 2017. Available from: Accessed 13 September 2017.

11. Basden A and Clark E. Data integrity in a general practice computer system (CLINICS). International Journal Bio-Medical Computing 1980;11:511–9. Available from:

12. Yucha C and Reigeluth C. The use of computers in nursing education, practice and administration. Computer Education 1983;7(4):223–6. Available from:

13. Sengstack P and Boicey C. Mastering Informatics: A Healthcare Handbook for Success. Indianapolis, IN: Sigma Theta Tau, 2015. PMid:25921891.

14. Sur R and Dahm P. History of evidence based medicine. Indian Journal of Urology 2011;27(4):487–9. Available from: PMid:22279315; PMCid:PMC3263217.

15. Smith G. Electronic Health Record. Los Angeles, CA: California Department of Alcohol and Drug Programs, 1980.

16. Gans D, Kralewski J, Hammons T and Dowd B. Medical groups’ adoption of of electronic health records and information systems. Health Affairs 2005;24(5):1323–33. Available from: PMid:16162580.

17. Rumball-Smith J, Shekelle P and Bates D. Using the Electronic Health record to understand and minimise overuse. Journal of the American Medical Association 2017;317(3):257–8. Available from: PMid:28114561.

18. Hudis C. Comment on Health Care Competition. Alexandria, VA: American Society of Clinical Oncology, ASCO Website, 2014.

19. Klimt C, Becker S, Fox BS and Ensminger F. A transcribed emergency record at minimum cost. Annals of Emergency Medicine 1983;12(9):556–8. Available from:

20. Wyatt J. Clinical data systems, part 1: data and medical records. The Lancet 1994;344:1543–7. Available from:

21. Menachemi N and Collum T. Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy 2011;4:47–55. Available from: PMid:22312227; PMCid:PMC3270933.

22. Cresswell K, Mozaffar H, Lee L, Williams R and Sheikh A. Safety risks associated with the lack of integration and interfacing of hospital health information technologies: a qualitative study of hospital electronic prescribing systems in England. British Medical Journal of Quality and Safety 2017;26:530–41.

23. Furukawa M, King J, Patel V, Hsiao CJ, Adler-Milstein J and Jha AK. Despite substantial progress in EHR adoption, health information exchange and patient engagement remain low in office settings. Health Affairs 2014;33(9):1672–9. Available from: PMid:25104827.

24. Tange H. How to approach the structuring of the medical record? Towards a model for flexible access to free text medical data. International Journal of Biomedical Computing 1996;42:27–34. Available from:

25. Roos L, Sharp S and Wadja A. Assessing data quality: a computerised approach. Social Science and Medicine 1989;28(2):175–82. Available from:

26. Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, et al. Role of computerized physician order entry systems in facilitating medication errors. Journal of the American Medical Association 2005;293(10):1197–203. Available from: PMid:15755942.

27. Greenhalgh T, Potts HW, Wong G, Bark P and Swinglehurst D. Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. The Millbank Quarterly 2009;87(4):729–88. Available from: PMid:20021585; PMCid:PMC2888022.

28. Grumbach K, Lucey C and Johnston C. Transforming from centres of learning to learning health systems: the challenge for academic health centres. Journal of the American Medical Association 2014;311(11):1109–10. Available from: PMid:24643597.

29. Bhandari R, Feinstein AB, Huestis SE, Krane EJ, Dunn AL, Cohen LL, et al. Pediatric-collaborative health outcomes information registry (Peds-CHOIR): a learning health system to guide pediatric pain research and treatment. PAIN 2017;157(9):2033–44. Available from: PMid:27280328; PMCid:PMC4988911.

30. Crawford L, Matczak GJ, Moore EM, Haydar RA and Coderre PT. Patient-centered drug development and the Learning Health System. Learning Health Systems 2017;1(3):1–7.

31. Faden R, Kas NE, Goodman SN, Provost P, Tunis S and Beuchamp TL. An ethics framework for a Learning Health Care System: a departure from traditional research ethics and clinical ethics. Ethical Oversight of Learning Health Care Systems, Hastings Centre Report Special Report. 2013.

32. SCOAP Collaborative; Writing Group for the SCOAP Collaborative; Kwon S, Florence M, Grigas P, Horton M, Horvath K, Johnson M, et al. Creating a learning healthcare system in surgery: Washington State’s Surgical Care and Outcomes Assessment Program (SCOAP) at 5 years. Surgery 2011;151(2):146–52. PMCid:PMC4208432.

33. Gold M, Hossain M and Mangum A. Consumer engagement in Health IT: distinguishing rhetoric from reality. eGEMS 2015;3(1):1190. Available from: PMid:26665120; PMCid:PMC4672873.

34. Flores M, Glusman G, Brogaard K, Price ND and Hood L. P4 Medicine: how systems medicine will transform the healthcare sector and society. Personalised Medicine 2013;10(6):565–76. Available from: PMid:25342952; PMCid:PMC4204402.

35. Friedman C, Allee NJ, Delaney BC, Flynn AJ, Silverstein JC, Sullivan K, et al. The science of learning health systems: foundations for a new journal. Learning Health Systems 2016;1.

36. Foley T and Vale L. What role for learning health systems in quality improvement within healthcare providers? Learning Health Systems 2017;1(4).

37. Forrest C, Margolis P, Seid M and Colletti RB. PEDSnet: how a prototype pediatric learning health system is being expanded into a national network. Health Affairs 2014;33(7):1171–7. Available from: PMid:25006143.

38. Everson J. The implications and impact of 3 approaches to health information exchange: community, enterprise, and vendor-mediated health information exchange. Learning Health Systems 2016;1:e10021.

39. Etheredge L. A rapid-learning health system. Health Affairs 2007;26(2):107–18. Available from: PMid:17259191.

40. Curcin V. Embedding data provenance into the learning health system to facilitate reproducible research. Learning Health Systems 2017;1(2):e10019.

41. Faden R, Kass N, Whicher D, Stewart W and Tunis S. Ethics and Informed Consent for Comparative Effectiveness Research with prospective electronic clinical data. Medical Care 2013;51(8):S53–7. Available from:

42. Kitson A and Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. Quality in Health Care 1998;7:149–58. Available from: PMid:10185141; PMCid:PMC2483604.

43. Cherry B, Carter M, Owen D and Lockhart C. Factors affecting electronic health record adoption in long-term care facilities. Journal for Healthcare Quality 2008;30(2):37–47. Available from: PMid:18411891.

44. Jamoom E, Patel V, Furukawa MF and King J. EHR adopters vs. non-adopters: Impacts of, barriers to, and federal initiatives for EHR adoption. Healthcare Amsterdam 2014;2(1):33–9. Available from: PMid:26250087; PMCid:PMC4878018.

45. Nguyen L, Bellucci E and Nguyen L. Electronic health records implementation: an evaluation of information system impact and contingency factors. International Journal of Medical Informatics 2014;83:779–96. Available from:

46. Tang P, Ash JS, Bates DW, Overhage JM and Sands DZ. Personal health records: definitions, benefits and strategies for overcoming barriers to adoption. Journal of the American Medical Informatics Association 2006;13(2):121–6. Available from: PMid:16357345; PMCid:PMC1447551.

47. Schumaker R and Reganti K. Implementation of Electronic Health Record (EHR) system in the heathcare industry. International Journal of Privacy and Health Information Management 2014;2(2):57–71. Available from:

48. Ludwick D and Doucette J. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. International Journal of Medical Informatics 2009;78:22–31. Available from:

49. Kaye R, Kokia E, Shalev V, Idar D and Chinitz D. Barriers and success factors in health information technology: a practitioners perspective. Journal of Management and Marketing in Healthcare 2010;3(2):163–75. Available from:

50. McGinn C, Grenier S, Duplantie J, Shaw N, Sicotte C, Mathieu L, et al. Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Medicine 2011;9(46):46. Available from:

51. Lluch M. Healthcare professionals’ organisational barriers to health information technologies: a literature review. International Journal of Medical Informatics 2011;80:849–62. Available from:

52. Mclachlan S, Dube K, Buchanan D, Lean S, Johnson O, Potts H, et al. Learning Health Systems: the research community awareness challenge. BCS Journal of Innovation in Health Informatics 2018;25(1):38–40. Available from:

53. NASEM. The learning healthcare system in America. 2011. Available from: Accessed January 2011.

54. NASEM. The learning healthcare system in America. 2011. Available from: Accessed May 2011.

55. Wong G, Greenhalgh T, Westhorp G, Buckingham J and Pawson R. RAMESES publication standards: realist syntheses. BMC Medicine 2013;11(1):21. Available from: Available from:

56. Bellack J. Creating a continuously learning health system through technology: a call to action. Journal of Nursing Education 2016;55(1):3–5. Available from: PMid:26812374.

57. Yoder-Wise P. The continuously learning health system: recommendations for the Josiah Macy Jr. Foundation. Journal of Continuing Education in Nursing 2015;46(9):379–80. Available from: PMid:26352036.

58. Bernstein J, Friedman C, Jacobsen P and Rubin J. Ensuring public health’s future in a national-scale learning health system. American Journal of Preventative Medicine 2015;48(4):480–7. Available from: PMid:25700654.

59. Brody H and Miller F. The research-clinical practice distinction, learning health system and relationships. Hastings Centre Report 2013;43(5):41–7. Available from: PMid:24092591.

60. Vaismoradi M, Turunen H and Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nursing and Health Sciences 2013;15(3):398–405. Available from: PMid:23480423.

61. Stumme G. Formal concept analysis. In: Staab S and Studer R (Eds.), Handbook on Ontologies. Berlin, Germany: Springer, 2009, pp. 177–99. Available from:

62. Friedman C, Wong A and Blumenthal D. Achieving a nationwide learning health system. Science Transitional Medicine 2010;2(57):1–3. Available from:

63. Deeny S and Steventon A. Making sense of the shadows: Priorities for creating a learning healthcare system based on routinely collected data. BMJ Quality Safety 2015;24:505–15. Available from: PMid:26065466; PMCid:PMC4515981.

64. Bradley E, Curry LA, Ramanadhan S, Rowe L, Nembhard IM and Krumholz HM. Research in action: using positive deviance to improve quality of health care. BMC Implementation Science 2009;4:25.

65. Lewis G, Kirkham H and Vaithianathan R. How health systems could avert ‘triple fail’ events that are harmful, are costly, and result in poor patient satisfaction. Health Affairs 2013;32(4):669–76. Available from: PMid:23569046.

66. Foley T and Fairmichael F. Site visit to Geisinger Health Systems. The Learning Healthcare Project 2015. Available from: Accessed August 2017.

67. Garg A, Adhikari NK, Mcdonald H, Rosas-Arellano M, Devereaux PJ, Beyene J and Hayes RB. Effects of computerised clinical decision suport systems on practitioner performance and patient outcomes. Journal of the American Medical Association 2005;293(10):1223–38. Available from:

68. Wyatt J and Spiegelhalter D. Field trials of medical decision-aids: potential problems and solutions. Fifteenth Annual Symposium on Computer Applications in Medical Care. Washington, DC: American Medical Informatics Association, 1991.

69. Kawamoto K, Houlihan CA, Balas EA and Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. British Medical Journal 2005;330:765–8. Available from: PMid:15767266; PMCid:PMC555881.

70. Abernethy A, Ahmad A, Zafar SY, Wheeler JL, Reese JB and Lyerly HK. Electronic patient-reported data capture as a foundation of rapid learning cancer care. Medical Care 2010;48(6):S32–8. Available from: PMid:20473201.

71. Ye Y, Wamukoya M, Ezeh A, Emina JB and Sankoh O. Health and Demographic Surveillance Systems: a step towards full civil registration and vital statistics in sub-Saharan Africa? BMC Public Health 2012;12:741. Available from:

72. Ethier J. Integrating Resources for Translational Research: A Unified Approach for Learning Health Systems. Paris, France: Universite Pierre et Marie Curie, 2016.

73. Sacristan J. Patient-centered medicine and patient-oriented research: improving health outcomes for individual patients. BMC Medical Informatics and Decision Making 2013;13:6. Available from:

74. Mirnezami R, Nicholson J and Darzi A. Preparing for precision medicine. New England Journal of Medicine 2012;366(6):489–91. Available from: PMid:22256780.

75. Jameson J and Longo D. Precision medicine—personalized, problematic and promising. New England Journal of Medicine 2015;372(23):2229–34. Available from: PMid:26014593.

76. Harwood J, Butler C and Page A. Patient-centred care and population health: establishing their role in orthapaedic practice. The Journal of Bone and Joint Surgery 2016;98(10):e40. Available from: PMid:27194502.

77. Yu P, Artz D and Warner J. Electronic health records (EHRs): supporting ASCO’s vision of cancer care. ASCO Educational Book 2014;2014:225–31.

78. Cresswell, Bates D and Sheikh A. Ten key considerations for the successful optimisation of large-scale health information technology. Journal of the American Medical Informatics Association 2017;24(1):182–7.

79. Rioth M, Warner J, Savani BN and Jagasia M Next generation long term transplant clinics: improving resource utilisation and the wuality of care through health information technology. Bone Marrow Transplantation 2016;51(1):34–40. Available from: PMid:26367235; PMCid:PMC4915825.

80. Pritchard D, Moeckel F, Villa MS, Housman LT, McCarty CA and McLeod HL. Strategies for integrating personalised medicine into healthcare practice. Personalised Medicine 2017;14(2):141–52.

81. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic review: impact of health information technology on quality, efficiency and cost of medical care. Annals of Internal Medicine 2006;144(10):742–52. Available from: PMid:16702590.

82. Buntin M, Burke MF, Hoaglin MC and Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominately positive results. Health Affairs 2011;30(3):464–71. Available from: PMid:21383365.

83. Yu P. Knowledge bases, clinical decision support systems, and rapid learning in oncology. Journal of Oncology Practice 2015;11(2):e206–11. Available from: PMid:25715002.

84. Zalon M. Technology and continuously learning health systems. The Journal of Continuing Education in Nursing 2016;47(6):243–5. Available from: PMid:27232218.

85. Chorpita B, Daleiden E and Bernstein A. At the intersection of Health Information Technology and Decision Support: measurement feedback systems... and beyond. Administration and Policy in Mental Health and Mental Health Services Research 2016;43(3):471–7. Available from: PMCid:PMC5300738.

86. Etheredge L. Rapid learning: a breakthrough Agenda. Health Affairs 2014;33(7):1155–62. Available from: PMid:25006141.

87. Fihn S, Francis J, Clancy C, Nielson C, Nelson K, Rumsfeld J, et al. Insights fron advanced analytics at the Veterans Affairs Health Administration. Health Affairs 2014;33(7):1203–11. Available from: PMid:25006147.

88. Angus D. Fusing randomised trials with big data: the key to self-learning health care systems. Journal of the American Medical Association 2015;314(8):767–8. Available from: PMid:26305643.

89. Dreyer N. Making observational studies count: shaping the future of comparitive effectiveness research. Epidemiology 2011;22(3):295–7. Available from: PMid:21464648.

90. Muir Gray J. The shift in personalised and population medicine. The Lancet 2013;382(9888):200–1.

91. LeFevre M. From authority- to evidence-based medicine: are clinical practice guidelines moving us forwards or backwards? Annals of Family Medicine 2017;15(5):410–12. Available from: PMid:28893809.

92. Boonstra A and Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Services Research 2010;10(1):231.

93. Hillstead R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, et al. Can electronic medical record systems transform health care? Potential health benefits, savings and costs. Health Affairs 2005;24(5):1103–17. Available from: PMid:16162551.

94. AMA. Shared electronic medical records. 2016. Available from: Accessed 3 September 2017.

95. Davidson A. Creating value: unifying silos into public health business intelligence. eGEMS 2015;2(4):1172.

96. Denaxas S, Friedman CP, Geissbuhler A, Hemingway H, Kalra D, Kimura M, et al. Discussion of “combining health data uses to ignite health system learning.” Methods of Information in Medicine 2015;54(6):488–99. Available from: PMid:26538343.

97. Wilk A and Platt P. Measuring physicians’ trust: a scoping review with implications for public policy. Social Science and Medicine 2016;165:75–81. Available from: PMid:27497858.

98. Lean S, Guesgen HW, Hunter I and Tretiakov A. Working toward automated coding in general practice. Palmerston North, New Zealand: Massey University, 2016.

99. Faden R, Beauchamp T and Kass N. Informed consent, comparative effectiveness, and learning health care. New England Medical Journal 2014;370(8):766–8.

100. Marsolo K, Margolis PA, Forrest CB, Colletti RB and Hutton JJ. A digital architecture for a network-based learning health system - Integrating chronic care management, quality improvement and research. eGEMS 2015;3(1):1168. Available from: PMid:26357665; PMCid:PMC4562738.

101. Nishimura A, Carey J, Erwin PJ, Tilburt JC, Murad MH and McCormick JB. Improving understanding in the research informed consent process: a systematic review of 54 interventions tested in randomized control trials. BMC Medical Ethics 2013;14(28):28. Available from:

102. Feeley T, Sledge GW, Levit L and Ganz PA. Improving the quality of cancer care in America through health information technology. Journal of the American Medical Informatics Association 2014;21:772–5. Available from: PMid:24352553; PMCid:PMC4147622.

103. Hughes K, Ambinder EP, Hess GP, Yu PP, Bernstam EV, Routbort MJ, et al. Identifying health information technology needs of oncologists to facilitate the adoption of genomic medicine: recommendations from the 2016 American society of clinical oncology omics and precision oncology workshop. Journal of Clinical Oncology 2017;35:3153–9.


  • There are currently no refbacks.

This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT