Effectiveness of local support for the adoption of a national programme – a descriptive study

Christopher Pearce

Director of Research, Inner East Melbourne Medicare Local, 10 Lakeside Drive, Ringwood East, Victoria 3134, Australia

Jenny Bartlett

Healthcare Consultant, South Melbourne, Victoria, Australia

Adam McLeod

Director of Strategy and Integrated Care, Inner East Melbourne Medicare Local, Victoria, Australia

Paula Eustace

Research Officer, Inner East Melbourne Medicare Local, Victoria, Australia

Rod Amos

Manager of Integration Services, Inner East Melbourne Medicare Local, Victoria, Australia

Marianne Shearer

Chief Executive Officer, Inner East Melbourne Medicare Local, Victoria, Australia

Cite this article: Pearce C, Bartlett J, McLeod A, Eustace P, Amos R, Shearer M. Effectiveness of local support for the adoption of a national programme – a descriptive study. Inform Prim Care. 2014;21(4):171–178.

Copyright © 2014 The Author(s). Published by BCS, The Chartered Institute for IT under Creative Commons license http://creativecommons.org/licenses/by/4.0/.

Author address for correspondence:

Christopher Pearce

Director of Research

Inner East Melbourne Medicare Local

10 Lakeside Drive, Ringwood East

Victoria 3134, Australia

Email: cpearce@iemml.org.au


ABSTRACT

Background Change management in health care is a complex and time-consuming endeavour, and no less so in implementing technological systems. In deploying a nationwide programme, the personally controlled electronic health record (PCEHR), the Australian Government employed a number of national and local change management programmes.

Objective This article describes the processes undertaken and the experiences of introducing the PCEHR into 74 general practices across a specific area of metropolitan Melbourne.

Method An online survey was developed by an independent evaluator and offered to all participating practices. The response rate was 82%.

Results The deployment and testing of the eHealth infrastructure and the roll- out of the PCEHR were deeply supported through face-to-face, locally contextualised support processes. The area Medicare Local (ML), an organisation that provides support services to general practice and allied health in the community, provided support and programme coordination. This support occurred in the environment of a number of other initiatives to improve adoption.

Conclusion The impact and value of this support in the registration and adoption process was explored in an online survey and found to be the key factor in practice engagement and success. ML support was seen as instrumental in improving adoption and was more effective than other activities. This article highlights the role of local support, in this case, MLs, in the effective implementation of eHealth programmes across a range of stakeholder groups, in particular, general practice, and the potential for the lessons learned from the engagement model of such an entity to be more generally applied.

Keywords: computerised, eHealth, general practice, independent, medical record systems, practice association, social change

What this paper adds:

• Local support is more valued, and is more likely to produce change than central support

• Large-scale programmes must take into account small-scale needs

• Incentives must be balanced by support

• General practice is able to be flexible and adopt change rapidly.


INTRODUCTION

Worldwide, health care is in the grips of a fundamental change in keeping with society as a whole. The adoption of computers and digital technology is transforming health care across the globe, albeit in different ways and at different levels.1,2 It follows that this change is accompanied by significant efforts to facilitate change, with varying results. Most literature examines the often problematic change processes in organisations with central command and control systems.36 The resistance to change has been ascribed to the failure to appreciate the extent of disruption of moral and ethical processes, including medico-legal ones.7 Locally focussed programmes have shown effectiveness in the UK.8 Their primary care information systems programme represented a practice-based data quality exercise in the context of their shared record programme.9 Practices were required to participate in the local activities to enable uploading to the shared ‘Spine’. This article describes the activities undertaken in Australia, without such a central system, for adopting a large-scale Australia-wide roll-out and adoption of the personally controlled electronic health record (PCEHR).

General practice represents the cornerstone of primary care in Australia and is made up of multiple small businesses with fragmented systems and funding mechanisms that challenge a ‘whole-of-system’ approach to clinical care.10 A particular solution to this problem was the establishment of the divisions of general practice programme, some 18 years ago. Divisions began as 120 Australian Governmentfunded organisations proposed to support general practice in change management methodology by engaging directly with practices. Importantly and different to other meso-level structures internationally, divisions did not hold funds for general practice, but engaged with them. Divisions were shown to be effective in delivering change,11 had profile over other peak bodies12 and were particularly effective in helping general practice adopt computers in the late 1990s.13 Subsequent policy changes sought to make divisions more efficient through amalgamations and being recreated as Medicare Locals (MLs) in addition to broadening their focus to include allied health and an extended focus on consumer engagement. There are currently 61 MLs nationally. The government has just announced that the Medicare Local program will cease next year, to be replaced by larger ‘Primary Care Networks’.

Divisions proved effective in facilitating the computerisation of general practices.8 Using the ‘adoption triangle’ that balances the three arms of need, incentives and support, divisions provided the support component to assist general practitioners (GPs) in the 1990s reach almost full computerisation.14 Complex prescribing rules provided the need, and government financial aid provided the incentives, via the practice incentive programmes (PIP), including electronically managed PIP, the ePIP.13 The PIP is aimed at supporting general practice activities that encourage continuing improvements, quality care, enhance capacity and improve access and health outcomes for patients (www.medicareaustralia.gov.au/provider/incentives/pip/index.jsp). The ePIP, in particular, was targeted at the adoption and integration of unique health identifiers into practice systems.

In the May 2010 budget, the Australian government announced plans and funding to develop and implement a national-scale PCEHR.15 The decision was based on the 2008 National Health and Hospitals Reform Commission’s recommendation to establish a PCEHR as one of the recommendations to create ‘an important systemic opportunity to enable person-centred care, support informed consumer decision making, improve quality and safety of care, reduce waste and inefficiency, and improve continuity and health outcomes for patients.16 Funding of AUD467 million was provided to design and build the first release of the PCEHR. Implementing the programme represented the four adoption problems related to behaviour, cognition, economy and technology.17

The PCEHR is a patient-focussed record consisting of a view service that allows collation of clinical information deposited in approved repositories. Clinician and patient provided information includes demographic data, prescribing information, discharge summaries and general practice summaries akin to UK’s summary care record.18 Information is available to both clinicians and consumers. A complete description is published elsewhere.15 Implementation of the programme required software in practices to adopt new health identifiers, secure exchange of information via public key infrastructure and the creation of new types of standards- based clinical documents. Legislation and regulation required the creation of dedicated roles and responsibilities within the practice.

In conjunction with design and implementation, there was an extensive change and adoption programme. One of the planks of the change and adoption programme was to fund exemplar programmes in geographical areas around pre- existing eHealth expertise, using MLs. Given the need for collaborative adoption processes across potentially ‘ disruptive’ sectors,19,20 MLs, with their extensive connections to practices,21 were ideal to implement such programmes. Other activities outside the ML programmes included the appointment of a ‘Change and Adoption Partner’ to develop online and other resources (which can be found at www.ehealth.gov.au) and joint funding to two peak bodies, the Royal Australian College of General Practitioners (RACGP) and the Australian College of Rural and Remote Medicine, to deliver seminars across the country. In addition, the Australian Medical Association (AMA) was contracted to develop a code of conduct for electronic records.

Unlike other earlier eHealth initiatives, such as electronic prescribing, there was no perceived need for the PCEHR at the practice level. GPs were expected to prepare and upload information to the PCEHR with no benefit to them, and in fact, would likely create work given the need to ensure data accuracy. The PIP was configured to support the adoption of the necessary infrastructure of digital certificates and health identifiers, electronic prescribing and structured records. The challenge was then to deliver support in that environment.

The PCHER implementation programme and the PCEHR itself have faced significant issues and criticism, and are, at the time of writing, under review by government. The discussion in this article focusses on the role of MLs in the engagement of a national eHealth programme, but indicates the possibilities for any other institutional support. Quantitative data collected during the projects had demonstrated the success of the project (in terms of certificates acquired and practices connected), yet what had been instrumental in creating that success was not clear. The aim of this article was to explore the experience of the programme and the nature of the change in more detail.


METHOD

The Inner East Melbourne Medicare Local (IEMML) is situated in the metropolitan east of Melbourne, servicing 174 member practices across a catchment of 620,000 people. IEMML participated in two separate eHealth implementation programmes over 18 months, funded by the Department of Health and Ageing (DoHA) and supported by the National eHealth Transition Authority (NEHTA): Wave 1 transition to PCEHR and eHealth change and adoption. IEMML eHealth support officers spent, on average, between 10 and 15 h with each practice. Box 1 outlines the activities undertaken.

To better understand the adoption framework, an independent evaluator (JB) undertook an evaluation on behalf of IEMML. The focus was on the experiences of 90 participating member practices and their interaction with the external agencies involved, in particular, the support provided by IEMML staff. The method used was an online survey, available for completion over a two-week period. It included standardised responses to specific questions and free text comments.

In June 2013, 90 general practices within IEMML were invited to complete a survey to evaluate their experiences of the eHealth pilot programme over the previous 18 months. Evaluations were independent of the ePIP requirements.

We analysed the data from a realist perspective, using a methodology previously used to explore the success and failure of the UK National programme for IT and practice nursing in Australia.22,23 A realist perspective is useful in assessing complex interventions, as it aims to develop explanatory analyses of why and how these interventions may work in particular settings and contexts. The realist’s mantra is: ‘Context (C)’ plus causal link with an appropriate ‘Mechanism (M)’ results in an ‘Outcome (O)’; in other words, ‘ C + M = O’. Part of the realist perspective is that effects are reported according to the three Ws: ‘What Works, for Whom, and in What circumstances’. Our results are reported according to the C + M = O structure.


CASE STUDY/RESULTS

Context

Eighty-four individual staff responses were received from 74 practices (82% response rate). The majority of the staff who completed the survey occupied senior positions with their organisations as practice managers (50%) or GPs (45%). Two-thirds of the respondents were the responsible officer for eHealth (a legislated position with defined responsibilities) with their organisations and one-quarter acted as their organisation’s eHealth maintenance officer as defined in the Personally Controlled Electronic Records Act (2012).

Responses were received from all sizes of general practices, with over half (52%) from practices employing 2–5 GPs (Table 1). Most practices (85%) are currently providing afterhours services. Of those who are not currently providing such services, a substantial minority expressed their interest in finding out about support from IEMML to assist their practice in offering after-hours services in the future.

Responding practices use a range of clinical and administrative software packages. Medical Director was the dominant clinical software program in use (in 70% of practices) and Pracsoft, the most commonly used administrative software program (in 67% of practices).

Almost half of the respondents (46%) reported that their practices participated in a targeted mail-out to their patients, with the aim of recruiting patients to the PCEHR. Those that did use this communication tool were cautiously supportive of its value. There was reasonable awareness amongst respondents of the local eHealth letter drop and advertising campaigns used to raise patient awareness, but less confidence that these strategies had much traction in terms of delivering new patient registrations to the PCEHR in their organisations.

Box 1: Change activities provided

Pre-ePIP

• Initial consultations, promotion and education about the PCEHR.

• Assisted practices to sign for PCEHR certificate. Pre-populated forms were created to simplify the process for the practices.

• Provided advice on data cleansing.

• Value add activities:

◦ Located MBS site certificates (physical media), identified certificate store location on server, located MBS claiming certificate password (original documentation or extracted from certificate store)

◦ Promoted the benefits of upgrading operating software (as a start) and possibly considering updating hardware to maximise security.

Post-ePIP

• Provided training to third party IT providers about how to manage the eHealth site certificates and configuration in the practice software.

• Initial configuration of certificates to enable connection to the PCEHR system.

• Training on how to use the GP software to upload shared health summaries.

• Supported practice manager to:

◦ Collect individual practitioner identifiers

◦ Demonstrate linking the health identifiers via administration section of the practice software

• Practices were assisted to complete the following pre-populated paperwork

◦ Application for health care provider identifier organisation

◦ Application for National Authentication for Health Services (NASH) certificate

◦ Participation agreement

• Developed eHealth resource kit that was intended to be used to hold all collateral relevant to eHealth: the resource kit was an A4 folder that included

◦ Cheat sheets, step-by-step Instructions, glossary, key contact details (HI service, system operator, and so on)

◦ Document/letter folder

◦ Inserts that could hold up to three CDs (NASH, MBS claiming Certificate)

• Secure messaging delivery (SMD) deployment requirements

◦ Confirmed which SMD provider they currently used and confirmed which provider they wanted to use moving forward.

◦ Made sure their certificate of currency was stored safely

• Assisted practices to log onto HPOS to record the correct endpoint location service.

• Confirmed connection to the PCEHR and completed a test transmission.

Table 1 Practice size

Mechanism

The organisational leadership of eHealth implementation within participating practices was seen to come predominantly from GPs and practice managers and external leadership came predominantly from the ML. Not surprisingly, eHealth pilot practices reported a high degree of knowledge and awareness of eHealth and eHealth readiness. The main factors that motivated practices to engage with the eHealth implementation pilot programme were (Table 2):

• Their strong existing relationship with the driver ML

• Access to financial incentives for participation

• A desire to see their practices to be early adopters of the latest technology

• Leadership by interested individual GPs within their practices

GPs and practice managers were not surprisingly seen to be leading eHealth implementation within responding practices, with GPs identified as proving strong organisational leadership in 46% of responses and practice managers in 41% of responses. Interestingly, a substantial minority of responses (13%) identified reception staff, practice nurses or other staff as their organisation’s eHealth leaders.

THE ROLE OF THE ML

Table 3 lists and ranks the overall types of support practices received. There was a consistent view across all practices that their organisation’s participation in the eHealth implementation pilot had been beneficial and that the ML staff had been of material assistance in their roll-out of eHealth strategies in their practices. The majority of respondents identified the assistance of the ML staff in adoption of eHealth as a very important determinant of the speed and success in meeting the programme goals. Indeed, several respondents noted, in their commentary, that they considered the support they received from the ML staff to have been critically important to their organisation’s eHealth implementation achievements.

Table 2 Motivation

There was a somewhat wider range of opinion regarding the extent to which the education and training support provided by clinical software vendors met practice needs. Overall, the education and training provided by the RACGP was not perceived as particularly useful, and no comments were made about programmes and resources provided by the change and adoption partner. When asked to evaluate the technical support offered to their organisations, the support provided by the ML was strongly recognised and valued. IEMML was seen as the predominant source of leadership and direction for eHealth implementation external to their own organisation, with significantly fewer practices identifying Medicare Australia, RACGP, NEHTA and the AMA as key influencers for their organisation’s eHealth programmes.

Outcome

Table 4 represents the results of respondents to views on the programme outcomes. Relatively few practices (13%) reported facing resistance to eHealth implementation amongst their staff, with additional commentary identifying that individual doctors, in some practices, were perceived to be the most common source of resistance to change. This finding contradicts widespread media coverage attesting that GPs were resistant.

However, most practices (59%) reported having faced some eHealth implementation issues over the course of the eHealth pilot implementation, with a wide range of such issues noted. The most frequent issue of concern was a lack of awareness of and support for eHealth amongst their patients and the community in general.

Ongoing barriers to effective eHealth implementation were identified as:

• Inadequate patient and broader community engagement with the PCEHR

• The time required for doctors for use of the PCEHR

• Inadequate remuneration for time devoted to use of the PCEHR by GPs

Respondents were generally satisfied with their decision to implement eHealth systems in their practices. They indicated that their staff had gained knowledge and skills in eHealth and that there had been positive impacts of eHealth on the quality of patient care in their practices. However, eHealth was not generally seen to, as yet, improve the efficiency of care processes or information and communication flow between health care facilities. There was little support for the notion that eHealth programmes and processes have made more time available for delivering patient care.


DISCUSSION

The right approach to developing, implementing and routinely using effective eHealth can help health care providers use eHealth systems to improve the safety and quality of care and expand their capacity to meet the increasing demand for service provision within available resources. Greater efficiencies in health care are dependent on exploiting the power of electronic information to help ensure that patients get the right care, involving the right clinicians, at the right time, to deliver the right outcomes. This article represents the experience of one ML, but uptake data from the national programme suggest that the experience would be similar in the other tasked MLs.

Table 3 Leadership sources

Table 4 Attitudes to implementation programme

The realist framework (C + M = O) allows us to better understand the mechanism of change beyond simply observing it. The relevant context is of a distributed, uncoordinated setting of general practice, with needs and drivers differing from those of the policy initiative. As previously observed, general practice behaves in ways like a complex adaptive system, where the actions of many are needed to create a whole, functioning system.8 The mechanism we applied was of extended and focussed support, driven by incentives but not need, and in the context of other adoption initiatives. Our findings are that the support was both valued and was the significant factor in driving change (outcome). This effect was much stronger than that provided by other programmes.

The findings emphasise the multiple foci that must be placed in instituting change. The lack of need (clinicians identified no particular use of the PCEHR) was countered by the PIP incentive, but that alone was not enough. In keeping with earlier experience,7 the intensive and contextually focussed support was more valued and effective than the ‘when needed’ online support and the more traditional seminar programmes. This is an indicator of future directions for any such programmes, and adds to the literature that supports local (versus national) change and adoption activities.

Several factors have to be in place for eHealth to succeed. The eHealth applications must focus on meeting particular needs and change a particular clinical or operational process. Smart people and multidisciplinary teams are required to drive the change needed to realise the benefits from eHealth. It is not enough to replicate the technology component of a proven eHealth investment; the individual organisational dynamics and needs must be addressed. Individual eHealth applications need to be part of an evolving series of investments to create a sustained eHealth benefit.

Neither eHealth applications nor information by itself brings benefits. Gains come from changing processes or working practices. Types of potential benefit from eHealth include better care, safer care and more efficient care.24 In keeping with other research, the influence of local organisations, such as MLs, far exceeds the influence of distant, peak bodies.

Approaching eHealth in this manner recognises that there is a ‘journey to benefits’ that must be taken by all MLs to assist health care providers and help consumers realise the benefits of eHealth. This journey commences with eHealth awareness and then progresses to eHealth readiness, adoption and meaningful use of eHealth, culminating in the realisation of improved societal health and economic outcomes.

The evaluation indicates that participating practices are still very much in the process of early adoption of eHealth systems. There is still relatively modest reported everyday experience of use of the PCEHR. A significant proportion of practices do not have ongoing systems in place to orientate, educate and train new staff in eHealth. Many practices report that their staff would not know what to do if an issue arose with eHealth that potentially compromised patient care.

The widely held view of survey respondents was that the major impediment to future success was a lack of interest for, knowledge of and support for eHealth amongst the general community. Experience in other health care settings would suggest that the best way to overcome such a barrier is via clinicians directly advocating with patients for the desired intervention. Using the power of clinician advocacy to enhance consumer awareness of the PCEHR could well be a key theme for future work on progressing its implementation within practices working with MLs in the future. Other means of increasing interest in the general community would be to encourage the use of the consumer facing parts of the programme, through the provided online consumer portal.15 Importantly, these local adoption activities should be a key focus of the new primary care networks.

Although it is encouraging that many respondents already believe that eHealth implementation has improved the quality of the care processes in their practices, it will be some time before eHealth systems are meaningful in everyday use in the care of most patients in the participating practices and before there would be the expectation of substantive realised benefits to care processes and outcomes.

Conflicts of interest

All authors apart from JB are employed by IEMML. CP was the clinical design lead for the PCEHR programme prior to this study.


REFERENCE

1. Jha AK, Doolan D, Grandt D, Scott T and Bates DW. The use of health information technology in seven nations. International Journal of Medical Informatics 2008 Dec;77(12):848–54. PubMed PMID: 18657471. eng. http://dx.doi.org/10.1016/j.ijmedinf.2008.06.007. PMid:18657471.

2. Jones SS, Rudin RS, Perry T and Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Annals of Internal Medicine 2014 Jan 7;160(1):48–54. PubMed PMID: 24573664.

3. Greenhalgh T, Stramer K, Bratan T, Byrne E, Mohammad Y and Russell J. Introduction of shared electronic records: multisite case study using diffusion of innovation theory. British Medical Journal 2008;337:a1786. PubMed PMID: 18948344. Epub 2008/10/25. eng.

4. Scott JT, Rundall TG, Vogt TM and Hsu J. Kaiser Permanente’s experience of implementing an electronic medical record: a qualitative study. British Medical Journal 2005 Dec 3;331(7528): 1313–6. PubMed PMID: 16269467. http://dx.doi.org/10.1136/bmj.38638.497477.68. PMid:16269467; PMCid:PMC1298854.

5. Protti D, Bowden T and Johansen I. Adoption of information technology in primary care physician offices in New Zealand and Denmark, part 5: final comparisons. Informatics in Primary Care 2009;17(1):17–22. PubMed PMID: 19490769.

6. Cresswell KM, Bates DW and Sheikh A. Ten key considerations for the successful implementation and adoption of large-scale health information technology. Journal of the American Medical Informatics Association 2013 Jun;20(e1):e9–13. PubMed PMID: 23599226. Pubmed Central PMCID: 3715363.

7. Garrety K, McLoughlin I, Wilson R, Zelle G and Martin M. National electronic health records and the digital disruption of moral orders. Social Science & Medicine 2014 Jan;101:70–7. PubMed PMID: 24560226.

8. Teasdale S and Bainbridge M. Interventions for improving information management in family practice. Studies in Health Technology and Informatics 1997 Feb;43:806–10.

9. Horsfield P. Trends in data recording by general practice teams: an analysis of data extracted from clinical computer systems by the PRIMIS project. Informatics in Primary Care 2002;10(4):227–34.

10. Pearce C, Phillips C, Hall S, Sibbald B, Porritt J, Yates R et al. Following the funding trail: financing, nurses and teamwork in Australian general practice. BMC Health Services Research 2011;11:38. PubMed PMID: 21329506. Pubmed Central PMCID: 3050696. http://dx.doi.org/10.1186/1472-6963-11-38. PMid:21329506; PMCid:PMC3050696.

11. Scott A and Coote W. Whither divisions of general practice? An empirical and policy analysis of the impact of divisions within the Australian health care system. Medical Journal of Australia 2007 Jul 16;187(2):95–9. PubMed PMID: 17635092. English. PMid:17635092.

12. Pearce C, Shearer M, Phillips C, Hall S, Kljakovic M, Glasgow NJ et al. Views of GPs and practice nurses on support needed to respond to pandemic influenza: a qualitative study. Australian Health Review 2011 Feb;35(1):111–5. PubMed PMID: 21367342. Epub 2011/03/04. eng.

13. Pearce C. The adoption of computers by Australian general practice – a complex adaptive systems approach. OMICS Journal of General Practice 2013 August;1(3):1–3.

14. Kidd M. The computerisation of Australian general practice 1998–2001 – what did we get for AU$15,000,000. Informatics in Primary Care 2002;10(1):25–9.

15. Pearce C and Bainbridge M. A personally controlled electronic health record for Australia. Journal of the American Medical Informatics Association 2014 Mar 20;21(4):707-13. PubMed PMID: 24650635.

16. National Health and Hospitals Reform Commission. A healthier future for all Australians - final report June 2009. Canberra: Commnwealth of Australia; 2008.

17. Pare G, Raymond L, de Guinea A, Poba-Nzaou P, Trudel M, Marsan J et al. Barriers to organizational adoption of EMR systems in family physician practices: a mixed-methods study in Canada. International Journal of Medical Informatics 2014;83(8):548-58.

18. Coiera E. Do we need a national electronic summary care record? Medical Journal of Australia 2011 Jan 17;194(2):90–2. PubMed PMID: 21241224.

19. Garrety K, McLoughlin I and Zelle G. Disruptive innovation in health care: business models, moral orders and electronic records. Social Policy and Society 2014 March:1–14. Epub 18 Dec 2013.

20. Garrety K, Dalley A, McLoughlin I, Wilson R and Yu P. Managing collaboration across boundaries in health information technology projects. Studies in Health Technology and Informatics 2012;178:58–63. PubMed PMID: 22797020.

21. Pearce C, Shearer M, Gardner K, Kelly J and Xu TB. GP networks as enablers of quality of care: implementing a practice engagement framework in a general practice network. Australian Journal of Primary Health 2012;18(2):101–4. PubMed PMID: 22551830. Epub 2012/05/04. eng.

22. Swindells M and de Lusignan S. Lessons from the English National Programme for IT about structure, process and utility. Studies in Health Technology and Informatics 2012;174:17–22. PMid:22491103.

23. Phillips CB, Pearce C, Hall S, Kljakovic M, Sibbald B, Dwan K et al. Enhancing care, improving quality: the six roles of the general practice nurse. Medical Journal of Australia 2009 Jul 20;191(2):92–7. PubMed PMID: 19619094. Epub 2009/07/22. eng.

24. Stroetmann K, Jones T, Dobrev A and Stroetmann V. eHealth is worth it, the economic benefits of implemented eHealth solutions at ten European sites. Office for Official Publications of the European Communities. Bonn. 2006 PubMed PMID: 8412074682786243460.

Refbacks

  • There are currently no refbacks.


This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT