Coding errors in an analysis of the impact of pay-for-performance on the care for long-term cardiovascular disease: a case study

Simon de Lusignan, Benjamin Sun, Christopher Pearce, Christopher Farmer, Paul Stevens, Simon Jones


Objective There is no standard method of publishing the code ranges in research using routine data. We report how code selection affects the reported prevalence and precision of results.

Design We compared code ranges used to report the impact of pay-for-performance (P4P), with those specified in the P4P scheme, and those used by our informatics team to identify cases. We estimated the positive predictive values (PPV) of people with chronic conditions who were included in the study population, and compared the prevalence and blood pressure (BP) of people with hypertension (HT).

Setting Routinely collected primary care data from the quality improvement in chronic kidney disease (QICKD—ISRCTN56023731) trial.

Main outcome measures The case study population represented roughly 85% of those in the HT P4P group (PPV = 0.842; 95%CI = 0.840–0.844; < 0.001). We also found differences in the prevalence of stroke (PPV = 0.694; 95%CI = 0.687– 0.700) and coronary heart disease (PPV = 0.166; 95%CI = 0.162–0.170), where the paper restricted itself to myocardial infarction codes.

Results We found that the long-term cardiovascular conditions and codes selected for these conditions were inconsistent with those in P4P or the QICKD trial. The prevalence of HT based on the case study codes was 10.3%, compared with 11.8% using the P4P codes; the mean BP was 138.3 mmHg (standard deviation (SD) 15.84 mmHg)/79.4 mmHg (SD 10.3 mmHg) and 137.3 mmHg (SD 15.31)/79.1 mmHg (SD 9.93 mmHg) for the case study and P4P populations, respectively (< 0.001).

Conclusion The case study lacked precision, and excluded cases had a lower BP. Publishing code ranges made this comparison possible and should be mandated for publications based on routine data.


clinical coding; computerised; heart diseases; hypertension; incentive; medical records system; reimbursement; research design

Full Text:



Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1999;276(8):637–9.

Mäkelä M, Kaila M and Stein K. Mind sharpeners for scientists: the EQUATOR Network. International Journal of Technology Assessment in Health Care 2011;27(2):99–100. PMid:21429289.

Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P and Rigby M. STARE-HI--statement on reporting of evaluation studies in health informatics. Yearb Medical Informatics 2009;23–31. PMid:19855867.

de Lusignan S. Codes, classifications, terminologies and nomenclatures: definition, development and application in practice. Informatics in Primary Care 2005;13(1):65–70. PMid:15949178.

Serumaga B, Ross-Degnan D, Avery AJ, Elliott RA, Majumdar SR, Zhang F, et al. Effect of pay for performance on the management and outcomes of hypertension in the United Kingdom: interrupted time series study. BMJ 2011;25;342:d108.

Stevens PE, Farmer CK and de Lusignan S. Effect of pay for performance on hypertension in the United Kingdom. American Journal of Kidney Diseases 2011;58(4):508–11. PMid:21816527.

NHS Department of Health Informatics Directorate. Technology Reference data Update Distribution. NHS Read Browser version 11. (accessed 26 March 2014).

NHS Primary Care Commissioning. QOF Implementation: Business Rules. (accessed 26 March 2014).

de Lusignan S, Gallagher H, Chan T, Thomas N, van Vlymen J, Nation M, et al. The QICKD study protocol: a cluster randomised trial to compare quality improvement interventions to lower systolic BP in chronic kidney disease (CKD) in primary care. Implementation Science 2009;4:39.

de Lusignan S, Gallagher H, Jones S, Chan T, van Vlymen J, Tahir A, et al. Using audit-based education to lower systolic blood pressure in chronic kidney disease (CKD): results of the quality improvement in CKD (QICKD) trial [ISRCTN: 56023731]. Kidney International 2013;84(3):609–20. 10.1038/ki.2013.96. PMid:23536132; PMCid:PMC3778715.

NHS Information Centre for Health and Social Care. QOF 2009/10 Data Tables. talogue?productid=5204&topics=0%2fPrimary+care+service s&kwd=Q&sort=Relevance&size=10&page=6#top (accessed 26 March 2014).

de Lusignan S, Tomson C, Harris K, van Vlymen J and Gallagher H. Creatinine fluctuation has a greater effect than the formula to estimate glomerular filtration rate on the prevalence of chronic kidney disease. Nephron Clinical Practice 2011;117(3):c213–24. PMid:20805694.

Wilcox M. Illogical placing of codes within a clinical classification. Informatics in Primary Care 2009;17(2):131; discussion 132. PMid:19807955.

Gomez GB, de Lusignan S and Gallagher H. Chronic kidney disease: a new priority for primary care. British Journal of General Practice 2006;56(533):908–10. PMid:17132377; PMCid:PMC1934049.

Etheredge LM. Creating a high-performance system for comparative effectiveness research. Health Affairs (Millwood) 2010;29(10):1761–7. PMid:20921473.

Sullivan P and Goldmann D. The promise of comparative effectiveness research. JAMA 2011;305(4):400–1. PMid:21266687.

Hirsch BR, Giffin RB, Esmail LC, Tunis SR, Abernethy AP and Murphy SB. Informatics in action: lessons learned in comparative effectiveness research. Cancer Journal 2011;17(4): 235–8. PMid:21799331.

Pace WD, Cifuentes M, Valuck RJ, Staton EW, Brandt EC and West DR. An electronic practice-based network for observational comparative effectiveness research. Annals of Internal Medicine 2009;151(5):338–40. PMid:19638402.

Devoe JE, Gold R, McIntire P, Puro J, Chauvie S and Gallia CA. Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers. Annals of Family Medicine 2011;9(4):351–8. PMid:21747107; PMCid:PMC3133583.

de Lusignan S, Wells SE, Hague NJ and Thiru K. Managers see the problems associated with coding clinical data as a technical issue whilst clinicians also see cultural barriers. Methods of Information in Medicine 2003;42(4):416–22. PMid:14534643.

de Lusignan S and van Weel C. The use of routinely collected computer data for research in primary care: opportunities and challenges. Family Practice 2006;23(2):253–63. PMid:16368704.

Tai TW, Anandarajah S, Dhoul N and de Lusignan S. Variation in clinical coding lists in UK general practice: a barrier to consistent data entry? Informatics in Primary Care 2007;15(3):143–50. PMid:18005561.

Pearce C, Gardner K, Shearer M and Kelly J. A divisions worth of data. Australian Family Physician 2011;40(3):167–70. PMid:21597524.

de Lusignan S and Mimnagh C. Breaking the first law of informatics: the Quality and Outcomes Framework (QOF) in the dock. Informatics in Primary Care 2006;14(3):153–6. PMid:17288700.

Iezzoni LI, Heeren T, Foley SM, Daley J, Hughes J and Coffman GA. Chronic conditions and risk of in-hospital death. Health Services Research 1994;29(4):435–60. PMid:7928371; PMCid:PMC1070016.

McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, et al. Does clinical evidence support ICD-9-CM diagnosis coding of complications? Medical Care 2000;38(8):868–76. PMid:10929998.

de Lusignan S, Khunti K, Belsey J, Hattersley A, van Vlymen J, Gallagher H, et al. A method of identifying and correcting miscoding, misclassification and misdiagnosis in diabetes: a pilot and validation study of routinely collected data. Diabetic Medicine 2010;27(2):203–9. PMid:20546265.

de Lusignan S, Sadek N, Mulnier H, Tahir A, Russell-Jones D and Khunti K. Miscoding, misclassification and misdiagnosis of diabetes in primary care. Diabetic Medicine 2012;29(2): 181–9. http://dx.doi.org10.1111/j.1464-5491.2011.03419.x.

Hassan Sadek N, Sadek AR, Tahir A, Khunti K, Desombre T and de Lusignan S. Evaluating tools to support a new practical classification of diabetes: excellent control may represent misdiagnosis and omission from disease registers is associated with worse control. International Journal of Clinical Practice 2012;66(9):874–82. PMid:22784308; PMCid:PMC3465806.

Blak BT, Thompson M, Dattani H and Bourke A. Generalisability of The Health Improvement Network (THIN) database: demographics, chronic disease prevalence and mortality rates. Informatics in Primary Care 2011;19(4):251–5. PMid:22828580.

Bourke A, Dattani H and Robinson M. Feasibility study and methodology to create a quality-evaluated database of primary care data. Informatics in Primary Care 2004;12(3):171–7. PMid:15606990.

de Lusignan S, Stephens PN, Adal N and Majeed A. Does feedback improve the quality of computerized medical records in primary care? Journal of the American Medical Informatics Association 2002;9(4):395–401. PMid:12087120; PMCid:PMC346626.

Kumarapeli P, Stepaniuk R, de Lusignan S, Williams R and Rowlands G. Ethnicity recording in general practice computer systems. Journal of Public Health (Oxford) 2006;28(3):283–7. PMid:16840765.

Clinical Informatics Research Group. QICKD Dictionary. (accessed 26 March 2014).

Clinical Informatics Research Group. Osteoporosis Dictionary. (accessed 26 March 2014).

de Lusignan S, Liaw ST, Michalakidis G and Jones S. Defining datasets and creating data dictionaries for quality improvement and research in chronic disease using routinely collected data: an ontology-driven approach. Informatics in Primary Care 2011;19(3):127–34. PMid:22688221.

Liaw ST, Rahimi A, Ray P, Taggart J, Dennis S, de Lusignan S, et al. Towards an ontology for data quality in integrated chronic disease management: a realist review of the literature. International Journal of Medical Informatics 2012. doi:pii: S1386-5056(12)00193-1. PMid:23122633.



  • There are currently no refbacks.

This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT