Accuracy of screening tools for Pap smears in general practice

Catherine Harding

The University of Notre Dame Australia, School of Medicine Sydney – Rural Clinical School, Australia

Alexa Seal

The University of Notre Dame Australia, School of Medicine Sydney – Rural Clinical School, Australia

Geraldine Duncan

The University of Notre Dame Australia, School of Medicine Sydney – Rural Clinical School, Australia

Ronald Huynh

The University of Notre Dame Australia, School of Medicine Sydney – Rural Clinical School, Australia

Robert McWilliam

The University of Notre Dame Australia, School of Medicine Sydney – Rural Clinical School, Australia

Louis Pilotto

University of New South Wales, Faculty of Medicine – Rural Clinical School, Australia

Stephanie Blake

University of New South Wales, Faculty of Medicine – Rural Clinical School, Australia

Ken Mackey

Lockhart Medical Practice, Australia

Author address for correspondence

Catherine Harding

The University of Notre Dame Australia,

School of Medicine Sydney – Rural Clinical School,

5050 Hardy Avenue, Wagga Wagga,

NSW 2650, Australia.

Email: catherine.harding@nd.edu.au

Cite this article: Harding C, Seal A, Duncan G, Huynh R, McWilliam R, Pilotto L, Blake S and Mackey K. Accuracy of screening tools for Pap smears in general practice. J Innov Health Inform. 2016;23(3):555–559.

Copyright © 2016 The Author(s). Published by BCS, The Chartered Institute for IT under Creative Commons license http://creativecommons.org/licenses/by/4.0/


ABSTRACT

Background Data extraction tools (DETs) are increasingly being used for research and audit of general practice, despite their limitations.

Objective This study explores the accuracy of Pap smear rates obtained with a DET compared to that of the Pap smear rate obtained with a manual file audit.

Method A widely available DET was used to establish the rate of Pap smears in a large multi-general practice (multi-GP) in regional New South Wales followed by a manual audit of patient files. The main outcome measure was identification of possible discrepancies between the rates established.

Results The DET used significantly underestimated the level of cervical screening compared to the manual audit. In some instances, the patient file contained phone/specialist record of Pap smear conducted elsewhere, which accounted for the failure of the DET to detect some smears. Those patients who had Pap smears whose pathology codes differed between time intervals, i.e. from different pathology providers or from within the same provider but using a different code, were less likely to have had their most recent Pap smear detected by the DET (p < 0.001).

Conclusion Data obtained from DETs should be used with caution as they may not accurately reflect the rate of Pap smears from electronic medical records.

How this fits in DETs are increasingly being used for research and audit of general practice. This study explores the accuracy of Pap smear rates obtained with a DET compared to that of the Pap smear rate obtained with a manual file audit The DET tested significantly underestimated the level of cervical screening compared to manual screening. Data obtained from DETs should be used with caution as they may not accurately reflect the rate of Pap smears from electronic medical records.

Keywords: cervical smear, data extraction tools, electronic medical records, Pap smear


INTRODUCTION

With the development of computer systems in Australian general practice, the potential has come for increased access to primary healthcare information on a much larger scale1 and the potential for evidence-based information to improve healthcare.2 This study attempts to explore the accuracy of Pap smear data available through a primary care database and accessible via an extraction tool.

It has been recognised that data extraction tools (DETs) have limitations relating to data entry and incomplete data.3 Current DETs do not work with all clinical software programs and the way in which the software architecture of clinical systems has been established does not easily lend itself to extraction of data.1 In this study, the screening rate obtained via a DET was compared to the rate obtained using a ­practice audit of patient files for a large multi-general practice ­(multi-GP) practice in regional New South Wales.

Although there has been a decline in the incidence of ­cervical cancer in Australia, there are still a substantial number of women in New South Wales who are either ­under-screened or who have never been screened,4 particularly rural women.5 One barrier to addressing this issue is the quality of available data relating to under-screened women and previous studies have shown variations in rates depending on the source of the data.6

This cross-sectional study attempted to explore the ­reliability of Pap smear data available through a primary care databases and accessible through a DET compared with that of data available from manual audit.


METHODS

The focus of this report is a large multi-GP practice in a large regional centre in NSW, Australia. Two DETs were ­trialed but only one was compatible with the practice software. The ­current rate of Pap smears for all women aged 20–69 was determined via the compatible DET using multiple search terms and synonyms for Pap smear (Table 1).

Table 1 List of search terms/synonyms for Pap smear used within the DET

A manual audit of files from a random sample of 100 patients within each of the four categories (listed in Table 2) output by the DET was undertaken to establish the ­accuracy of the tool in terms of recording Pap smears. This audit involved inspection of the pathology requests and results and review of specialists’ letters and other scanned documents. Outcome measures included the identification of relative rates of Pap smears and deficiencies in recording within the practice (number of women in the practice with hysterectomy, Pap smear from an outside source and Pap smear from another pathology provider). Sensitivity and specificity were calculated for the four groups as a whole. Rates were compared by Chi-square tests and analysed using OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version 2.3.1 (Atlanta, USA). A two tailed p-value of <0.05 was considered significant.


RESULTS

Screening rates for eligible female patients are presented in Table 2. The DET indicated that more than half of all eligible female patients (aged 20–69, n = 2625) had never had a Pap smear, and that only 348 (13.3%) had had a Pap smear within the last two years, the current guidelines for cervical cancer monitoring.

A random sample of 100 patients from each of the four groups listed in Table 2 was audited. Overall, the DET correctly identified 161/163 patients (98.8% specificity) as never having had a Pap smear. The DET only detected the most recent Pap smear in 161/237 patients (67.9% sensitivity). The DET picked up an earlier smear in 66/237 (27.8%) of patients.

It was found that 7% of those patients who were recorded as never having had a Pap smear by the DET had actually had a Pap smear (Table 3). The DET either missed the Pap smear altogether or there was a phone/specialist record of the Pap smear having been conducted elsewhere. Of those patients who had not had a Pap smear in ≥ 4 years, 20% had actually had a Pap smear. The DET failed to pick up the most recent Pap smear in 18% of patients. In those patients who had had a Pap smear in the last 2–4 years, this failure to pick up the most recent Pap smear increased to 42%.

If the proportion of patients indicated in the random sample (42% of patients being wrongly recorded as having had a Pap smear between two and four years ago, when in fact, they should be included in the Pap smear within the last two years category) is applied to the original population of 302 patients having had a Pap smear within the last 2–4 years, this suggests that 127 additional patients should be included in the within the last two years category. This brings the updated Pap smear rate within two years in Practice A to 18.1%, which is significantly higher than the rate indicated by the DET (X2=23.24, p<0.001).

Those patients who had Pap smears that were coded differently (Table 1) between time intervals (from different pathology providers or from within the same provider but using a different code) were less likely to have had their most recent Pap smear picked up by the DET (p <0.001). No trend could be observed in the DET’s preferences for picking up or failing to detect smears from specific pathology providers.

Table 2 Pap smear status of eligible female patients (aged 20–69 with no hysterectomy in past medical history) as determined by the DET

Table 3 Random sample of 100 patients per group identified by the DET in Practice A


DISCUSSION

The average biennial cervical cancer screening rate in NSW for the 2009–2010 reporting period is 56.5%,7 with reportedly lower participation rates in rural and remote communities.8 The local health district is reported to have a significantly lower biennial screening rate [54.0 with 95% CI (53.6–54.4)] than the state average.

It was anticipated that the use of a DET would allow for rapid assessment of the cervical cancer screening rates in the practices and accurately reflect the participation rate in cervical screening. Such information would be useful in the effort to increase the rate of women following the current recommendation to undergo screening biennially. However, the DET failed to report accurate results, significantly underestimating the true level of cervical screening (X2=23.24, p=0.001).

One problem associated with using DETs may be the inability to archive information that is no longer relevant, for example woman who have left the practice. Essentially, data need to be accurate and complete, in addition to being entered in the correct field within the clinical software9 and in a format useable by the DET.

Of those patients identified by the DET as not having had a Pap smear, seven of 100 random patients had actually had a Pap smear. Three patients had phone/specialist records of Pap smears conducted elsewhere, which may account for the failure of the DET to pick up these results. Although not directly a problem of the DET, this does limit its usefulness in clinical practice, highlighting the importance of complete/accurate clinical data. The audit failed to identify a reason for why the other four patients were not identified.

The tool extracts data from two sources. There is a ­manual entry field called ‘Last Pap Date’, in addition to atomized pathology data that are extracted. The atomized pathology data are limited to test names that have been entered into the program. Pap smear results do not auto-populate in many clinical software programs and manual entry is time consuming and can be open to error.10 If these data are not entered manually, Pap smear rates determined via DETs will be underestimated.10

This appears to be a more significant problem with Pap smear data than other pathology tests, and accuracy of general practice records in terms of Pap smears has been previously questioned. Laurence et al.11 found that the cervical screening rate determined using only immediately available electronic medical records (EMRs) indicated a low screening rate in participating practices (45%). However, telephone follow-up and adjustments to the denominator indicated that the rate was 86%. In the current study, the DET identified that 13.3% of women in the practice had had a Pap smear within the previous two years; however, available Practice Incentives Program data stated that 63.5% of whole patient equivalents at this practice had claimed the Medicare item number for a Pap smear during the same reporting period. This is 50% points greater than the rate reported by the DET, highlighting the current limitation of the tool.

Schattner et al.12 suggest that there is a need to improve eHealth data transmission to increase accessibility of clinical data by DETs for Pap smear results, in addition to improving the functionality of DETs themselves.

Increasingly, DETs are being used for research and audit of general practice. A role for DETs has been suggested for gathering such diverse information from electronic patient databases as patient demographics, disease and risk factor profiles, immunisation rates and cancer surveillance.13 However, examples of their use often make no comment about the accuracy of the tools in terms of extracting clinical data despite the recognised limitations of using DETs.1316


CONCLUSION

The results from this study highlight the deficiencies in current electronic recording of Pap smears within general practice. DETs should be used with caution as they may underestimate the rate of Pap smears from EMRs.

Acknowledgements

The authors would like to thank the practice staff who supported this project and CoastCityCountry General Practice Training for funding this research.


REFERENCES

1. Pearce C and Shearer MEA. A division’s worth of data Australian Family Physician 2011;40:167–70. PMid:21597524.

2. Liaw ST and Tomlins R. Developments in information systems. In: General Practice in Australia, edition 1. Canberra: Australian government Department of Health and Ageing, 2004:544–86.

3. Bourke A and Dattani HEA. Feasibility study and methodology to create a quality-evaluated database of primary care data. Informatics in Primary Care 2004;12:171–7. http://dx.doi.org/10.14236/jhi.v12i3.124.

4. Morrell S, Perez DA, Hardy M, Cotter T and Bishop JF. Outcomes from a mass media campaign to promote cervical screening in NSW, Australia. J Epidemiol Community Health 2010;64(9):777–83. http://dx.doi.org/10.1136/jech.2008.084657. PMid:19822553.

5. Department Of Health And Ageing and Australian Institute of Health and Welfare. Cervical Screening in Australia 2005-2006. Canberra: AIHW, 2008.

6. Hancock L and Sanson-Fisher REA. Cervical cancer screening in rural New South Wales Health Insurance Commission data compared to self report. Australian and New Zealand Journal of Public Health 1998; 22:307–12. http://dx.doi.org/10.1111/j.1467-842X.1998.tb01382.x. PMid:9629814.

7. Cancer Institute NSW. Cervical Cancer Screening in New South Wales. Annual Statistical Report 2009–2010, 2013. Available from: https://www.cancerinstitute.org.au/getattachment/fb808844-079e-454b-84b2-d65d935cbc55/cervical-screeningreport-13.pdf. Accessed 3 April 2014.

8. Australian Institute of Health and Welfare. Cervical Screening in Australia 2007–2008: Data Report. Cancer series no. 54. Cat. no. CAN 50. Canberra: AIHW, 2010.

9. de Lusignan S, Valentin T, Chan T, Hague N, Wood O, van Vlymen J et al. Problems with primary care data quality: osteoporosis as an exemplar. Inform Prim Care 2004;12(3):147–56. http://dx.doi.org/10.14236/jhi.v12i3.120. PMid:15606987.

10. Stout R. Cervical Screening Mini Collaborative (CSMC) – Pap Final Report, 2009. Available from: www.gpgc.com.au/getfilelibfile.aspx?fk=996. Accessed 18 Jan 2016.

11. Laurence C, Burgess T, Beilby J, Symon B and Wilkinson D. Electronic medical records may be inadequate for improving population health status through general practice: Cervical smears as a case study. Australian and New Zealand Journal of Public Health 2004;28:317–9. http://dx.doi.org/10.1111/j.1467-842X.2004.tb00436.x. PMid:15704693.

12. Schattner P, Saunders M, Stanger L, Speak M and Russo K. Clinical data extraction and feedback in general practice: a case study from Australian primary care. Inform Prim care 2010;18(3):205–12. http://dx.doi.org/10.14236/jhi.v18i3.773.

13. Peiris D, Agaliotis M, Patel B, Patel A, Taylor R and Mamoon HEA. Validation of a general practice audit and data extraction tool. Australian Family Physician 2013;42(11):816–9. PMid:24217106.

14. Parker D. An Audit of osteoporotic patients in an Australian general practice. Aust Fam Physician 2013;42:423–7. PMid:23781552.

15. Liaw ST, Taggart J, Yu H and de Lusignan S. Data extraction from electronic health records - existing tools may be unreliable and potentially unsafe. Aust Fam Physician 2013;42(11):820–3. PMid:24217107.

16. van Vlymen J, de Lusignan S, Hague N, Chan T and Dzregah B. Ensuring the quality of aggregated general practice data: lessons from the Primary Care Data Quality Programme (PCDQ). In: Engelbrecht R et al. Connecting Medical Informatics and Bio-Informatics. European Federation of Medical Informatics. 2005;116:1010–5. PMid:16160391.

Refbacks

  • There are currently no refbacks.


This is an open access journal, which means that all content is freely available without charge to the user or their institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles in this journal starting from Volume 21 without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of open accessFor permission regarding papers published in previous volumes, please contact us.

Privacy statement: The names and email addresses entered in this journal site will be used exclusively for the stated purposes of this journal and will not be made available for any other purpose or to any other party.

Online ISSN 2058-4563 - Print ISSN 2058-4555. Published by BCS, The Chartered Institute for IT