Natural Language Processing for Ehr-based Pharmacovigilance a Structured Review
Original Paper
Abstruse
Background: The exploration of clinically relevant information in the free text of electronic health records (EHRs) holds the potential to positively impact clinical practice as well as knowledge regarding Crohn disease (CD), an inflammatory bowel disease that may touch on whatsoever segment of the alimentary canal. The EHRead technology, a clinical natural linguistic communication processing (cNLP) system, was designed to observe and extract clinical information from narratives in the clinical notes contained in EHRs.
Objective: The aim of this study is to validate the performance of the EHRead technology in identifying information of patients with CD.
Methods: We used the EHRead technology to explore and excerpt CD-related clinical information from EHRs. To validate this tool, we compared the output of the EHRead technology with a manually curated gilded standard to assess the quality of our cNLP system in detecting records containing any reference to CD and its related variables.
Results: The validation metrics for the principal variable (CD) were a precision of 0.88, a recall of 0.98, and an F1 score of 0.93. Regarding the secondary variables, we obtained a precision of 0.91, a think of 0.71, and an F1 score of 0.fourscore for CD flare, while for the variable vedolizumab (treatment), a precision, retrieve, and F1 score of 0.86, 0.94, and 0.90 were obtained, respectively.
Conclusions: This evaluation demonstrates the ability of the EHRead technology to identify patients with CD and their related variables from the gratuitous text of EHRs. To the all-time of our knowledge, this study is the beginning to use a cNLP organisation for the identification of CD in EHRs written in Spanish.
doi:10.2196/30345
Keywords
Introduction
Crohn affliction (CD) is a chronic inflammatory bowel illness (IBD) that leads to lesions in different sites along the length of the gastrointestinal tract and, occasionally, in other extraintestinal locations such equally peel, eyes, joints, mouth, and the hepatobiliary system []. Symptoms (including abdominal pain, diarrhea, fever, and weight loss) evolve in a relapsing and remitting manner, leading to bowel impairment and disability. CD is considered to be a heterogeneous disorder with a multifactorial etiology, in which genetics and environmental factors interact to manifest the affliction []. Although nigh patients with CD are diagnosed with an inflammatory phenotype, almost half of them do require surgeries derived from complications such as strictures, fistulas, or abscesses [].
Over the last years, most health care institutions accept moved away from paper clinical records toward electronic health records (EHRs) in which patients' longitudinal medical information is stored []. Since then, large volumes of digitalized real-globe clinical data have been generated at exponential rates. Although some clinical information independent in the EHRs are stored in structured fields, the majority of the relevant clinical information appears embedded in the free-text narratives written downward past health professionals [].
The surface area of computer scientific discipline dedicated to the analysis and representation of naturally occurring texts (written or oral) [] is called natural language processing (NLP). I of the applications of NLP focuses on the extraction of information from costless text captured in EHRs and is therefore referred to as clinical NLP (cNLP). And then far, cNLP systems have been successfully practical for the extraction of relevant clinical data using approaches such as regular expressions or machine learning. As a outcome, the quantity and quality of data captured from the EHRs have essentially increased over recent years []. Although incorporating data from free text into case detection through NLP techniques improves enquiry quality [-], 1 key claiming in this process is to ensure the validity of the results by assessing the detection performance.
In this context, as part of the PREMONITION-CD observational study, we aimed to assess the performance of the cNLP system EHRead applied science [-] in identifying medical records that comprise mentions of CD and its related variables when compared to the detection performed past proficient medical doctors. Because the manual review of free-text narratives is extremely time-consuming, valuable information routinely collected in clinical practice has largely remained unused for research purposes. Therefore, the validated automatic extraction of this information holds potential to accelerate our knowledge well-nigh CD and could have a positive impact in the management of these patients [,].
Methods
Ideals Approval and Consent to Participate
This study was conducted within the scope of the PREMONITION-CD project, a multicenter, retrospective study aimed at using NLP to observe gratuitous-text data in CD patients' EHRs. Before the showtime of data drove, the study was approved by the Spanish Ethics Committee, Agencia Española de Medicamentos y Productos Sanitarios, and the Madrid region Ideals Committee, Comité Ético de Investigación con Medicamentos Regional de la Comunidad de Madrid, with reference number IBD-5002 (May 2018). Approval from each of the hospitals participating in the study was also obtained. It was registered in ClinicalTrials.gov with the identifier number NCT03668249.
The study was conducted in compliance with legal and regulatory requirements and followed by and large accepted research practices described in the ICH Guideline for Proficient Clinical Practice, the Declaration of Helsinki in its latest edition, Practiced Pharmacoepidemiology Practices, and applicable local regulations.
Consent for Publication
In accordance with article fourteen.5 of the General Data Protection Regulation (GDPR), if obtaining consent is incommunicable or would involve a disproportionate endeavor, in detail for processing for archiving purposes in the public involvement, scientific or historical research purposes, or statistical purposes, the study is subject field to the conditions and safeguards referred to in Article 89.
Regarding Article 89 of the GDPR, processing in the public involvement or scientific research purposes shall be subject to appropriate safeguards and volition not require consent from each of the information subjects, in accordance with the GDPR, for the rights and freedoms of the data subject.
Availability of Data and Materials
Due to the retrospective nature of the research, information analysis did not require consent from the information subjects. Therefore, supporting data is subject to strict confidentiality agreements with each participating hospital and cannot be made openly available.
Data Source
Data were collected from viii hospitals of the Spanish National Healthcare Network from Jan 1, 2014, to Dec 31, 2018 (except for one participating site with electronic data available from 2013 to 2017).
Report Pattern
For this study, the assessed variables were CD, CD flare (a crucial variable for the characterization of the evolution of the disease), and vedolizumab (a biologic drug indicated exclusively for the treatment of IBD). The variables included in this study were selected by the senior study committee based on the PREMONITION-CD overall study objectives. The variables were detected when written directly in the EHRs, without inferences or prior consequence definitions. The man annotations served the purpose of the creation of a gold standard to which the EHRead technology was compared.
The EHRead engineering is an NLP arrangement designed to retrieve large amounts of biomedical information contained in EHRs [-] and convert the information into a structured representation ().
To perform this written report, we completed the following steps: EHR collection, processing using EHRead technology, cosmos of the aureate standard information set, and comparison of both outputs using standard measures of functioning ().
In the EHR collection stride, a data ready was selected that consisted of a sample collection of EHRs obtained primarily from the gastroenterology service (including consultation, hospitalization, and emergency reports), representing more than iii,900,000 patients. To obtain a representative data ready, 100 records were randomly selected from each of the 8 sites containing EHRs with and without CD-related data, amounting to a total of 800 clinical documents from 800 patients. After, all records were fully anonymized to come across legal and ethical requirements earlier they were annotated by physicians (annotators) to generate a gold standard for each participating site (see sections about annotation process and gold standard).
In parallel to the note job carried out past physicians, the EHRead applied science was applied on the free text of the aforementioned EHRs used to generate the gold standard (for more details see NLP System). By doing and then, the performance of the EHRead engineering could directly exist compared to man operation in detection of CD and secondary variables.
In the last stride of the evaluation, the performance of the EHRead technology was compared confronting the gilded standard to validate the capacity of the engineering in identifying records containing mentions of CD and its related variables. Therefore, both the detections of physicians and the EHRead applied science were transformed into binaries (0 no detection, 1 detection) for each variable to calculate the functioning metrics precision, recall, and F1 score using the library scikit-learn [].
NLP System
The principal phases of the NLP system were the post-obit:
- The section identification stage aims to detect the different parts of a clinical document, such as family medical history, physical exam, and treatment.
- The concept identification phase is when the organisation detects a medical concept. Specifically, the terminology considered by the EHRead engineering science is congenital upon SNOMED-CT (Systemized Nomenclature of Medicine–Clinical Terms), a leading platform of systematically organized and computer-readable collections of medical concepts. SNOMED-CT includes codes, concepts, synonyms, and definitions used in clinical documentation and is considered the nigh comprehensive terminology worldwide.
- The contextual data phase focuses on detecting the attributes of the already identified clinical terms within their textual context, both from an intention perspective (the term is either stated in an affirmative mode or negated, or is part of a theorize or stance) and from a temporal perspective (current or historical).
Annotation Process and Gold Standard
The manual revision of clinical texts was carried out by annotators specialized in gastroenterology. For the annotation chore, guidelines were jointly created by internal NLP experts and clinical experts. They included the variables to be annotated in the free text, along with recommendations on how to solve uncertainties. Following these guidelines, specialists reviewed the costless text of selected EHRs for the occurrence of the study variables to answer a prepare of yes/no questions: Does/did the patient have CD? Does the report country that the patient has had a flare? and Does the tape state that the patient was treated with Vedolizumab? The 2nd and tertiary questions were but asked if the first i was affirmative, meaning that the patient did have CD before or at the time point of the hospital visit. The annotators were non allowed to reply with yes to whatsoever of the questions based on inference.
Of the 100 records selected per site, xv were reviewed by two contained annotators to assess the interannotator understanding [,]. A low understanding indicates that the annotators had difficulties in linguistically identifying the relevant variables in the EHRs or that the guidelines are nevertheless inadequate in properly describing the notation task []. Thus, the interannotator agreement serves every bit a control mechanism to cheque the reliability of the annotation and further to constitute a target of functioning for the NLP system. For this task, the annotators were not allowed to communicate with each other or share information regarding the annotation process to avert bias. One time the annotations were finished, the interannotator agreement was calculated in terms of F1 score. Once the quality of annotations had been verified through the interannotator agreement and the disagreements had been resolved to build the final gilt standard, one of the two physicians annotated the remaining 85% of clinical records to consummate the gold standard.
Statistical Analysis
The operation of the EHRead technology in identifying CD and its related variables was compared with the gilt standard. The understanding between them was calculated using three metrics: precision (ie, positive predictive value), recall (ie, sensitivity), and their harmonic hateful F1 score []. Precision is the indicator of the accuracy of information retrieved by the organisation, recall is the indicator of the amount of information the system retrieves, and F1 score conveys the balance between precision and call back. In addition to those metrics, we calculated the 95% CI for each aforementioned measure out, since this provides information nigh the range in which the true value lies and thus how robust the metric is. The method used to calculate the 95% CIs is the Clopper-Pearson approach, one of the nigh mutual methods for computing binomial 95% CIs.
Results
The gilded standard data fix (N=800) consisted of 41.4% (north=331) medical records with CD, 21.3% (northward=170) with CD flare, and 10% (north=83) with vedolizumab handling. shows the interannotator understanding F1 scores of the gold standard for each investigated variable per site.
The interannotator agreement values were higher than 0.viii for all comparisons, indicating an almost perfect agreement according to the Landis and Koch scale []. In add-on, the overall agreement between all sites was nearly perfect [] for the three studied variables. The EHRead applied science results in terms of precision, retrieve, and F1 score are shown in .
The detection of the main variable (ie, CD) achieved a precision of 0.88, a call up of 0.98, and an F1 score of 0.93. Regarding the secondary variables, CD flare obtained a precision of 0.91, a recall of 0.71, and an F1 score of 0.80, while the variable vedolizumab was detected at a precision of 0.86, a recollect of 0.94, and an F1 score of 0.90.
| | F1 score | ||
| | Crohn illness | Crohn disease flare | Vedolizumab |
| Site 1 | 0.93 | 0.86 | 1.00 |
| Site 2 | ane.00 | 0.87 | 1.00 |
| Site iii | i.00 | i.00 | i.00 |
| Site four | 0.93 | 1.00 | 1.00 |
| Site v | 0.93 | 0.83 | 1.00 |
| Site 6 | 0.93 | i.00 | 1.00 |
| Site 7 | one.00 | 1.00 | i.00 |
| Site 8 | 1.00 | 0.85 | 1.00 |
| Average | 0.97 | 0.93 | 1.00 |
| Variable | Precision (95% CI) | Recall (95% CI) | F1 score (95% CI) |
| Crohn affliction | 0.88 (0.85-0.91) | 0.98 (0.95-0.99) | 0.93 (0.xc-0.95) |
| Crohn disease flare | 0.91 (0.85-0.95) | 0.71 (0.63-0.77) | 0.lxxx (0.72-0.85) |
| Vedolizumab | 0.86 (0.76-0.93) | 0.94 (0.86-0.98) | 0.90 (0.81-0.96) |
Discussion
The evaluation presented here is office of the observational, retrospective PREMONITION-CD written report, designed to characterize clinical and nonclinical variables of patients with CD. To the best of our noesis, this is the first multicentric written report using a cNLP system for the identification of prespecified CD-related variables from reports written in Spanish. The intrinsic characteristics of IBD and the current dilemmas associated with the medical management of affected patients present an opportunity for the implementation of large data research strategies. Artificial intelligence techniques complement current research efforts and might be central in disentangling the complication of the illness [] by assuasive central patient-centered information to be retrieved and analyzed at a larger population scale. In turn, big CD/IBD data sets volition enable the identification of clinical patterns, patient management, and predictors of disease that will ultimately improve patient intendance.
Although some clinical data is stored in structured fields of EHRs, the majority is contained in the narrative complimentary text []. The automated extraction of these data using modern NLP techniques has a strikingly positive touch on clinical exercise, since it enables the exploration of this valuable patient information at a calibration that was not possible earlier. Here, we evaluated Savana's EHRead technology, a cNLP organization designed to discover and extract clinically relevant data from the free text of EHRs [-], to place CD reports from narrative clinical data.
In contrast to other enquiry studies that applied NLP techniques on Castilian EHRs obtained from a single medical center [,], this report combined data from 8 large hospitals, thereby providing robustness and enabling generalizability. The capabilities of the EHRead technology allowed us to procedure a broad range of document types and to handle the dissimilar internal structures of clinical reports from the different participating sites. In addition, the inclusion of different sites enhanced the variability and richness of the language regarding the evaluated variables. Indeed, the variables evaluated were expressed in different ways across sites, including discrepancies in abbreviations or acronyms.
F1 scores college than 0.80 for all interannotator agreements ensure that the gold standard met the criteria to serve as reference. In addition, our study demonstrates a practiced performance of the EHRead engineering science in identifying reports that contain mentions of CD and CD-related variables. We obtained F1 scores higher than 90% for the principal variable and close to eighty% for the remaining variables (). Despite the intrinsic heterogeneity of EHRs resulting from a variability in physicians, information drove sites, and record completeness, EHRead was successful at pinpointing important information, equally reflected by these assessment parameters. Indeed, precision and recall were balanced for about of the variables, showing that the EHRead technology is non but accurate when detecting the evaluated variables merely as well in terms of retrieving a large amount of information.
Although this study deals with EHRs in Castilian, almost previous cNLP systems focused on data extraction from clinical reports in English language []. F1 scores of cNLP systems that target EHRs in English range from 0.71 to 0.92 [-]. Available dominion-based [,] or machine learning–oriented [] systems that identify medical entities in Spanish have reached F1 scores between 0.70 and 0.ninety. Even so, the cNLP systems targeting the Spanish language are still limited. A direct comparison between the EHRead technology and these country-of-the-art approaches is complicated due to differences in gold standard creation and utilise of linguistic communication. Still, the overall functioning of the EHRead technology beyond the viii participating sites with the achieved F1 scores demonstrates that the functioning is comparable to other state-of-the-art NLP systems bachelor in the clinical domain. Furthermore, compared to previous works that notice CD-related variables in English using NLP to increase or correctly allocate the number of patients with CD detected through the standard International Classification of Diseases-ix coding system [,], our study relies on a purely NLP-dependent detection approach. Having performed our report in Spanish is an added value, since information technology is a linguistic communication in which NLP has not been previously applied in CD studies, yet yielding robust results compared to previous approaches in English language.
A robust linguistic validation of the EHRead technology sets it forth equally a valuable methodology for future studies regarding IBD and CD. The expanding use of EHRs and the wealth of data contained within their costless text represent a unique source of data that benefits from the development of cNLP systems. Indeed, cNLP systems are dynamic and evolve with novel technologies that ameliorate concept identification []. This approach is suitable to improve detect clinical information of patients with IBD and CD in a real-world setting, which can provide insight to improve the medical direction of these patients.
In decision, this study presents an evaluation of the EHRead engineering, an NLP system for the extraction of clinical information from the narrative gratis text contained in EHRs. This evaluation conspicuously demonstrates the ability of the EHRead technology to identify mentions of CD and two related variables. Although further research is needed, the utilise of the EHRead technology facilitates the automatic big-scale analysis of CD, thus contributing to the improvement of clinical practice by generating real-world evidence. Robust data extraction and precise variable detection are key to support future studies using large data sets of patients with CD.
Acknowledgments
We would similar to give thanks Tamara Pozo, Marta Mengual, and Ana Sánchez Gabriel for their kind support during the report, and Stephanie Marchesseau for valuable comments on a previous version of this manuscript. Nosotros are grateful to Laura Yebes, Carlos Del Rio-Bermudez, Ana Lopez-Ballesteros, and Clara 50 Oeste for their assistance in writing and editing the manuscript, and the construction of figures funded by Takeda.
The PREMONITION-CD Study Group includes the following investigators: Carlos Castaño from Hospital Universitario (HU) Rey Juan Carlos, Madrid, Kingdom of spain; Ángel Ponferrada Díaz from HU Infanta Leonor, Madrid, Spain; María Chaparro and María José Casanova from HU de La Princesa, Madrid, Spain; Felipe Ramos Zabala from HM Hospitales, Madrid, Spain; Almudena Calvache from HU Infanta Elena, Madrid, Spain; Fernando Bermejo from HU de Fuenlabrada, Madrid, Spain; Noemí Manceñido from HU Infanta Sofía, Madrid, Spain; and Marta Calvo Moya from HU Puerta de Hierro, Majadahonda, Madrid, Spain.
This report was funded by Takeda Farmacéutica España S.A. The analyses conducted by Medsavana SL too as the participation of the Medsavana authors in the evolution of this manuscript were funded by Takeda Farmacéutica España S.A.
Authors' Contributions
All authors have fabricated substantial contributions to the formulation and pattern of the study, and conquering, analysis, and estimation of data, in addition to drafting and revising the manuscript.
Conflicts of Interest
JPG has served as a speaker, a consultant, and informational member for, or has received enquiry funding from, MSD, Abbvie, Hospira, Pfizer, Kern Pharma, Biogen, Takeda, Janssen, Roche, Sandoz, Celgene, Ferring, Faes Farma, Shire Pharmaceuticals, Dr. Falk Pharma, Tillotts Pharma, Chiesi, Casen Fleet, Gebro Pharma, Otsuka Pharmaceutical, and Vifor Pharma. IG has served as a speaker, a consultant, and advisory member for, or has received research funding from, Kern Pharma, Takeda, and Janssen. RP has served as a speaker for Takeda and Janssen. MIVM has served as a speaker, consultant, and informational member for, or has received funding from, MSD, Abbvie, Pfizer, Ferring, Shire Pharmaceuticals, Takeda, and Jannsen. FG has received educational grants from Janssen, MSD, Takeda, and Abbvie, and nonpersonal investigation grants from MSD, Janssen, Abbvie, Takeda, and Tilllots. CM, JA, VM, IT, and AFN are employees at Takeda Farmacéutica España Due south.A. LC is an ex-employee and SM is currently employed at Medsavana SL, which received funding from Takeda Farmacéutica España S.A. The remaining authors take no conflicts of interest to declare.
Abbreviations
| CD: Crohn affliction |
| cNLP: clinical natural language processing |
| EHR: electronic health record |
| GDPR: General Data Protection Regulation |
| IBD: inflammatory bowel disease |
| NLP: natural language processing |
| SNOMED-CT: Systemized Nomenclature of Medicine–Clinical Terms |
Edited past C Lovis; submitted 11.05.21; peer-reviewed past D Shung, Y Chen, FJ Sánchez-Laguna; comments to author 29.05.21; revised version received 22.07.21; accepted 02.01.22; published 18.02.22
Copyright©Carmen Montoto, Javier P Gisbert, Iván Guerra, Rocío Plaza, Ramón Pajares Villarroya, Luis Moreno Almazán, María Del Carmen López Martín, Mercedes Domínguez Antonaya, Isabel Vera Mendoza, Jesús Aparicio, Vicente Martínez, Ignacio Tagarro, Alonso Fernandez-Nistal, Lea Canales, Sebastian Menke, Fernando Gomollón, PREMONITION-CD Report Group. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 18.02.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/iv.0/), which permits unrestricted use, distribution, and reproduction in whatsoever medium, provided the original piece of work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
mitchellpultooper1981.blogspot.com
Source: https://medinform.jmir.org/2022/2/e30345
0 Response to "Natural Language Processing for Ehr-based Pharmacovigilance a Structured Review"
Postar um comentário