A1 Vertaisarvioitu alkuperäisartikkeli tieteellisessä lehdessä
Data and systems for medication-related text classification and concept normalization from Twitter: insights from the Social Media Mining for Health (SMM4H)-2017 shared task
Tekijät: Abeed Sarker, Maksim Belousov, Jasper Friedrichs, Kai Hakala, Svetlana Kiritchenko, Farrokh Mehryary, Sifei Han, Tung Tran, Anthony Rios, Ramakanth Kavuluru, Berry de Bruijn, Filip Ginter, Debanjan Mahata, Saif M. Mohammad, Goran Nenadic, Graciela Gonzalez-Hernandez
Kustantaja: OXFORD UNIV PRESS
Julkaisuvuosi: 2018
Journal: Journal of the American Medical Informatics Association
Tietokannassa oleva lehden nimi: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION
Lehden akronyymi: J AM MED INFORM ASSN
Vuosikerta: 25
Numero: 10
Aloitussivu: 1274
Lopetussivu: 1283
Sivujen määrä: 10
ISSN: 1067-5027
DOI: https://doi.org/10.1093/jamia/ocy114
Verkko-osoite: https://academic.oup.com/jamia/article/25/10/1274/5113021
Rinnakkaistallenteen osoite: https://research.utu.fi/converis/portal/detail/Publication/36971574
Objective: We executed the Social Media Mining for Health (SMM4H) 2017 shared tasks to enable the community-driven development and large-scale evaluation of automatic text processing methods for the classification and normalization of health-related text from social media. An additional objective was to publicly release manually annotated data.
Materials and Methods: We organized 3 independent subtasks: automatic classification of self-reports of 1) adverse drug reactions (ADRs) and 2) medication consumption, from medication-mentioning tweets, and 3) normalization of ADR expressions. Training data consisted of 15 717 annotated tweets for (1), 10 260 for (2), and 6650 ADR phrases and identifiers for (3); and exhibited typical properties of social-media-based health-related texts. Systems were evaluated using 9961, 7513, and 2500 instances for the 3 subtasks, respectively. We evaluated performances of classes of methods and ensembles of system combinations following the shared tasks.
Results: Among 55 system runs, the best system scores for the 3 subtasks were 0.435 (ADR class F1-score) for subtask-1, 0.693 (micro-averaged F1-score over two classes) for subtask-2, and 88.5% (accuracy) for subtask-3. Ensembles of system combinations obtained best scores of 0.476, 0.702, and 88.7%, outperforming individual systems.
Discussion: Among individual systems, support vector machines and convolutional neural networks showed high performance. Performance gains achieved by ensembles of system combinations suggest that such strategies may be suitable for operational systems relying on difficult text classification tasks (eg, subtask-1).Conclusions: Data imbalance and lack of context remain challenges for natural language processing of social media text. Annotated data from the shared task have been made available as reference standards for future studies (http://dx.doi.org/10.17632/rxwfb3tysd.1).
Ladattava julkaisu This is an electronic reprint of the original article. |  |
