Refereed journal article or data article (A1)

A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output




List of AuthorsMaarit Koponen, Leena Salmi, Markku Nikulin

PublisherSpringer Netherlands

Publication year2019

JournalMachine Translation

Journal name in sourceMachine Translation

Volume number33

Issue number1-2

Start page61

End page90

Number of pages30

ISSN0922-6567

DOIhttp://dx.doi.org/10.1007/s10590-019-09228-7

Self-archived copy’s web addresshttps://research.utu.fi/converis/portal/detail/Publication/40147635


Abstract

This paper presents a comparison of post-editing (PE) changes performed on English-to-Finnish neural (NMT), rule-based (RBMT) and statistical machine translation (SMT) output, combining a product-based and a process-based approach. A total of 33 translation students acted as participants in a PE experiment providing both post-edited texts and edit process data. Our product-based analysis of the post-edited texts shows statistically significant differences in the distribution of edit types between machine translation systems. Deletions were the most common edit type for the RBMT, insertions for the SMT, and word form changes as well as word substitutions for the NMT system. The results also show significant differences in the correctness and necessity of the edits, particularly in the form of a large number of unnecessary edits in the RBMT output. Problems related to certain verb forms and ambiguity were observed for NMT and SMT, while RBMT was more likely to handle them correctly. Process-based comparison of effort indicators shows a slight increase of keystrokes per word for NMT output, and a slight decrease in average pause length for NMT compared to RBMT and SMT in specific text blocks. A statistically significant difference was observed in the number of visits per sub-segment, which is lower for NMT than for RBMT and SMT. The results suggest that although different types of edits were needed to outputs from NMT, RBMT and SMT systems, the difference is not necessarily reflected in process-based effort indicators.


Downloadable publication

This is an electronic reprint of the original article.
This reprint may differ from the original in pagination and typographic detail. Please cite the original version.




Last updated on 2022-07-04 at 16:05