The rapid technologisation of translation has influenced the translation industry's direction towards machine translation, post-editing, subtitling services and video content translation. Besides, the pandemic situation associated with COVID-19 has rapidly increased the transfer of business and education to the virtual world. This situation has motivated us not only to look for new approaches to online translator training, which requires a different method than learning foreign languages but in particular to look for new approaches to assess translator performance within online educational environments. Translation quality assessment is a key task, as the concept of quality is closely linked to the concept of optimization. Automatic metrics are very good indicators of quality, but they do not provide sufficient and detailed linguistic information about translations or post-edited machine translations. However, using their residuals, we can identify the segments with the largest distances between the post-edited machine translations and machine translations, which allow us to focus on a more detailed textual analysis of suspicious segments. We introduce a unique online teaching and learning system, which is specifically "tailored" for online translators' training and subsequently we focus on a new approach to assess translators' competences using evaluation techniques-the metrics of automatic evaluation and their residuals. We show that the residuals of the metrics of accuracy (BLEU_n) and error rate (PER, WER, TER, CDER, and HTER) for machine translation post-editing are valid for translator assessment. Using the residuals of the metrics of accuracy and error rate, we can identify errors in post-editing (critical, major, and minor) and subsequently utilize them in more detailed linguistic analysis.

Department of Computer Science Constantine the Philosopher University in Nitra Nitra Slovakia

Department of Translation Studies Constantine the Philosopher University in Nitra Nitra Slovakia

Science and Research Centre University of Pardubice Pardubice Czech Republic

See more in PubMed

Al Abdullatif HA, Velazquez-Iturbide JA. Who will continue using MOOCs in the future? Personality traits perspective. IEEE Access. 2020;8:52841–52851. doi: 10.1109/ACCESS.2020.2979180. DOI

Benkova L, Munkova D, Benko Ľ, Munk M. Evaluation of english-slovak neural and statistical machine translation. Applied Sciences. 2021;7(2948):2948. doi: 10.3390/app11072948. DOI

Bollen KA, Arminger G. Observational residuals in factor analysis and structural equation models. Sociological Methodology. 1991;21:235–265. doi: 10.2307/270937. DOI

Da’u A, Salim N. Aspect extraction on user textual reviews using multi-channel convolutional neural network. PeerJ Computer Science. 2019;5(6):e191. doi: 10.7717/peerj-cs.191. PubMed DOI PMC

Doherty S. The impact of translation technologies on the process and product of translation. International Journal of Communication. 2016;10:947–969.

Doherty S. Issues in human and automatic translation quality assessment. Human Issues in Translation Technology; 2017. pp. 149–166.

Drlik M, Munk M. Understanding time-based trends in stakeholders’ choice of learning activity type using predictive models. IEEE Access. 2019;7:3106–3121. doi: 10.1109/ACCESS.2018.2887057. DOI

Drummond-Butt S. 31 new video marketing statistics to fuel your strategy in 2020. 2019. https://www.impactplus.com/blog/new-video-marketing-statistics. [10 November 2020]. https://www.impactplus.com/blog/new-video-marketing-statistics

Esfandiari MR, Shokrpour N, Rahimi F. An evaluation of the EMT: compatibility with the professional translator’s needs. Cogent Arts & Humanities. 2019;6(1):1601055. doi: 10.1080/23311983.2019.1601055. DOI

Gavrilenko N. Online model for teaching and learning the specialized translation. Eurasia Journal of Mathematics, Science and Technology Education. 2018;14(6):2711–2717. doi: 10.29333/ejmste/85421. DOI

Google Google translate API. 2016. https://cloud.google.com/translate/ [3 February 2016]. https://cloud.google.com/translate/

Gorozhanov AI, Kosichenko EF, Guseynova IA. Teaching written translation online: theoretical model, software development, interim results. SHS Web of Conferences. 2018;50(1):01062. doi: 10.1051/shsconf/20185001062. DOI

Hu K, O’Brien S, Kenny D. A reception study of machine translated subtitles for MOOCs. Perspectives: Studies in Translation Theory and Practice. 2020;28(4):521–538. doi: 10.1080/0907676X.2019.1595069. DOI

Ismail S, Nasser Alsager H, Omar A. The implications of online translation courses on instructors’ philosophy of teaching. Arab World English Journal. 2019;5(5):176–189. doi: 10.24093/awej/call5.13. DOI

Kordoni V, van den Bosch A, Kermanidis KL, Sosoni V, Cholakov K, Hendrickx I, Huck M, Way A. Enhancing access to online education: quality machine translation of {MOOC} content. Proceedings of the Tenth International Conference on Language Resources and Evaluation ({LREC}’16); Portorož, Slovenia: European Language Resources Association (ELRA); 2016. pp. 16–22.

Leusch G, Ueffing N, Ney H. CDER: efficient MT evaluation using block movements. Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2006).2006.

Marczak M. Translation pedagogy in the digital age: how digital technologies have been altering translator education. Journal of Computer Virology and Hacking Techniques. 2018;7(7):20. doi: 10.4000/angles.895. DOI

McCue T. E learning climbing to $325 bllion by 2025 UF Canvas Absorb Schoology Moodle. Forbes. 2018. https://www.forbes.com/sites/tjmccue/2018/07/31/e-learning-climbing-to-325-billion-by-2025-uf-canvas-absorb-schoology-moodle/?sh=654ef9493b39#6da093353b39. [10 November 2020]. https://www.forbes.com/sites/tjmccue/2018/07/31/e-learning-climbing-to-325-billion-by-2025-uf-canvas-absorb-schoology-moodle/?sh=654ef9493b39#6da093353b39

Munk M, Munkova D. Detecting errors in machine translation using residuals and metrics of automatic evaluation. Journal of Intelligent & Fuzzy Systems. 2018;34(5):3211–3223. doi: 10.3233/JIFS-169504. DOI

Munk M, Munkova D, Benko L. Towards the use of entropy as a measure for the reliability of automatic MT evaluation metrics. Journal of Intelligent & Fuzzy Systems. 2018;34(5):3225–3233. doi: 10.3233/JIFS-169505. DOI

Munk M, Munková D, Benko Ľ. Identification of relevant and redundant automatic metrics for MT evaluation. Multi-disciplinary Trends in Artificial Intelligence (MIWAI 2016) Book Series: Lecture Notes in Computer Science; Cham: Springer International Publishing; 2016. pp. 141–152.

Munková D, Kapusta J, Drlík M. System for post-editing and automatic error classification of machine translation. DIVAI 2016 : 11th International Scientific Conference on Distance Learning in Applied Informatics, Sturovo, May 2–4, 2016; Sturovo: Wolters Kluwer; 2016. pp. 571–579.

Munková D, Munk M, Benko Ľ, Absolon J. From old fashioned one size fits all to tailor made online training. Advances in Intelligent Systems and Computing; Berlin: Springer Verlag; 2020. pp. 365–376.

Munková D, Vanko J, Absolon J, Bánik T, Glovňa J, Benko Ľ, Machová R, Munk M, Welnitzová K. Mtfytfliť sa je ľudské (ale aj strojové): analtfytfza chtfytfb strojového prekladu do slovenčiny. Nitra: UKF; 2017.

Murray S. Moocs struggle to lift rock-bottom completion rates | Financial Times. 2019. https://www.ft.com/content/60e90be2-1a77-11e9-b191-175523b59d1d. [10 November 2020]. https://www.ft.com/content/60e90be2-1a77-11e9-b191-175523b59d1d

Olohan M, Salama-Carr M. Translating science. Translator. 2011;17(2):179–188. doi: 10.1080/13556509.2011.10799485. DOI

Öner S. Integrating machine translation into translator training: towards ‘Human Translator Competence’? transLogos Translation Studies Journal. 2019;2(2/2):1–26. doi: 10.29228/transLogos.11. DOI

Robinson B, Lobo M, Gutiérrez-Artacho J. The professional approach to translator training revisited. 2017.

Sennrich R, Barone AVM, Moorkens J, Castilho S, Way A, Gaspari F, Kordoni V, Egg M, Popovic M, Georgakopoulou Y, Gialama M, Van Zaanen M. TraMOOC—translation for massive open online courses: recent developments in machine translation. 20th Annual Conference of the European Association for Machine Translation, EAMT 2017. European Association for Machine Translation; 2017. p. 27.

Shah D. By the numbers: MOOCs in 2018—class central. 2018. https://www.classcentral.com/report/mooc-stats-2018. [10 November 2020]. https://www.classcentral.com/report/mooc-stats-2018

Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J. A study of translation edit rate with targeted human annotation. Proceedings of Association for Machine Translation in the Americas; 2006. pp. 223–231.

Varga D, Németh L, Halácsy P, Kornai A, Trón V, Nagy V. Parallel corpora for medium density languages. Proceedings of the RANLP 2005; 2005. pp. 590–596.

Wei Z. The development prospect of English translation software based on artificial intelligence technology. Journal of Physics: Conference Series. Institute of Physics Publishing.2020.

Wei Z. A probe into the mixed teaching design of college English translation in the era of educational big data. Advances in Intelligent Systems and Computing; Berlin: Springer; 2021. pp. 863–868.

Wrede O. Odborntfytf preklad v kontexte dištančného vzdelávania. Nitra, Slovakia: UKF v Nitre; 2012a.

Wrede O. Methodisch-didaktische Neuerungen in der universitären Übersetzerausbildung—selbstgesteuertes Lernen durch Blended Learning (dargestellt am Beispiel der Übersetzung juristischer Texte in der Sprachkombination Deutsch-Slowakisch) In: Zybatow L, Petrova A, Ustaszewski M, editors. Translationswissenschaft interdisziplinär: Fragen der Theorie und der Didaktik. Frankfurt am Main, Germany: Peter Lang; 2012b. pp. 405–412.

Xu X. Development method of Japanese translation teaching assistant platform based on information technology. Advances in Intelligent Systems and Computing; Berlin: Springer; 2021. pp. 508–513.

Zaidan OF, Callison-Burch C. Crowdsourcing translation: professional quality from non-professionals. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies; Portland: Association for Computational Linguistics; 2011. pp. 1220–1229.

Comparison of various approaches to tagging for the inflectional Slovak language

The use of residual analysis to improve the error rate accuracy of machine translation