Predicting the quality of a translation from attributes of a translator

The evaluation of the quality of a translation is still a time consuming manual operation depending on human intervention. Confidence Estimation (CE) explores the prediction of the quality of a translation from the attributes of a sentence (Blatz et al. 2004; Specia et al. 2009; Specia and Shah 2014). Cognitive translatology searches for a proficient translator from the dispositions of a person (O'Brien 2013; Jääskeläinen 2010; Muñoz Martín 2010a; House 2014). This study explored the prediction of the quality of translation between ‘Good’ and ‘Bad’ from certain attributes of a translator using a J48 decision tree algorithm with 10-fold cross validation on the WEKA machine learning platform. The data preparation for all J48 experiments (source text design, translation evaluation metrics, the selection of attributes, and the selection of candidates) were examined in a pilot study with 22 participants. The main study consisted of 82 participants with three groups of translators: professional translator, casual translator, and student. Each participant was described by the values of 146 attributes from six categories (arts, sports, pastime activities, life style, personality, and background). The evaluation of 25 sentences from two topics by four native English speakers was the basis of their quality scores. Krippendorff’s Alpha (Hayes and Krippendorff 2007) measured 0.7342 for Nugget Recall and 0.6079 for fluency suggesting acceptable agreement among the four evaluators after each evaluating 2,100 sentences. Six training experiments were carried out. Experiments 1-3 varied the quality threshold distinguishing Good from Bad. Experiment 4 compared manual and automatic feature selection methods. Experiment 5 excluded student translators, while Experiment 6 was restricted to professional translators. The highest F-Measure was 0.775 when the participants were limited to 17 ‘Good’ and 14 ‘Bad’ professional translators. It measured 0.774 when casual translators were included as participants. The study concluded that the prediction was possible with a two-tiered approach: one for casual and professional translators and the other for professional translators only. A Japanese language qualification, the length of Japanese language use, interest in going to opera, playing Scrabble or Contract Bridge, or familiarity with cryptic crossword puzzles were found to be influential attributes for the prediction.