Translation quality assessment in computer-assisted translation tools

The increasing interest in measuring translation quality according to objective methods and standard metrics has led some translation software companies to start integrating assessment-enabling features into their products. Despite the adequacy of computer-assisted translation tools for providing support to human assessment tasks and tool makers’ claims about their success in its implementation, the actual capacities of the tools for quality assessment still need to be demonstrated. Formal translation quality assessment is a complex activity that requires the creation of issue typologies, severity-based penalties, quality score calculations and quality report generation, among other elements. Also, since one of the assessment goals is to improve the quality, it is crucial that metrics, annotations and reports can be consumed by all stakeholders without the need of additional transformation. In this sense, interoperability aspects and standards must also be respected for the assessment effort to be useful and efficient. In this study, a set of five translation tools integrating assessment features were examined according to two main aspects: assessment feature implementation and assessment data interoperability. To carry out the evaluation, a list of sixteen items were defined under those two aspects, from the capacity of customising a quality model to the exchangeability of the assessment data produced. These items were primarily inspired in one of the most comprehensive quality frameworks to date: the Multidimensional Quality Metrics. The results confirm that computer-assisted translation environments do support the most basic characteristics of translation quality assessment, such as applying custom quality metrics or generating scores and reports. However, there is still room for improvement in many other aspects and, in particular, those related to the exchangeability of the assessment data. Finally, if computer-assisted translation tools aim to be the perfect instrument for the assessment of translation quality, then existing quality frameworks must be fully understood, followed and enabled by tool makers, with special emphasis on the normative aspects of the models and the standard formats used to store and exchange quality assessment metrics and results.