Forcing the fit: MQM in the age of automated evaluation
In the realm of translation evaluation, the Multidimensional Quality Metrics (MQM) have long been heralded as a robust framework grounded in human judgment.
Yet, as we venture deeper into the age of automated evaluation, there's a growing push to reconcile the meticulous, nuanced evaluations MQM is known for with the streamlined capabilities of automated systems. This evolution in approach is epitomized by the ongoing work on projects like COMET and insights gained from shared tasks such as WMT25.
The traditional MQM's strength lies in its comprehensive typology, initially sprawling over 100 categories. While thorough, such granularity inevitably poses challenges when interfacing with modern machine evaluation technology. The tide is shifting toward a reduced typology of just five categories, which promises to make model evaluations more tractable. The impetus is clear: simplification could bolster the efficiency and applicability of machine-driven assessment without jettisoning the nuanced insights human-centric evaluation brings.
Craig Stewart, an advocate for MQM's foundational role, underscores its enduring value, asserting “MQM was the most robust answer we had for human evaluation” and pointing to recent evidence that reinforces this view. Yet, the emphasis now shifts to a contextual understanding of quality—one that's informed by specific content, audience needs, and workflow stages. Such a tailored approach aligns more closely with the real-world applications and expectations of translation outputs today.
For language professionals, this transition is both a challenge and an opportunity. It calls for an increased synergy between human expertise and machine efficiency, demanding that evaluators not only master the traditional standards but also become adept in applying adaptive, contextual criteria. As the industry grapples with these evolving standards, practitioners must engage critically with how quality is defined in each unique scenario they encounter, ensuring that automated tools not only complement but enhance the human touch in translation evaluation.
LocReport is free and independent. If it helps you stay informed, consider buying us a coffee — it goes a long way.
☕ Buy me a coffee