Introducing Unbabel’s CUA 2.0, delivering consistency and clarity into the quality of your translations
Whenever you make a purchase, from a simple personal product to a major business service, there’s one thing that sets apart any successful transaction. It’s not the product itself, nor the price tag — it’s trust.
Would you purchase a car, a home, insurance, or anything else without knowing exactly what you’re getting? Measuring quality is easier when it comes to everyday products and services. For example, it’s not difficult to tell whether you’re purchasing clothes from high-end retail like Farfetch or a budget brand.
But things are a little different when you are not the end user of the services you are acquiring. Take translation services. Unless you can speak both the source and translated languages, it’s impossible to determine the quality. Unfortunately, many companies take advantage of this.
Almost every translation platform and language service provider claims to offer high-quality translations at scale, but how many of them have the ability to prove it? Is it even possible?
Customer Utility Analysis (CUA)
The quality evaluation of machine translations and specific translations is achieved through a combination of processes and frameworks, leveraging both human-led and automated methods. When it comes to surfacing quality to customers, Unbabel developed a visual framework to represent translation quality called Customer Utility Analysis (CUA).
We measure translation quality by following the MQM methodology, which is widely used in the translation industry as a standard. As MQM is a metric, it needs to be interpreted properly to avoid confusion and to be accessible to everyone. Due to these reasons, we bucketed the MQM scores of our translations into five different buckets, each one of them representing different levels of quality. As CUA is a visual framework, each quality bucket has a different color so that we can easily distinguish between good and bad quality. In other words, CUA is a visual representation of the quality of the translation delivered by Unbabel and is displayed to our customers on the Portal.
Below, we’ve outlined the thresholds and explanations for each category:
CUA 2.0 – Improving Quality Assessment and Transparency
Our recently launched and improved CUA 2.0 was designed to handle the translation of more complex content types. The earlier version of CUA was tailored specifically to pre-existing content types and employed varying thresholds for each, resulting in different interpretations. For instance, the definition of ‘green’ for Chat was not the same as the definition of ‘green’ for FAQs, but the MQM thresholds for these two very distinct content types were different. With the new version of CUA, we unified thresholds and definitions across all content types so that we eliminate different interpretations of the results.
This means that now every customer is provided with a uniform framework for every type of content we offer, ensuring a consistent and clear understanding of the quality.
Introducing the ‘Best’ Level
Finally, with CUA 2.0, we have also enhanced our quality evaluation framework by introducing a fifth and highest level, ‘Best,’ which represents the premium quality translation. This addition not only sets expectations for top-notch quality in more complex content, but also emphasizes value for money in lower quality levels.
Transparency: CUA in the Unbabel Portal
In order to ensure transparency when communicating with our customers, the new CUA 2.0 will also replace the existing CUA in the Portal and is an addition to the Pipeline Builder as a means of visualizing quality KPIs, and it is used for research and monitoring purposes in LangOps and other teams.
At Unbabel, we believe that trust and transparency are the keys to elevating business relationships to new heights of excellence.
