An effective questionnaire is one that has been psychometrically validated.
This primarily means the items are reliable (consistent) and valid (measuring what we intend to measure).
So if we say a questionnaire measures perceptions of website usability, it should be able to differentiate between usable and unusable websites and do so consistently over time.
With websites and software products that have an international reach, a substantial part of the user base likely doesn’t speak English, or speaks English as a second language.
Here are four steps to take to get your questionnaire ready for a different language than it was written in.
1. Translate the questionnaire into the target language: Ideally you can have more than one person do the translating but it’s not essential. While Google’s translation is a quick tool to get the gist of text in a foreign language, don’t rely on it for translating. Below are the SUPR-Q items originally written in English and translated into French, along with the anchor labels to the response options for most of the items (strongly disagree to strongly agree).
|Original English SUPR-Q Items||Translated French SUPR-Q Items|
|This website is easy to use||L’utilisation de ce site web est facile.|
|It is easy to navigate within the website.||La navigation sur ce site est aisee.|
|The website keeps the promises it makes on its website.||Le site tient ses promesses.|
|I feel confident conducting business with this website.||Ce site m’inspire confiance.|
|How likely are you to recommend this website to a friend or colleague?||Pensez-vous recommander ce site a un ami ou un collegue?|
|I will likely visit this website in the future.||Il y a de fortes chances que je revienne sur ce site dans l’avenir.|
|I find the website to be attractive.||Je trouve ce site web tres attrayant.|
|The website has a clean and simple presentation.||La presentation de ce site est claire et simple.|
|Strongly Disagree — Strongly Agree||Pas du tout d’accord — Tout a fait d’accord|
2. Back translate: To get a sense of how effective the translation was, have another independent person translate the translated questionnaire back into the original language. Here are the original French SUPR-Q items translated back into English.
|Original English Items||French back to English Items|
|This website is easy to use||This website is user friendly.|
|It is easy to navigate within the website.||This site is easy to navigate.|
|The website keeps the promises it makes on its website.||The site keeps its promises.|
|I feel confident conducting business with this website.||I trust this site.|
|How likely are you to recommend this website to a friend or colleague?||Would you recommend this site to a friend or colleague?|
|I will likely visit this website in the future.||I am most likely to return to the site in the future.|
|I find the website to be attractive.||I find this site very pleasant|
|The website has a clean and simple presentation.||The presentation of this site is clear and simple.|
|Strongly Disagree — Strong Agree||Do not agree at all — Completely agree|
You can see from the items above that there’s generally good agreement for most items and a few that need to be addressed. But like the game of telephone, things get a little lost in translation. At this point you’ll want to have the two translators reconcile the differences.
3. Pretest the items: Administer the newly translated and refined items to a small sample (5-20 people) as you would with a typical set of participants. Unlike a typical administration, however, ask the participants of this study to comment on the items. Have them comment on words that don’t quite translate well, are ambiguous, or where another word or phrase will do. Adjust the items again.
During this process look for words or phrases that don’t translate well or lose their meaning. For example, in rating the visual appeal of a product the item from the German Attrakdiff scale “takes me distant from people to brings me closer to people” just doesn’t flow smoothly in English. As a side note, some words don’t work well in the original language to native speakers. We’ve seen some native and non-native English speakers have trouble understanding the word “cumbersome” from item 8 on the System Usability Scale (SUS). Multiple researchers have found using awkward works just as well statistically and is better understood.
4. Validate the items: Even after carefully translating and back translating items, there’s no guarantee this newly translated version will have the same psychometric properties as the original one. Now administer the new questionnaire to a larger sample size to validate the items. You’ll want to aim for 5 people per item, so for a 10 item instrument you’ll want a minimum of 50 people to revalidate the items with. It’s not always possible to include this step when translating a questionnaire when using it for real-world applications. However, this is an essential step for a higher stakes questionnaire and ones that will be used repeatedly.
For an example of how most of these steps are performed, see a recent translation of the CSUQ from English to Turkish that Jim Lewis helped conduct.
Learn More: UX Measurement Boot Camp
Intensive Training on UX Methods, Metrics and Measurement
|Fall 2020: Delivered Online|