As e-commerce platforms embrace global audiences, the demand for precise, context-aware translations has surged. Traditional translation tools, while effective to an extent, often miss out on brand-specific nuances and cultural subtleties. AI-driven language models offer advanced capabilities in translations, going beyond just interpreting literal text. They understand the underlying context, sentiment, and cultural symbolism, ensuring the most accurate and effective representation.
We recommend getting started with OpenAI’s language models based on their performance, ease-of-use, documentation, fine-tuning capabilities, and multimodal features.
For most applications, GPT-3.5 is cheaper and faster than GPT-4. However, GPT-4’s multimodal features allow you to translate text in images, whereas GPT-3.5 lacks multimodality.
Setting up your own language models is challenging, especially if you’re self-hosting. If you need to customize your own, consider smaller language models like Mistral by Hugging Face. For multimodality, also consider Adept’s new open-source multimodal Fuyu-8B.
The linguistic adeptness of GPT-3.5 and GPT-4 encompasses a wide linguistic span, including but not limited to Afrikaans, Russian, and Indonesian. (See OpenAI’s report; accuracy above 80% generally denotes good proficiency.)
GPT-4 outperforms Google Translate when translating from English to French, Italian, German, Russian, and Portuguese among others but may struggle with Urdu, Burmese, and Telugu. However, Google Translate may offer superior translation quality for some languages. Despite this, the focused linguistic expertise of GPT in more than a dozen languages showcases a robust and nuanced understanding, offering a high-quality translation service within its area of linguistic competence.
Smarter internationalization (i18n)
Language models are a powerful tool for generating translations of website content, including product descriptions and reviews.
Whereas Google Translate struggles to grasp the context of what we’re translating, language models generate context-aware translations and excel at handling nuances.
Let’s take a branded product name like Raspberry Pi as an example. Google Translate will correctly translate instances of Raspberry Pi™ but will convert all other instances of Raspberry Pi to Tarte aux framboises (raspberry pie) or Framboise Pi with some coaxing.
With language models, you can skip appending ™ throughout your website and instead give it a pre-prompt explaining that Raspberry Pi refers to a branded product and to keep Raspberry Pi as is in the French translation.
With something simpler like a Save button, you won’t need to specify that Save refers to saving content, rather than someone’s life, as GPT-3.5 will understand the context from the entire website.
Similarly, language models are proficient in translating idioms and providing culturally appropriate translations.
Prompt engineering and architecting tips for i18n
Language models return slightly different results each time, unlike machine translations, but can be made more consistent by, for example:
| Provide context | "You are a translator, able to see JSON objects and convert the values to French. You are working with [company name]’s website, so you want to maintain their tone, voice and brand-specific terms. Ensure that you keep in mind the context of websites and user interactions when looking at particular words." |
| Specify the expected input and output formats | "You are given a JSON object, and you respond only with a new translated JSON object, no other text." |
| Restrict the length of the translation | "Any text you translate should be the same length, if not shorter, than the English version to maintain page consistency, as elements are dependent on text not exceeding certain character limits." |
| Set the temperature | Temperature controls the randomness of the output. You can set the temperature between 0 and 1. A higher temperature results in more creative and imaginative text. Test different temperature values to find what works best for you. |
For languages that are longer than English (e.g., French, Spanish), it’s especially important to check the translated length to avoid having text overflowing from buttons and other components.
You can set up custom validation to return a warning to the user if the translation is too long, or self-heal by recalling the API if the returned object is too large and ask it to shorten it. Self-healing is a great way to make systems smarter.
Let’s say, for instance, that hyphens are unacceptable characters. You can return a prespecified error if a user inputs a hyphen. If a string containing hyphens was passed anyway, the system can remove the hyphens.
Prompting the language model to use short hands is especially useful for longer languages: Feel free to use intelligent short hands or truncating rules that make sense for French. Be careful, however, as it can aggressively shorten words and provide nonsensical abbreviations.
Remember to also check that the returned object is valid before saving it to your CMS (e.g., a proper JSON object with the required structure).
Translate new content automatically with a CMS trigger
By connecting a language model with your headless CMS, you can generate translations automatically every time new content is saved.
A headless CMS essentially stores all of your website copy and other content as JSON objects. You can use a Webhook or an API to retrieve all the content you’ll need to translate and localize. You can also upload HTML files to GPT-4 if you’re unable to bulk download website content. (To learn more about how to query headless CMS data, read our guide.)
You can trigger new translations whenever there’s new CMS content data from an API endpoint:
import requests
import json
OPENAI_API_KEY = "api-key-goes-here"
prompt = ("Translate the following JSON object from English to French. "
"Only rename the values and leave the keys or otherwise "
"non-phrasing language unchanged")
def translate_json(json_obj):
# Define the endpoint and headers for the OpenAI API
# Replace <model-name> with the actual endpoint
endpoint = "https://api.openai.com/v1/engines/<model-name>/completions"
headers = {
"Authorization": f"Bearer {OPENAI_API_KEY}",
"Content-Type": "application/json",
}
# Formulate the instruction for translation
instruction = f"{prompt}: {json.dumps(json_obj)}"
# Prepare the data for the API request
# Adjust max_tokens based on the size of your JSON object
data = {
"prompt": instruction,
"max_tokens": 500
}
# Send a request to the API
response = requests.post(endpoint, headers=headers, json=data)
# Check for a successful response
if response.status_code == 200:
# Extract the translated JSON object from the response
translated_text = response.json()['choices'][0]['text'].strip()
try:
translated_json_obj = json.loads(translated_text)
return translated_json_obj
except json.JSONDecodeError:
print("Failed to decode the translated text as JSON")
return None
else:
print(response.status_code)
print(response.text)
return NoneSanity, Contentful, and other headless CMSes offer localization plugins that you can use to upload translated product page content and generate localized pages.
Many headless CMSes offer localization plugins that allow content authors to manage translations without developer support.
Cost considerations
With language models, you can create a highly efficient and intelligent translation system that saves your organization time and resources. However, existing workflows with machine translation are ill suited to how you should architect language models.
For instance, frequently translating website content can quickly rack up your bill. To reduce cost and improve functionality for teams managing translations, you should consider only translating parts of an object that changed, rather than the entire object. You should also cache and reuse translations to avoid re-translating the same phrases.
You could also set up a flag for users to trigger a full re-translation where translating only the new or revised strings would lose the necessary context cues.
Language models can be 3-4 times more expensive than Google Translate, but prices are dropping rapidly with each new model. Here are the estimated costs for translating 50 pages, assuming there are approximately 300 words or 4 paragraphs per page, as well as character limits based on language models’ token limits and Google Translate’s maximum request size of 100KB.
Prices for language models are falling quickly as they scale, become more efficient (e.g., with sparsely networked architecture), and benefit from more computing power.
Final thoughts
In any international setting, the ability to effectively communicate with a global audience has become paramount. Traditional translation tools, although beneficial, often fall short when nuances, brand-specific language, and cultural subtleties come into play.
The advent of AI-driven language models presents an exciting breakthrough, delivering context-rich, culturally attuned translations, bridging the gap between businesses and their diverse clientele. As we've explored, not only can these models capture brand terminologies and idiomatic expressions with precision, but they can also seamlessly integrate with various content management systems and offer significant cost benefits in the long run.
Embracing this transformative approach ensures that brands maintain authenticity, relevance, and engagement across all international expansions.
