In my last post (longer ago than I intended), I discussed the approaches I've come across to adding i18n to a database schema, and outlined my theory that the whole problem could be recast as one of data types. In order to explore how this would work in practice, I've put together a prototype "library" using only a standard install of PostgreSQL that shows how simple it could be, and why it makes sense.

Views and Functions

One of the things I haven't really touched on with my examples so far is that you will not normally interact directly with all the tables or data created by any i18n solution. You don't want to write double the code to perform the same query, just because it contains translated items, so you will generally try to wrap it up in a set of functions and/or views that provide a high level of abstraction.

What these functions look like will be closely related to the structure you've given your data, and some solutions lend themselves to simpler wrappers than others:

• If you have a separate table for every translated column, you may end up with dozens of functions like "get_country_name( language_code, country_id )"
• If you have a big table of text but reference items by category you might instead call "get_translated_item( 'database', 'country', 'name', country_id, language_code )"
• If you instead store the text_id as a foreign key reference, you might be able to simply say "Select translate( name_text_id, 'de' ) From country"

Another thing that will commonly be handled by functions or views is the fact that you won't have a full set of translations of every item in every language, so your code needs to do something other than break when a value is missing. The general approach is to have a "fall-back" language - code somewhere that says "if there's no French available, English will have to do". For some languages, there might be multiple levels of fall-back, e.g. if BrazilianPortuguese is unavailable, try European Portuguese, then English as a last resort.

If you store your fall-back rules in the database itself, any of the above functions could be handling all this logic behind the scenes. You will rarely need the ability to prioritise languages differently for different users.

Modification

Most solutions lend themselves to some variation of the above for simplifying the retrieval of translated content, but it is not always easy to simplify the modification of that content, with so many tables involved.

In the "atomic i18n" model, all your data is stored inside the original tables, so the abstraction functions don't need to know anything about the underlying data structure - they can just manipulate the values directly. So the function translate(item, language_code) just takes an item of type translated_text and looks for the appropriate language - the item might well be a particular table column, but it could just as easily be something calculated on the fly.

More significantly, the function set_translation(item, language_code, translation) can take a translated_text value, and return a new translated_text value, without updating any underlying tables. So to modify your data, you can just use INSERT and UPDATE queries, just as you would if it was a normal string:

• To create a new value, use an INSERT statement, with an appropriate representation of the set of translations to be added:
  Insert Into country ( name, currency ) Values ( i18n.translation_pair('en', 'Austria'), 'EUR' );
• To update a value, use an UPDATE statement:
  Update country Set name = translation_set(name,  'de', 'Österreich') Where id = 4
• To delete a value, use a DELETE statement - no need to look for dependencies, or set up cascade triggers.

Environment Variables

One of the things that the above retrieval functions all need - for obvious reasons - is which language you are translating into. But when you come to using that in an application, that becomes a bit of a pain - every time you mention a translation function, you have to insert a placeholder or a variable from the calling application to say what the current user's language is.

But what would be really nice is if you didn't have to pass any languages at all - I just want to say "the current user speaks German; now go away and do this query, but in German please". Or, more technically, to set an environment variable at the top of the query, and let the rest of the query silently refer to it.

Using Environment Variables in PL/pgSQL

When I started looking into this, I was working with Postgresql, and discovered three things:

1. some of the procedural languages available for Postgresql, such as PL/Perl, have support for session-wide global variables
2. the most common, PL/pgSQL, doesn't
3. my shared host only had PL/pgSQL installed.

So I've written a simple pair of functions - "env.set(item, value)" and "env.get(item)" - implemented using only PL/pgSQL. By wrapping them like this, I figure they could easily be replaced by completely equivalent - and much simpler - functions in PL/Perl, or whatever other language you may have to hand.

The implementation itself takes advantage of the way PostgreSQL handles temporary tables - namely, that they last for the entire "session" of queries, wherever they were created. So you can run one function that creates a temporary table, and another that accesses it, and they'll "see" the same table - but if you do that twice in separate "sessions", each session will have its own copy. Voilá, somewhere to put your environment variables.

Operators

The other thing you'll notice about the examples so far, is that they are quite wordy - there are various functions involved, and in the middle of a complex query they could get quite confusing. Luckily, Postgres also supports custom operators, so you can replace some of that wordiness with punctuation.

Whether using operators rather than functions makes things more or less readable will depend on the context, but the simplest operation, requesting a translation, now becomes as simple as using the -> (translate) operator:

Select id, name->'en' as name, currency From country;

Or using the environment variable approach, using the ->! operator, which means "translate to currently set language":

Select i18n.set_lang('en');

Select id, name->! as name, currency From country;

And, of course, operators can also make our updates simpler; this example uses || (combine) and ==> (create translation pair):

Update country Set name = name || ('fr' ==> 'Autriche') Where id = 4

Full Implementation

The best way to see how it all works is to play around with a working implementation, so I've put a fully documented "library" online as pg-atom: i18n. The implementation consists of the following parts:

• an env schema, consisting of env.set and env.get functions, currently implemented in Pl/Pgsql, using temporary tables
• an i18n schema, containing:
  • a language table, which defines the mapping from codes to numeric IDs and fall-back logic
  • a translated_text type, which is implemented as a standard Postgres text[] array, using the numeric language IDs as keys and the translation strings as values
  • a set of functions which implement the functionality for this new type
• a set of operators for the more i18n functions; I haven't declared these as inside the i18n schema, because that would make them usable only if you set your search_path variable first.

Limitations and Future Possibilities

It's important to note that I haven't as yet had a chance to test this in anything like a production environment, so it may still be a bit ropy. In particular, there may be performance issues, as I'm not sure how the functions and operators will interact with query plans, indexes, etc.

The other thing to point out is that while the abstraction is quite elegant, the underlying implementation is not ideal: Postgres arrays have strictly numeric, ordered, indexes, and cannot contain nulls, whereas I am effectively using them to emulate an associative array (or perhaps some kind of tuple). In fact, the only way to add something with a particular key to an array (in this case, the language ID is used as the key) is to repeatedly "pad" the array with dummy values until it is long enough; for this reason, the current implementation treats empty strings as though they were nulls.

There are other ways to simulate an associative array, but more tantalisingly, there is a module distributed with Postgres called hstore which implements true associative arrays. A better translated_text type could probably be built "on top of" that, and typecasts between translated_text and hstore values would be pretty handy.

The most obvious task when making any application multi-lingual is to make the UI translatable; there are absolutely tons of options for this, libraries in all sorts of languages, recipes, discussions, etc. But for many applications there's another task, just as important, which is taking a database, and allowing someone to translate the data inside that.

There are various approaches to "database i18n" too - generally involving re-designing your schema in such a way that the translations are to some extent part of the "normalised" data. But these tend to make the schema somewhat unwieldy, and writing ad hoc queries and reports becomes tricky. So it occurred to me that if you treat i18n not as a schema problem, but as a data type problem, you could save some of that pain.

The Problem

Just to be clear, let's have an example we can work through: if I have a table of countries in my database, and one of the columns is the name of the country, that name isn't going to be the same in all the languages of my new super-duper internationalised system. So let's say we start with this:

Existing Solutions

I've come upon various approaches to the database i18n problem, but they all group down to two basic ideas: using one big table, and treating i18n as a problem of normalization.

One Big Table

The basic idea of the "one big table" solution is to extract all the text that needs translating, and put it in one big list, ready for translation. This is, after all, how UI i18n usually works, so why not just treat your data as "some more translation entries"? As an example, this appears to be (I haven't actually experimented with it) what the built-in i18n system in CakePHP will do for you.

So, for our example, we create a new "text" table which contains all the text we want to translate. The primary key on this table will be a combination of some kind of unique ID, plus a language code; something like this:

Now probably the simplest way to use this is to replace the "name" column in our original "country" table with a reference to this new table - giving us something like this:

This looks like nicely normalised data - you can set up a foreign key from the "name_text_id" to the "text" table, and so forth. But what kind of a table is "text"? It has nothing to do with your schema - it's not really representing anything; if you think about it, you could just as well have a table called "numbers" as well, and reference it whenever you want a numeric value...

A slightly neater variant of this is to replace the "id" in the "text" column with some reference back to the table and column being translated. Maybe we already have a table holding all our UI messages, appropriately categorised. So, with a bit of extra normalisation, we might end up with this:

This is quite attractive - all your translations are in one place, so writing an interface to keep them up to date should be quite easy - but as well as the meaninglessness of the "text" table, we now have another oddity, which I call the "anorexic table problem".

The problem is that the "country" table has given away most of its data to the "text" table - without joining through, you can only learn the currency. And this is a mild case: you could start off with a table with 5 columns, decide all of them except the primary key need translating, and be left with a glorified sequence - a table with nothing but a primary key.

This isn't unique to i18n - indeed many extreme forms of normalisation pretty much require you to create such tables - but in practice, it's a nuisance, and a downside of a lot of i18n schemas.

I18n as N11n

The other set of i18n schemas I've come across can be broadly called "normalisation-based solutions" - they take the existing schema, add multiple languages to it, and apply the rules of normalisation to the result.

There are various ways this can end up looking, but the idea is to treat the translations on each table as you would any 1-to-many relationship: with a new table. So for our example, we might create this:

Now, this example only has one column to translate, but I've called my new table "country_name", implying that if there were multiple columns to translate, I would have created multiple new tables. So now not only do I have an "anorexic" country table, I have a huge number of new tables cluttering up my schema.

You don't have to go this far - you could instead create a "country_i18n" table, with columns for each of the bits of country information you need to store, which would at least keep this explosion down a bit. This seems to be (again, I'm afraid I haven't experimented) what the Symfony framework offers you.

There are a few other variations on this, as well - I came upon this article with various examples (demonstrated, oddly, in terms of XML, not a DBMS). One worth mentioning is the idea of base + translations, which avoids the "anorexic table problem" by leaving the base language in the original table, and only the translations in a new structure. This is certainly an interesting compromise - but it requires you to formally pick a single "base" language, and I suspect the inconsistency of where data is stored would make writing maintenance code trickier.

Translations as Sub-Atomic Particles

Both of the approaches I've mentioned, and all the variations within them, frame the underlying problem in the same terms: translations are additional pieces of data, and your schema needs to accommodate this data. But there is actually a different way of looking at the problem, and that is that the translations are all representations of the same value - and the act of translating them is about selecting a representation, not filtering and combining additional data.

In other words, translated text is a data type.

Now, one of the principles often cited in relation to databases is that of atomicity - values in the database should be atomic, with structure defined at a relational level, not within each value. But as many people have pointed out - see this blog post for example - it is quite hard to pin down just what atomic means.

If you're not allowed any structure, you shouldn't really be storing strings - they are, after all, an ordered list of characters. But, crucially, that has no bearing on the data you are trying to represent - you might want to model a "country", but you don't want to model a "name", you just want to store and use it. And - in my view - having multiple translations of that name needn't change that fact. As far as your schema is concerned, all the translations of a particular name form a single atom, and the individual strings are just a part of that value.

So just as you don't use the schema to link together the letters 'G', 'e', 'r', 'm', 'a', 'n', and 'y', you shouldn't use it to link together the translations 'Germany', 'Allemagne', and 'Deutschland'. Instead, you select a data type that represents the kind of data you are storing.

The Atomic I18n Solution

So if we go back to our original example, and look at what it would mean to treat i18n as a problem of data types, not schema design, what would it look like? Obviously, the table still has to change, but not by much - we just change the type of the "name" column to our new "translated text" type. Once we've stored our extra translations, we end up with something a bit like this:

Note that we don't need to create any new tables - our schema is effectively unchanged - and we've avoided the "anorexic table problem". We still have to update any existing queries to select the name in a particular language, rather than all of them at once, but we don't have to pull more tables into the query just to see what we're doing.

Of course, this relies on a "translated text" type being available in our DBMS, along with all the functions we'll need to interact with it.

This post has grown somewhat longer than I expected, so I have split it into two parts; in the next part, I will go into the details of my prototype implementation, which aims to create a user-friendly translated_text type from standard features of PostgreSQL.