Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Multibyte for Rails: A Unicode Solution for Rails?

Multibyte for Rails: A Unicode Solution for Rails?

The issue of proper Unicode support for Ruby on Rails continues to generate lots of discussion and development activity.

The Multibyte for Rails project was started by Julian 'Julik' Tarkhanov with the 'unicode_hacks' plugin. Manfred Stienstra, Jan Behrens and Thijs van der Vossen later joined the development team.

ActiveSupport::Multibyte extends the standard Ruby string with a chars proxy method. This proxy allows you to use a multibyte encoded string as a sequence of characters (or Unicode codepoints) instead of bytes. On chars you can call multibyte-safe implementations of all standard Ruby string methods. The proxy also provides methods for Unicode normalization, composition and decomposition and preliminary support for working with grapheme clusters.

Three months ago Julian submitted a test implementation of his ActiveSupport::Multibyte string extension patch to the Rails core team for inclusion in Ruby on Rails proper. According to a recent thread started by project team member and rails-core mailing list regular Manfred Siesta, the Multibyte team has been working continually on improvements to their extension for three months:

The code has been completely refactored to be more transparent and easier to understand. There is now a single optional accelerated backend and all multibyte-safe operations have a pure Ruby implementation. Test structure and coverage has also been greatly improved.

For anyone interested in trying out ActiveSupport::Multibyte, it is available as a plugin and can be converted to a patch using the included 'create_patch' rake task.

Why won't DHH and the core team just go ahead and patch Rails the way that folks such as Julian and his Multibyte team propose?

First of all, there is a performance penalty incurred by multibyte string operations. According to the project FAQ:

Multibyte safe operations through a proxy are obviously slower than single-byte operations directly on the string. The proxy introduces two levels of indirection and multibyte safe operations are more complex than single-byte operations and therefore slower.

A quick benchmark shows that for example a multibyte safe slice operation through the proxy is on average 50 times slower than a single-byte slice operation. Even though this makes the performance impact seem severe, remember that most of the string operations do not need to be multibyte safe. For a typical Rails application you're unlikely to even notice a performance penalty, but you have the satisfaction of knowing that you'll never break your user's text ever again.

Then there's the problem of UTF-8 and Unicode being terribly unpopular in Japan and China because of the Han Unification issue. On the other hand, as Sam Ruby pointed out in a brief comment to the rails-core mailing list:

Java and C# seem to do OK in Japan. I would also imagine that ASCII wouldn't be very popular in Japan. :-)

In the end, it seems that it will take more time before any single solution to the question of internationalization is adopted by Rails core. Copenhagen blogger Casper Fabricius sheds light on the situation based on recent comments by DHH at his user group:

...shouldn’t 5 or more plugins for internationalization indicate quite clearly that the Rails community craves unified support implemented in the core?

No, DHH answered, from the Core Team’s point of view, this means that people want to support and implement internationalization in a lot of different ways, and that there is no universal solution that will make everybody happy. Even inside the Core Team people can’t agree how it should be done. Although, DHH added, I can’t rule out that the 37signals needs internationalization, is the day that Rails get it.

Rate this Article