Multibyte for Rails: A Unicode Solution for Rails?

Posted by Obie Fernandez on Sep 25, 2006

Community

Ruby

Topics

Internationalization ,

Ruby on Rails

Tags

i18n ,

Rails Plugins ,

Unicode

The issue of proper Unicode support for Ruby on Rails continues to generate lots of discussion and development activity.

The Multibyte for Rails project was started by Julian 'Julik' Tarkhanov with the 'unicode_hacks' plugin. Manfred Stienstra, Jan Behrens and Thijs van der Vossen later joined the development team.

ActiveSupport::Multibyte extends the standard Ruby string with a chars proxy method. This proxy allows you to use a multibyte encoded string as a sequence of characters (or Unicode codepoints) instead of bytes. On chars you can call multibyte-safe implementations of all standard Ruby string methods. The proxy also provides methods for Unicode normalization, composition and decomposition and preliminary support for working with grapheme clusters.

Three months ago Julian submitted a test implementation of his ActiveSupport::Multibyte string extension patch to the Rails core team for inclusion in Ruby on Rails proper. According to a recent thread started by project team member and rails-core mailing list regular Manfred Siesta, the Multibyte team has been working continually on improvements to their extension for three months:

The code has been completely refactored to be more transparent and easier to understand. There is now a single optional accelerated backend and all multibyte-safe operations have a pure Ruby implementation. Test structure and coverage has also been greatly improved.

For anyone interested in trying out ActiveSupport::Multibyte, it is available as a plugin and can be converted to a patch using the included 'create_patch' rake task.

Why won't DHH and the core team just go ahead and patch Rails the way that folks such as Julian and his Multibyte team propose?

First of all, there is a performance penalty incurred by multibyte string operations. According to the project FAQ:

Multibyte safe operations through a proxy are obviously slower than single-byte operations directly on the string. The proxy introduces two levels of indirection and multibyte safe operations are more complex than single-byte operations and therefore slower.

A quick benchmark shows that for example a multibyte safe slice operation through the proxy is on average 50 times slower than a single-byte slice operation. Even though this makes the performance impact seem severe, remember that most of the string operations do not need to be multibyte safe. For a typical Rails application you're unlikely to even notice a performance penalty, but you have the satisfaction of knowing that you'll never break your user's text ever again.

Then there's the problem of UTF-8 and Unicode being terribly unpopular in Japan and China because of the Han Unification issue. On the other hand, as Sam Ruby pointed out in a brief comment to the rails-core mailing list:

Java and C# seem to do OK in Japan. I would also imagine that ASCII wouldn't be very popular in Japan. :-)

In the end, it seems that it will take more time before any single solution to the question of internationalization is adopted by Rails core. Copenhagen blogger Casper Fabricius sheds light on the situation based on recent comments by DHH at his user group:

...shouldn’t 5 or more plugins for internationalization indicate quite clearly that the Rails community craves unified support implemented in the core?

No, DHH answered, from the Core Team’s point of view, this means that people want to support and implement internationalization in a lot of different ways, and that there is no universal solution that will make everybody happy. Even inside the Core Team people can’t agree how it should be done. Although, DHH added, I can’t rule out that the 37signals needs internationalization, is the day that Rails get it.

1. Back to top

   Unicode support should be part of the language not a library

   Sep 26, 2006 2:54 PM by Faui Gerzigerk

   I really don't think it makes sense to put Unicode support into Rails. Why is this not part of Ruby itself? And if it has to be a library, it must be implemented in C, and it must be fast. The argument that you actually don't need it very often and thus it can be 50 times slower is just an excuse for not doing it properly. I mean we're talking about operations like getting the length of a string, searching in a string, extracting substrings, doing regular expressions, etc. These are very frequent operations.
   
   And we have all the character conversion stuff going on as well. Every time I read in something from an XML file/message or from a database or from an HTTP request, some conversion may be required. This has to be very fast indeed!

   Reply

2. Back to top

   Re: Unicode support should be part of the language not a library

   Sep 26, 2006 10:26 PM by Christian Romney

   Speed would be nice. Mutibyte text would be nicer. I wouldn't let speed get in the way of actually having the feature, especially as Ruby is already slow compared to many other languages yet is perfectly fast enough for all of my needs. The day it gets faster, well, that's just a bonus.

   Reply

3. Back to top

   Re: Unicode support should be part of the language not a library

   Sep 27, 2006 1:02 AM by Devin Ben-Hur

   Why is this not part of Ruby itself?

   
   
   If you really want to know, it's mostly because the ideographic asian languages (japanese, chinese, etc.) were given the shaft when
   unicode was developed by a bunch of arrogant westerners.
   
   Ruby's inventor is Japanese and Japan is still its main locus of use and development. Matz doesn't particularly like or need unicode, so he hasn't previously built support in to the language. However, he doesn't really to desire to cause other people pain, so he is actively working on improved unicode support for Ruby 1.9/2.0.
   
   You can read his
   ongoing thoughts and conversations. Try expressing your concerns via ruby-talk -- much more useful than complaining in a forum the language designer/implementor is unlikely to ever see.

   Reply

• Older >