BT

Multibyte for Rails: A Unicode Solution for Rails?

by Obie Fernandez on Sep 25, 2006 |

The issue of proper Unicode support for Ruby on Rails continues to generate lots of discussion and development activity.

The Multibyte for Rails project was started by Julian 'Julik' Tarkhanov with the 'unicode_hacks' plugin. Manfred Stienstra, Jan Behrens and Thijs van der Vossen later joined the development team.

ActiveSupport::Multibyte extends the standard Ruby string with a chars proxy method. This proxy allows you to use a multibyte encoded string as a sequence of characters (or Unicode codepoints) instead of bytes. On chars you can call multibyte-safe implementations of all standard Ruby string methods. The proxy also provides methods for Unicode normalization, composition and decomposition and preliminary support for working with grapheme clusters.

Three months ago Julian submitted a test implementation of his ActiveSupport::Multibyte string extension patch to the Rails core team for inclusion in Ruby on Rails proper. According to a recent thread started by project team member and rails-core mailing list regular Manfred Siesta, the Multibyte team has been working continually on improvements to their extension for three months:

The code has been completely refactored to be more transparent and easier to understand. There is now a single optional accelerated backend and all multibyte-safe operations have a pure Ruby implementation. Test structure and coverage has also been greatly improved.

For anyone interested in trying out ActiveSupport::Multibyte, it is available as a plugin and can be converted to a patch using the included 'create_patch' rake task.

Why won't DHH and the core team just go ahead and patch Rails the way that folks such as Julian and his Multibyte team propose?

First of all, there is a performance penalty incurred by multibyte string operations. According to the project FAQ:

Multibyte safe operations through a proxy are obviously slower than single-byte operations directly on the string. The proxy introduces two levels of indirection and multibyte safe operations are more complex than single-byte operations and therefore slower.

A quick benchmark shows that for example a multibyte safe slice operation through the proxy is on average 50 times slower than a single-byte slice operation. Even though this makes the performance impact seem severe, remember that most of the string operations do not need to be multibyte safe. For a typical Rails application you're unlikely to even notice a performance penalty, but you have the satisfaction of knowing that you'll never break your user's text ever again.

Then there's the problem of UTF-8 and Unicode being terribly unpopular in Japan and China because of the Han Unification issue. On the other hand, as Sam Ruby pointed out in a brief comment to the rails-core mailing list:

Java and C# seem to do OK in Japan. I would also imagine that ASCII wouldn't be very popular in Japan. :-)

In the end, it seems that it will take more time before any single solution to the question of internationalization is adopted by Rails core. Copenhagen blogger Casper Fabricius sheds light on the situation based on recent comments by DHH at his user group:

...shouldn’t 5 or more plugins for internationalization indicate quite clearly that the Rails community craves unified support implemented in the core?

No, DHH answered, from the Core Team’s point of view, this means that people want to support and implement internationalization in a lot of different ways, and that there is no universal solution that will make everybody happy. Even inside the Core Team people can’t agree how it should be done. Although, DHH added, I can’t rule out that the 37signals needs internationalization, is the day that Rails get it.

Hello stranger!

You need to Register an InfoQ account or to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Unicode support should be part of the language not a library by Faui Gerzigerk

I really don't think it makes sense to put Unicode support into Rails. Why is this not part of Ruby itself? And if it has to be a library, it must be implemented in C, and it must be fast. The argument that you actually don't need it very often and thus it can be 50 times slower is just an excuse for not doing it properly. I mean we're talking about operations like getting the length of a string, searching in a string, extracting substrings, doing regular expressions, etc. These are very frequent operations.

And we have all the character conversion stuff going on as well. Every time I read in something from an XML file/message or from a database or from an HTTP request, some conversion may be required. This has to be very fast indeed!

Re: Unicode support should be part of the language not a library by Christian Romney

Speed would be nice. Mutibyte text would be nicer. I wouldn't let speed get in the way of actually having the feature, especially as Ruby is already slow compared to many other languages yet is perfectly fast enough for all of my needs. The day it gets faster, well, that's just a bonus.

Re: Unicode support should be part of the language not a library by Devin Ben-Hur

Why is this not part of Ruby itself?


If you really want to know, it's mostly because the ideographic asian languages (japanese, chinese, etc.) were given the shaft when
unicode was developed
by a bunch of arrogant westerners.

Ruby's inventor is Japanese and Japan is still its main locus of use and development. Matz doesn't particularly like or need unicode, so he hasn't previously built support in to the language. However, he doesn't really to desire to cause other people pain, so he is actively working on improved unicode support for Ruby 1.9/2.0.

You can read his
ongoing thoughts and conversations
. Try expressing your concerns via ruby-talk -- much more useful than complaining in a forum the language designer/implementor is unlikely to ever see.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

3 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2013 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT