The Unicode Debate Rekindled

Posted by Obie Fernandez on Jun 17, 2006

Sections

Development,

Architecture & Design

Topics

Ruby ,

Programming

Tags

JRuby ,

Rails ,

Unicode

The great ongoing Unicode in Ruby debate sprung back to life this week with a thread on the ruby-talk mailing list asking, is there a plan to get Unicode support into the language anytime soon? The seemingly simple question ballooned into a huge discussion that included Matz and members of the Rails and JRuby teams.

Unicode is an industry standard designed to support computer encoding of text and symbols from all of the writing systems of the world. Unicode characters can be encoded (as bytes) using any of several schemes termed Unicode Transformation Formats (UTF). Does Ruby support Unicode? It depends on who you ask, hence the perennial debate about how to improve Unicode support in future versions of the language.

The question of Unicode support is important for Charles Nutter, one of the leads of the JRuby project. Since Java does support Unicode, and quite well at that, it is particularly embarrassing for JRuby, running on the JVM, to not support it. Charles asks the blogsphere, "What should that support look like?"

As for Rails, DHH has taken the position that it is the responsibility of Rails application developers to handle Unicode properly, rather than creating a different set of string manipulation methods for Rails. That hasn't stopped people from asking for Rails-based solutions, or proposing them. Julian Tarkhanov has probably done the most to help ease the pain of Unicode in Rails. His proxy solution adds a character array property to Ruby's String objects that handles Unicode characters properly.

Unicode specifies the encoding of the characters of strings,rather than the glyphs (renderings) for such characters, which has led to controversies over the usefulness of Unicode for Chinese, Japanese and Korean languages. The process of Han unification, where essentially a common set of glyphs was developed for use by all three languages was controversial, with most of the opposition coming from Japan. Opponents of Han unification state that it steamrolls over thousands of years of cultural tradition, misses many of the subtleties that are one of the most important features of these languages, and renders serious literature and academic research in these languages impossible. Since Ruby is from Japan, it should not be surprising that support for Unicode in Ruby has been controversial too.

All Ruby programs are written (encoded) in 7-bit ASCII, Kanji or UTF8. If a code set other than ASCII is used, a global option named KCODE must be set appropriately. The option is used by Ruby's string manipulation methods, because internally Ruby keeps string data as a stream of bytes (one byte per character). Some Unicode encodings specify multiple bytes per character. In contrast, Java and other modern languages have a relatively easier time handling different types of character encodings and multi-byte characters because their String objects are arrays of character objects.

For further reading, see the Fingertips writeup and the RedHanded article

1. Back to top

   already a possible solution?

   by Alex Popescu

   It looks like Rob Leslie has already proposed a solution on the ruby-talk ML.
   
   ./alex
   --
   .w( the_mindstorm )p.

   Reply

• Older >