BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Lucene Oracle Integration Looks to Surpass Oracle Text

| by Ian Roughley Follow 0 Followers on Oct 01, 2007. Estimated reading time: 2 minutes |

New work to enhancement LUCENE-724 (http://issues.apache.org/jira/browse/LUCENE-724) by Marcelo Ochoa allows for better integration of Lucene as a domain index from within the Oracle database. The updates, that were commissioned by Lending Club (www.lendingclub.com), allow greater flexibility than using Oracle Text - the results of the work is open source that anyone can use.

Why put Lucene in the database? Marcelo Ochoa answers this question with:

Oracle include a full feature Enterprise Engine named Oracle Text made in C and fully integrated to the Oracle Engine, but:
  • you can not control which functionality will be included into next release
  • and you can not easily customize it for your needs

 

The OJVMDirectory, a Lucene Integration running inside the Oracle JVM will allow developers to:

  • Indexing both structured and unstructured data
  • Create Lucene "synthetic document" that are comprised of fields populated from diverse tables, which eliminates the need for complex joins at query time
  • Query the Lucene index directly from Java

 

The features of the new release include:

  • Synchronized with latest Lucene 2.2.0 production
  • Replaced in memory storage using Vector based implementation by direct BLOB IO, reducing memory usage for large index
  • Support for user data stores, which means you are not limited to only one column/index at a time (limited imposed by Data Cartridge API on 10g); now you can index multiples columns at base table and columns on related tabled joined together
  • User Data Stores can be customized by the user, it means writing a simple Java Class users can control which column are indexed, padding used or any other functionality previous to document adding step
  • There is a DefaultUserDataStore which gets all columns of the query and built a Lucene Document with Fields representing each database columns these fields are automatically padded if they have NUMBER or rounded if they have DATE data, for example
  • lcontains() SQL operator support full Lucene's QueryParser syntax to provide access to all columns indexed, see examples below
  • Support for DOMAIN_INDEX_SORT and FIRST_ROWS hint, it means that if you want to get rows order by lscore() operator (ascending,descending) the optimizer hint will assume that Lucene Domain Index will returns rowids in proper order avoided an inline-view to sort it
  • Automatic index synchronization by using AQ's Call Back
  • Lucene Domain Index creates extra tables named IndexName$T and an Oracle AQ named IndexName$Q with his storage table IndexName$QT at user's schema, so you can alter storage's preference if you want
  • ojvm project is at SourceForge.net CVS, so anybody can get it and collaborate ;)
  • Tested against 10gR2 and 11g database

 

Examples and information can be found at dbprism/ojvm project, and the code can be downloaded from the project also.

More details on the integration can be found at Marcelo's blog.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Oracle-Lucene Integration: A powerful alternative. by Joaquin Delgado

Both Marcelo and I worked on this project and would like to clarify that its purpose is not to replace Oracle Text based on performance characteristics (nothing can beat a low level C implementation :-), but rather to provide a powerful and flexible open source alternative which brings a myriad of possibilities that will for sure advance the state-of-the-art in both search (information retrieval) and relational database fields.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT