LinkedIn Open Sources IndexTank, a Customizable Indexing Engine
LinkedIn has open sourced IndexTank, a document indexing engine that runs on the cloud and lets users customize the indexing process and tweak the results.
IndexTank was launched about a year ago and it was acquired by LinkedIn in October, and was recently open sourced. IndexTank is a cloud service similar to Google Custom Search running on top of Amazon Web Services, and providing websites the ability to index their own content that is later searchable by their visitors. IndexTank claims their users have complete control over what is indexed, when, and how the results are sorted. That means a website can promote at the top of search results the documents they prefer to show up first, and not relying on Google’s search algorithm.
Unlike many websites, IndexTank does not crawl web pages in order to index them, but rather the websites send data to be indexed to indexing engine. As a result, a document can be indexed right after its creation, providing live results. Also, the service is adds free.
IndexTank has three main components:
- Index Engine – the engine indexes only simple text. PDF, MS Doc, and other document types need to be converted to text in order to be indexed.
- API – a RESTful interface accessed via Java, Python, .NET, Ruby and PHP clients.
- Nebulizer – a multitenant framework hosting an unlimited number of indexing engines running on an IaaS infrastructure.
IndexTank claims they have attracted thousands of customers in one year, the most notable being Reddit, but the company was not yet on profit at the time of being acquired by LinkedIn.