LinkedIn recently open sourced Cubert, its High Performance Computation Engine for Complex Big Data Analytics. Cubert is a framework written for analysts and data scientists in mind.Developed completely in Java and expressed as a scripting language, Cubert is designed for complex joins and aggregations that frequently arise in the reporting world.
At the 2014 QCon San Francisco conference, LinkedIn's Lin Qiao gave a talk on their Gobblin project (also summarized in a blog post) that is a unified data ingestion system for their internal and external data sources.
Twitter’s engineering group, known for various contributions to open source from streaming MapReduce to front-end framework Bootstrap recently announced open sourcing an algorithm that can efficiently recommend content. LinkedIn also open sourced a Machine Learning library of its own, ml-ease. In this article we present the algorithms and what they mean for the open source community.
Facebook, Google, LinkedIn, and Twitter have decided to make sure that a relational databases is “web-scale”, so they have put their efforts behind WebScaleSQL, a branch of MySQL 5.6 Community Edition.
Apache has released Kafka 0.8, the first major release of Kafka since becoming an Apache Software Foundation top level project. Apache Kafka is publish-subscribe messaging implemented as a distributed commit log, suitable for both offline and online message consumption. It is a messaging system initially developed at LinkedIn for collecting and delivering high volumes of event and log data.
LinkedIn has open sourced IndexTank, a document indexing engine that runs on the cloud and lets users customize the indexing process and tweak the results.
Last week SpringSource released a first milestone for Spring Social, a Spring-based template for accessing Twitter, LinkedIn, Tripit and Facebook from within Java programs. Rather than exposing generic, URL-based APIs, the Spring Social APIs are designed specifically for each site and make integrating with those sites straight forward. InfoQ examines the new API as well as some alternatives.