LinkedIn’s Joel Koshy details their Kafka usage, debugging and monitoring two production incidents in using the core Kafka infrastructure concepts, semantics and behavioral patterns to plan for and detect similar problems in the future.
LinkedIn has recently open sourced QARK, a static analysis tool meant to discover potential security vulnerabilities existing in Android applications written in Java.
LinkedIn recently open sourced Cubert, its High Performance Computation Engine for Complex Big Data Analytics. Cubert is a framework written for analysts and data scientists in mind.Developed completely in Java and expressed as a scripting language, Cubert is designed for complex joins and aggregations that frequently arise in the reporting world.
At the 2014 QCon San Francisco conference, LinkedIn's Lin Qiao gave a talk on their Gobblin project (also summarized in a blog post) that is a unified data ingestion system for their internal and external data sources.
Twitter’s engineering group, known for various contributions to open source from streaming MapReduce to front-end framework Bootstrap recently announced open sourcing an algorithm that can efficiently recommend content. LinkedIn also open sourced a Machine Learning library of its own, ml-ease. In this article we present the algorithms and what they mean for the open source community.
Facebook, Google, LinkedIn, and Twitter have decided to make sure that a relational databases is “web-scale”, so they have put their efforts behind WebScaleSQL, a branch of MySQL 5.6 Community Edition.
Apache has released Kafka 0.8, the first major release of Kafka since becoming an Apache Software Foundation top level project. Apache Kafka is publish-subscribe messaging implemented as a distributed commit log, suitable for both offline and online message consumption. It is a messaging system initially developed at LinkedIn for collecting and delivering high volumes of event and log data.
LinkedIn has open sourced IndexTank, a document indexing engine that runs on the cloud and lets users customize the indexing process and tweak the results.
Last week SpringSource released a first milestone for Spring Social, a Spring-based template for accessing Twitter, LinkedIn, Tripit and Facebook from within Java programs. Rather than exposing generic, URL-based APIs, the Spring Social APIs are designed specifically for each site and make integrating with those sites straight forward. InfoQ examines the new API as well as some alternatives.