BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage News Chronon - Airbnb’s End-to-End Feature Platform at QCon SF 2023

Chronon - Airbnb’s End-to-End Feature Platform at QCon SF 2023

At QCon SF, Airbnb staff software engineer Nikhil Simha presented Airbnb's solution to address the challenges of managing and serving the vast number of features used in machine learning models. The platform focuses on four key areas: core APIs, training data generation, feature serving, and feature observability.

Machine learning models typically use many features to generate a single prediction. This leads to an explosion in data pipelines and a high request fanout during prediction. Additionally, feature schemas evolve with every model iteration, making manual management cumbersome. Models can also fail silently with shifts in input data, making observability challenging. Furthermore, for use cases that involve ranking, the per-inference latency budgets are extremely tight.

Chronon aims to address these challenges by providing full support for the entire training data generation pipeline, including feature bootstrap, label computation, and training set generation at a large scale. The platform also supports advanced feature computation, such as feature derivations, feature chaining, and external and contextual feature support. This works in batch, streaming, and application-serving environments.

The platform uses Python to allow for more flexibility and reusability of code compared to SQL. It also incorporates time as a first-class citizen, allowing for the expression of time windows. This is particularly useful in fraud detection, where recent data is the most important.

Chronon also introduces the concept of windows, a variant of sliding and hopping windows. These windows provide the benefits of both types, offering freshness and cost-effectiveness. They are also more memory-efficient than sliding windows, making them easier to scale. These feature views are maintained by data and are a unit of reuse, making them immutable once they are online.

The system is currently in closed beta, intending to go open-source soon.

About the Author

Rate this Article

Adoption
Style

BT