Apache Flink 1.2 Released with Dynamic Rescaling, Security and Queryable State
Apache Flink 1.2 was announced and features dynamic rescaling, security, queryable state, and more. The release resolved 650 issues, maintains compatibility with all public APIs and ships with Apache Kafka 0.10 and Apache Mesos support.
Flink’s dynamic rescaling allows one to change the parallelism of a streaming job or of an operator within the job. This feature restores the job from a savepoint using a different parallelism and improves the operational management of streaming pipelines by scaling up and down accordingly, minimizing the downtime to a few seconds. Currently it's supported for some operators like Kafka consumers, Window operations and custom user state.
On security, authentication support was added against external services using Kerberos. It supports authentication against Zookeeper, Kafka, HDFS and YARN. Experimental SSL-based communications over the wire were also added to encrypt messages in transit between Flink executors.
Queryable state experimental feature exposes operator’s keyed state via an API. This can be useful to sense the beat of streaming pipelines without having to write the state into a data store. The state can be queried using a thread-safe asynchronous QueryableStateClient and some utilities are provided to handle serialization of state on both operator implementation and client.
Apache Kafka 0.10 support was also introduced as data source and sink. Kafka 0.10 connector can consume and produce messages with Flink’s internal event time. The Kafka consumer can emit records with Kafka message’s timestamp as event time, but doesn’t emit watermarks to advance ‘time’.
Apache Mesos was also added as a deployment option. Flink also supports DC/OS deployments, and Marathon orchestration platform can be leveraged to run highly available Flink clusters – Marathon runs all Flink components as Mesos tasks across the cluster and, whenever Flink master fails, another one is started. The state (running tasks) is kept and recovered from Zookeeper.
The Table API also got improvements. Window aggregations are supported in three different modes: tumbling windows, sliding windows and session (gap) windows. All modes support windowing on event time and processing time. Tumbling and sliding modes also support windowing on row count. Flink’s Table API and SQL introduced experimental access for POJO field access and expanded the list of built-in functions.