BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

Apache Flink 1.2 Released with Dynamic Rescaling, Security and Queryable State

| by Alexandre Rodrigues on Feb 15, 2017. Estimated reading time: 1 minute |

Apache Flink 1.2 was announced and features dynamic rescaling, security, queryable state, and more. The release resolved 650 issues, maintains compatibility with all public APIs and ships with Apache Kafka 0.10 and Apache Mesos support.

Flink’s dynamic rescaling allows one to change the parallelism of a streaming job or of an operator within the job. This feature restores the job from a savepoint using a different parallelism and improves the operational management of streaming pipelines by scaling up and down accordingly, minimizing the downtime to a few seconds. Currently it's supported for some operators like Kafka consumers, Window operations and custom user state.

On security, authentication support was added against external services using Kerberos. It supports authentication against Zookeeper, Kafka, HDFS and YARN. Experimental SSL-based communications over the wire were also added to encrypt messages in transit between Flink executors.

Queryable state experimental feature exposes operator’s keyed state via an API. This can be useful to sense the beat of streaming pipelines without having to write the state into a data store. The state can be queried using a thread-safe asynchronous QueryableStateClient and some utilities are provided to handle serialization of state on both operator implementation and client.

Apache Kafka 0.10 support was also introduced as data source and sink. Kafka 0.10 connector can consume and produce messages with Flink’s internal event time. The Kafka consumer can emit records with Kafka message’s timestamp as event time, but doesn’t emit watermarks to advance ‘time’.

Apache Mesos was also added as a deployment option. Flink also supports DC/OS deployments, and Marathon orchestration platform can be leveraged to run highly available Flink clusters – Marathon runs all Flink components as Mesos tasks across the cluster and, whenever Flink master fails, another one is started. The state (running tasks) is kept and recovered from Zookeeper.

The Table API also got improvements. Window aggregations are supported in three different modes: tumbling windows, sliding windows and session (gap) windows. All modes support windowing on event time and processing time. Tumbling and sliding modes also support windowing on row count. Flink’s Table API and SQL introduced experimental access for POJO field access and expanded the list of built-in functions.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and dont miss out on content that matters to you

BT