Stream Processing & Apache Flink - News and Best Practices

Ververica Platform 2.12 is Released!

Written by Ververica | 22 December 2023

Check out the newest features! The latest update to Ververica Platform has now been released and is generally available (GA) for all users.

This release includes several new, exciting features and other improvements requested by the community. Below is a quick summary. Full details can be found in the 2.12.0 release notes.

IMPORTANT! Read before updating to Ververica Platform 2.12.0

Ververica Platform 2.12.0 supports multiple versions of Apache Flink®, but it's important to note that Apache Flink 1.18 specifically **does not** support Google Cloud Storage checkpointing.

If you currently use Google Cloud Storage (GCS) and plan to upgrade to Ververica Platform 2.12.0, ensure you do not use Apache Flink 1.18. You can continue to use Apache Flink versions 1.17 or 1.16 with Ververica Platform 2.12.0.

Please refer to the Apache Flink issue tracker, FLINK-33793, for more information on this issue. We plan to address this in future 2.12 patch versions, where the Flink 1.18 images will resolve this issue.

What’s new in Ververica Platform 2.12.0

Ververica Platform 2.12 supports the following versions:

  • Apache Flink® 1.18
  • Apache Flink® 1.17.2

See Flink 1.18 release

Experimental support for Java 17 was added in Flink 1.18.0 and is now supported in Ververica Platform 2.12.0.

Ververica Platform 12.2.0 supports Spring Boot 3.1.5.

Features

Pluggable Certificates for Flink Internal / External communications

Users can now start jobs with their own certificates to enable SSL for Flink internal and external communications. With pluggable certificates, users will be able to achieve the following actions:

  • Generate certificates for Ververica Platform and Flink Jobs
  • Provide deployments with custom SSL Flink configuration
  • Provide configurations for Ververica Platform installation with custom keystore

See Pluggable certificates for more information.

Audit Logs

With Release 2.12.0, users can now enjoy audit logs' observability and compliance capabilities. Track activities and events within the Ververica Platform interface. Audit logs will provide a record of events and activities that occur within Ververica Platform for security, compliance, and troubleshooting purposes.

Users can filter and search audit logs based on specific criteria such as timeframe, username, and API tokenId. Additionally, users are able to export audit logs for analysis and reporting purposes in formats such as CSV or JSON. Aside from viewing audit logs directly on the interface, audit logs can be fetched via API.

See Audit logs for more information.

Flink SQL Templating

In an effort to promote consistency across teams and eliminate duplicate efforts, Ververica Platform now offers Flink SQL Templating. Users now have an interface for saving their queries to a library. Team members can now create queries using predefined templates from the queries saved to their query library. This enables users to save and reuse commonly used queries, removing potential inconsistencies when repeating tasks.

Most importantly,  query templating makes sharing queries across different Ververica Platform instances within the same organizations easier. Multiple teams within the same organization can benefit from using standardized defined queries and using or modifying them for their own use cases.

See SQL templating for more information.

Themes

If you are using multiple instances of Ververica Platform, we've introduced themes to differentiate environments effectively. With themes, system administrators can assign a name and color scheme for their different environments.

Theme editor

Example use case

See Themes for more information.

Autopilot

We have enhanced the Autopilot feature by introducing four new configurable properties. These additions enable more precise customization to suit specific use cases:

  • Smoothing factor: Defines the number of time windows that are considered to calculate various task metrics.
  • Minimum Consecutive Seen Changes: Number of consecutive recommendations of a specific type that should occur before VVP Autopilot applies recommendation.
  • Lag Threshold Seconds: Minimal number of seconds needed to catch up with lag given the throughput of the task, above which the recommendations can be applied.
  • Minimum Workload Distribution: Difference (in percentages) between minimal and maximal workload among tasks that VVP Autopilot requires to calculate the recommendation.

See Autopilot for more information.

Enhancements & Improvements

Added support for Apache Derby and Oracle databases JDBC connector

Ververica Platform supports MariaDB, MySQL 5.5.3+, PostgreSQL 8.2+, Oracle, and Apache Derby out of the box. You can add a JDBC driver for any other DBMS by adding a JAR file with a matching JDBC driver to the connector.

Helm charts to set readOnlyRootFilesystem per container​

An immutable root filesystem can prevent malicious binaries from being added to PATH and increase attack costs. An immutable root filesystem prevents applications from writing to their local disk. This is desirable in the event of an intrusion as the attacker cannot tamper with the filesystem or write foreign executables to disk.

Added Protobuf format to the list of packaged formats with VVP

There is a new format for Table API Connectors, which is Protobuf since 1.16.

Bug Fixes

Fixing bugs is an integral part of our release cycle! The following bugs have been squashed:

  • Prevent un-intended copying of deploymentName/deploymentId
  • CPU limit is not reflected on the VVP UI properly
  • Added missing "deletecollection" permission used in Flink HA
  • Resource usage tracking API does generate reports for the current day
  • Inconsistency in VVP Resource Usage tracking

Vulnerability Fixes (outside of Apache Flink®)

And as always, we are continuously stamping out vulnerabilities to ensure your deployments remain secure. The following vulnerabilities are now removed:

Upgrade

As always, we recommend upgrading via Helm using the following commands:

$ helm repo add ververica https://charts.ververica.com
$ helm repo update
$ helm upgrade [RELEASE] ververica/ververica-platform --version 5.8.0 --values custom-values.yaml

Downloads

If you are an existing customer, you already know where to find platform images and archives – see Ververica Platform 2.12.0 Release Notes for all links and complete details of all changes in this release.

If you are new to the platform, you’ll find everything you need to know on the Ververica Platform Downloads page.

More Resources

Learn more about Ververica Platform

Ready to get started? Contact us 
View full post