Stream Processing & Apache Flink - News and Best Practices

Apache Paimon 0.6 Released

Written by Giannis Polyzos | 19 December 2023

Introduction

As the original creators of Apache Flink®, Ververica is proud to support technologies that enhance the capabilities of Flink. With this in mind, we’re excited to highlight the release of Apache Paimon Version 0.6! Congratulations to the Apache Software Foundation and the community for this accomplishment. 

Apache Paimon provides a streaming storage layer that enables Flink to perform stream processing directly on the data lake. With this release, Paimon reaches a new level of maturity and continues to raise the bar on streaming storage.

Key Highlights of Apache Paimon 0.6:

  • The Paimon change data capture (CDC) integration now provides support for mainstream data ingestion and integrates with Flink, Kafka, Pulsar, and Mongo. 
  • Paimon extends Flink as a batch read and streaming source. It also supports the Flink time travel query and call procedures to optimize performance. 
  • The updates to the primary key table help maintain high throughput performance, speed up queries with read-optimized tables, perform special aggregations on fields, and improve compaction by over 20%.
  • The introduction of a new metrics system for measuring read and write behavior enhances the built-in metrics to measure operations.
  • New modules now support Paimon Spark, Paimon Trino, and Paimon Presto.

Paimon + Streamhouse 

We believe that Paimon has the potential to extend Ververica’s Enterprise Streaming and Analytics platform to encompass a concept we call Streamhouse. We envision Streamhouse as a solution for bridging the cost/latency gap between data lakehouse batch processing and real-time stream processing. For an overview of Streamhouse, see our recent blog.

Ready to learn more about Ververica and test our capabilities for yourself? Get started with Ververica Cloud. 

Resources
View full post