Embracing the Future with Apache Flink® 2.0

The Apache Flink® community has just unveiled the preview release of Apache Flink 2.0, marking a pivotal moment in the evolution of data processing. This isn't just another update; it's a bold leap toward fulfilling the true promise of a unified batch and stream processing engine. At Ververica, we're thrilled about this release, as it aligns with our vision and the work we've been doing to push the boundaries of real-time data processing.

Modernizing Legacy to Meet Today's Demands

Over the years, Apache Flink has been a powerhouse in stream processing. But let's face it—the data landscape has changed dramatically. The lines between batch and stream processing have blurred, and the need for a seamless, unified engine is more critical than ever. Flink 2.0 addresses this head-on by modernizing legacy components and introducing features that combine batch and stream processing together like never before.

One significant change is the removal of outdated APIs, such as the DataSet API and the Scala versions of the DataStream and DataSet APIs. While some may see this as a hurdle, it's a stride towards simplifying development and maintenance. By encouraging users to adopt the more versatile DataStream API and Table API/SQL, Flink is streamlining the development experience and paving the way for future innovations.

In addition, many users are migrating their code from Scala to Java or SQL. This trend isn't just in Flink; other open-source projects like Apache Kafka are also moving towards Java. Whether this is a welcome change could be debated, but one thing is clear: it allows for a larger community of contributors and makes the technology more accessible.

Apache Paimon Nears Version 1.0: Building the Streaming Lakehouse

Alongside Flink's evolution, Apache Paimon is gearing up for its 1.0 release.  Paimon plays a pivotal role in realizing the streaming lakehouse concept which is a modern, unified architecture that combines the best of data lakes and data warehouses for real-time analytics.

The integration of Flink and Paimon brings significant SQL optimizations, making it easier to build and manage streaming lakehouses. This is a huge milestone. It means we can handle dynamic data updates and queries with varying levels of freshness, catering to a wide range of analytical needs. Whether you need data refreshed to the day or the second, this integration has you covered.

Table formats like Paimon have distinct advantages over alternatives like Iceberg and Hudi, especially regarding real-time data. The enhanced integration in Flink 2.0 means better performance, more efficient resource usage, and the ability to handle larger datasets with ease.

Disaggregated State Storage: Bigger, Faster, More Flexible

One of the most exciting features in Flink 2.0 is the introduction of disaggregated state storage and management. This is a game-changer. Flink decouples compute and storage resources by using Distributed File Systems (DFS) as the primary storage. Here's what that means:

• Scalability: You can now handle massive datasets—including those with hundreds of terabytes—without worrying about local disk constraints.
• Flexibility: Jobs can be rescaled faster and more efficiently, adapting to changing workloads without a hitch.
• Performance: By utilizing asynchronous execution models, resource spikes are reduced, and checkpoint optimization ensures a smoother experience.

At Ververica, we've been championing disaggregated state storage in our engine for some time. Seeing these concepts being adopted in Flink 2.0 is not just validating, it's exhilarating. It means we're all moving in the right direction together, and the future of data processing is looking brighter than ever.

Trends from Flink 1.20 to 2.0: A Rapid Evolution

If we look back at Flink's 1.20 release, it's clear that the groundwork was being laid for these significant changes. Materialized Tables were introduced as an MVP feature, and now in 2.0, they're getting the enhancements needed for production-ready use.

Adaptive Batch Execution is another feature in 1.20 that's advanced in 2.0. By dynamically optimizing logical and physical plans based on execution insights, Flink is unlocking its full potential in batch processing and OLAP workloads.

What's remarkable is how quickly these advancements are happening. The time between the 1.20 and 2.0 releases is shrinking, reflecting the growing momentum and interest in Flink. It's a testament to the community's dedication and the increasing demand for powerful, efficient data processing tools.

The Data Processing Boom: OLAP and Beyond

We're in the midst of an explosion of data processing tools, especially with Online Analytical Processing (OLAP). Businesses are hungry for faster insights from ever-growing datasets, and tools that can handle both real-time and historical data efficiently are in high demand.

Flink's enhancements in OLAP capabilities position it as a leader in this space. The need for data is skyrocketing, but so is the need for efficiency. Modern use cases demand more data and faster processing, without breaking the bank on resources.

The Impact of Large Language Models: More Data, More Demands

Let's talk about the elephant in the room: Large Language Models (LLMs) and other data-hungry applications. These tools are incredible, but they require vast amounts of data to train and operate effectively. In turn, this mandates the need to support hybrid workloads that can handle both batch and streaming data seamlessly.

Flink 2.0 rises to this challenge. Its unified approach and support for hybrid workloads make it an invaluable tool in an era where new applications demand more data and more sophisticated processing capabilities.

If there's one thing we've learned, it's that new tools merely require more data—nothing else.

Conclusion: Embracing the Future with Flink 2.0

Apache Flink 2.0 isn't just a step forward; it's a leap into the future of data processing. By modernizing its legacy components, embracing disaggregated state storage, and enhancing integrations with projects like Apache Paimon, Flink is setting new standards for what's possible.

At Ververica, we see Flink 2.0 as a crystallization of much of the work we've been doing. It's a realization of our efforts to simplify data engineering workflows, enhance scalability, and meet the evolving needs of data-driven applications. We're excited to support Flink 2.0 for our customers and to integrate its powerful features into our offerings.

As the demand for data continues to grow, (and let's be honest, it's certainly not slowing down) the importance of efficient, scalable, and unified data processing can't be overstated. Flink 2.0 is a significant step towards meeting these challenges head-on.

The future is here, and it's streaming in real-time. Let's embrace it together!

Note: This blog post is inspired by the Apache Flink 2.0 preview release notes and reflects the perspectives of Ververica, the original creators of Flink, who continue to contribute to Flinks development.

More Resources

Upgrade Notes

The Flink community tries to ensure that upgrades are as seamless as possible. However, certain changes may require users to make adjustments to certain parts of the program when upgrading to version 2.0. Please refer to the release notes for a comprehensive list of adjustments to make and issues to check during the upgrading process.

Getting Started with Flink

• Get started with Veverica's Unified Streaming Data Platform and unlock the full power of Apache Flink as a managed service.
• Apache Flink is available for download.
• For additional information on the release of 2.0, check out The Apache Flink Site.

Get Involved

The vitality of Flink relies on continued community growth, which would not be possible without each and every contributor to the project. The Flink community welcomes contributions from anyone with a passion for open source, messaging and streaming! Looking for more ways to stay connected with the Flink community? Check out the following resources:

• Ready to start contributing to the project? Start with the Apache Flink Contributors guide.
• Follow @ApacheFlink on Twitter/X , and join the Flink community on Slack.

Thank You!

It would be remiss not to thank the many contributors to the project and development of Apache Flink 2.0, in particular:

• Disaggregated State: Dr. Yuan Mei
• MT: Lincoln Li and Xintong Song
• Apache Paimon: Jingsong Li
• New APIs and async state APIs: Jark Wu and Team 

Ververica offers our congratulations and thanks to all Apache Flink contributors!