Highlights from Flink Forward Berlin 2024

Flink Forward Berlin 2024 is a wrap, and Ververica is proud to have hosted the 10-year anniversary of Apache Flink® with the community! Flink Forward is the only conference dedicated entirely to Apache Flink and data streaming, and this year it was exciting to welcome speakers, sessions, and attendees from companies like Apple, Alibaba, Intesa Sanpaolo, Mercedes-Benz, Pinterest, Toyota Motor Europe, Uber, and many more!

Read on for a recap of the 4-day event, and thanks to photographer Jan Michalko for the images provided in this blog.

Welcome banner at Flink Forward Berlin 2024, organized by Ververica at the EUREF Campus in Berlin, Germany

Flink Forward at a glance

• 460+ registrations and attendees from Apple, Airbus, IBM, Workday, Bloomberg, American Express, and more!
• 65+ speakers from companies like AWS, Apple, Pinterest, Uniper, Stripe, and Redpanda
• A journey through the Past, Present and Future of Apache Flink
• AI Expert Panel
• 40 breakout sessions spanning 3 tracks including: tech deep dives, use cases, and FlinkCDC deep dives
• PMC Panel Discussion
• Expo Hall filled with Sponsor booths including Datorios, Decodable, Evoura, WarpStream, Cloudera, and StreamNative
• 3 sold-out Apache Flink training sessions attended by 150+ learners
• 4 sponsor lunch demos from Gold Sponsors
• A Partner booth featuring Conductor, Hivemind, Redpanda, and Steadforce
• 3 surprise announcements
• 1 inflatable squirrel (nickname: Patch)
• Numerous talented karaoke Flink Fest participants

Apache Flink: Past, Present, and Future

As the organizers of Flink Forward, and the original creators of Apache Flink, Ververica kicked off the opening session with a journey through the Past, Present and Future of Apache Flink.

Past

Starting with Stephan Ewan, Co-creator of Flink, Apache Flink PMC member, and Founder of Restate, and Feng Wang, Senior Director of Engineering at Alibaba and Apache Flink Committer, we explored a brief history of how Flink became what we recognize today, from the start of the initial Stratosphere Project at the Technical University of Berlin, through the merge of Flink and Blink at Alibaba, and into the future of unified streaming and batch processing capabilities.

Present

Joining the stage next, Siddhartha Choudhury from Booking.com and Sudha Ramraj from Uniper shared how their companies are using Flink and Ververica to serve their data streaming needs.

Each then described the specific use cases they are solving with this technology and the positive impact Flink has had on internal team enablement.

Next, Mark Pybus (Evoura), Erik Schmiegelow (Hivemind Technologies), Filip Yonov (Aiven), and Sijie Guo (StreamNative), joined Ververica's Field CTO, Ben Gamble and Partner Alliance Manager, Chris Horsnell to discuss the importance of Partners and when users should consider introducing Flink into their organizations. The Partner panel discussed the massive requirements related to data size and speed that businesses face, and how real-time data is still not widely adopted by data engineers and organizations that are more comfortable with batch processing and ETL. They shared their insights on how businesses can successfully adopt real-time data processing with Flink to solve complicated use cases.

Future

The final part of the Opening Session featured two major announcements and an exciting introduction.

First, Jark Wu, Head of Flink SQL at Alibaba Cloud,  Apache Flink PMC Member and Committer introduced Fluss (Flink Unified Streaming Storage), which promises streaming storage for next-gen data analytics. Fluss is engineered to offer a high-performance, scalable, and fully integrated solution for real-time data processing with Apache Flink, further driving towards a complete unified batch and streaming data platform.

Learn more about Fluss in the new blog.

Next up, Igor Kersic, Head of Product at Ververica, took the stage to introduce Ververica’s newest deployment option: Bring Your Own Cloud. This deployment provides a managed experience that uses your existing cloud resources to leverage the flexibility and scalability of Ververica’s Unified Streaming Data Platform while maintaining full control over your cloud infrastructure. This new deployment option joins Self-Managed and Self-Service options, and provides an effective solution that aligns with zero trust principles. With BYOC, you retain absolute data sovereignty, as your data is stored in object storage and hosted on a cloud environment fully controlled by your organization, ensuring full oversight and security.

Ready to learn more? Check out the blog, or contact Ververica to get started.

Surprise Announcement

The final part of the Opening Session featured two major announcements and an exciting introduction.

There were plenty of surprises in store at Flink Forward Berlin, but one of the most exciting was the official announcement of the release of Apache Flink 2.0! Feng Wong returned to the stage alongside Ververica’s CEO, Alexander Walden, and presented what Flink users can expect from 2.0.

One major new feature is the decoupling of compute and storage resources by using a Distributed File Systems (DFS) as the primary storage. Users will benefit in multiple ways

• Scalability: You can now handle massive datasets—including those with hundreds of terabytes—without worrying about local disk constraints.
• Flexibility: Jobs can be rescaled faster and more efficiently, adapting to changing workloads without a hitch.
• Performance: By utilizing asynchronous execution models, resource spikes are reduced, and checkpoint optimization ensures a smoother experience.

Another very exciting introduction are Materialized tables, designed to simplify the development of data processing pipelines by allowing uniform SQL operations and automatic data freshness management. Now, users can define batch and streaming transformations to data in the same way, accelerate ETL pipeline development, and manage task scheduling automatically.

In short, by modernizing its legacy components, embracing disaggregated state storage, and enhancing integrations with projects like Apache Paimon, Flink 2.0 is setting new standards for what's possible. We are confident that Flink 2.0 will provide more efficient, scalable, and unified data processing for a growing number of use cases and businesses. Ready to learn more? Read the blog: “Embracing the Future of Apache Flink 2.0”.

Expert Apache Flink Training Courses

Prior to the conference, learners from around the globe gathered to participate in sold out, two-day, in-person Apache Flink Training Courses, offered by Ververica Academy.

Masterclass

Designed for those already familiar with Apache Flink, the Masterclass guided attendees in unlocking the full potential of real-time stream processing. Four thorough half-day sessions, each focused on a specific topic and opened by an Apache Flink PMC Member, followed by hands-on workshops led by Ververica’s Flink experts. Topics included:

• Apache Flink: From Data to Intelligence
• Bridging Data Silos with Flink CDC
• Flink SQL Origins and the Future: Insights from the Original Creator
• Evolving to a Streaming Lakehouse with Apache Flink

Bootcamp

Led by Ken Krugler, this intensive training program coached Apache Flink users in core Flink concepts and advanced data processing techniques. By translating complex Flink concepts into practical exercises based on real-world scenarios and leveraging Ververica Cloud services, participants left the course empowered to tackle their toughest data challenges while gaining a deep understanding of Flink and how to optimize the scalability and efficiency of their cloud-based solutions.

This program is not just about learning; it’s about mastering Apache Flink and leading the future of data processing.

Ready to continue learning Flink? Sign up for the free online Apache Flink classes currently offered.

Breakout Sessions

Speakers covered a multitude of topics on streaming at scale, handling performance and troubleshooting, building real-life use cases, and included a track dedicated entirely to Flink CDC talks. Top talks included:

• Apache Paimon + Flink: Build Streaming Pipeline on Lakehouse (Jingsong Li at Alibaba)
• Automate Apache Flink Tuning For Highly Elastic Scaling (Ioannis Stavrakantonakis with Ververica)
• Building a Streaming-First Platform: Tools and Lessons from Our Flink Migration Journey (Mohsin Niazi at Marshall Wace)

Breakout sessions will be available on demand soon! Subscribe to Flink Forward communications to ensure you are among the first to be notified when the recorded content is ready for viewing.

AI Expert Panel

At the conclusion of Day One, Mike Gualtieri (Forrester VP & Principal Analyst), Ben Gamble (Ververica Field CTO), Gunnar Morling (Decodable Software Engineer), and David Anderson (Apache Flink Committer and Confluent Software Practice Lead) at #FlinkForward Berlin appeared on the mainstage to discuss what #AI means for the streaming data industry.

During this moderated panel session, they explored whether AI is all hype, how AI is shaping data streaming projects now and into the future, what innovations will redefine how we access and process information, and how AI is set to transform the way we use data in real-time.

PMC Panel

To start Day Two, an informal, moderated panel session featuring Apache Flink PMC experts shared their Flink experiences and discussed the past, present, and future of Apache Flink. Apache Flink PMC Members and Committers including Dr. Yuan Mei (Director of Engineering at Alibaba), Jark Wu (Head of Flink SQL at Alibaba Cloud), Xintong Song (starter and promoter of Flink 2.0), Leonard Xu  (Flink CDC Lead), and Jingsong Li (research and development of streaming computing within Alibaba), Gyula Fora (Software Engineer at Apple), Maximilian Michels (Software Engineer at Apple) and Jing Ge (Head of Engineering at Ververica) dove into these questions and more, offering insights and answering questions about Flink during this interactive session.

Closing Remarks from Flink Forward Berlin

At the conclusion of the 2-day event, Ben Gamble joined the stage once more to briefly present Ververica’s Unified Streaming Data Platform. Powered by VERA, the cloud-native engine revolutionizing Apache Flink, this Unified Streaming Data Platform allows you to derive insights, make decisions and take action with data from any source. Businesses can connect, process, analyze and govern continuous streaming of data in real-time, via the deployment method of your choice. 100% compatible with open source Flink, this solution offers an efficient, secure, highly elastic and scalable way to start using Flink for real-time and batch processing jobs. Built to democratize stream processing, we recommend reading the new VERA Whitepaper and contacting Verervica to learn more.

Lastly, Alex Walden, Ververica CEO, joined the stage one last time, ready to invite all participants to the private after-party, Flink Fest 2024.

Networking and Community

There’s nothing quite like finally connecting in-person and learning from other data streaming thought leaders, and Flink Forward Berlin had plenty of opportunities to network!

Don’t just take our word for it, here’s just a little of what attendees had to say about the event:

“It’s very valuable to see real-world examples, and patterns to solve these challenges. I’m looking forward to follow up and explore the code examples provided in this talk. (Referencing JinYun Soo’s talk ‘State’ The Obvious: Using Apache Flink’s State Processor Api To Deal With Nutty Issues) -Hartmut A. (eu-LISA)

“Checking in at Flink Forward, lots of interesting talks” - Simon Dahlbacka (Fellowmind)

“Great to chat with Stephan Ewen at Flink Forward. Stephan came by to talk about the data streaming world and in particular the importance of Apache Flink on the future of data.” -(Evoura)

“I had an incredible time at Flink Forward 2024 in Berlin. It was an amazing opportunity to connect with industry experts, explore the latest developments in streaming technology, and support hands-on learning sessions.” - Naci Simsek (Vererica)

“Flink 2.0 sees a fundamental re-architecture of the storage layer in preparation for a Cloud-Native Future. Great presentation at Flink Forward here in Berlin from Yuan Mei of Alibaba Group introducing ForSt DB, a new distributed storage engine for Flink 2.0.” -Alexander Dean (Snowplow) 

“…Fluss, a real contender to Kafka that goes beyond just storage—this full-fledged streaming platform. What makes it stand out? Apache Arrow powers it for super-fast data streaming. Stores cold data in open formats like Apache Iceberg and Paimon, ](and]) has Kafka API support for seamless integration with existing systems.” - GetinData

“Connecting with so many passionate voices in real-time data innovation was truly inspiring!” -Stav Elkayam (Datorios)

Additional networking occurred at the end of Day One during the Sponsor Cocktail hour in the Expo Hall. Here, attendees gathered to connect and enjoy the hosted bar. Thank you to all of the Sponsors for their participation and support!

Flink Fest

To close out Flink Forward Berlin 2024 and celebrate Apache Flink’s 10th Anniversary in style, Ververica rented out one of the top club venues in Berlin for a not-to-be-missed private after party! Complete with food, fun, networking, karaoke, open bar and exclusive swag, the party was an excellent way to bring together our widely dispersed Flink Community for one final celebration. Thanks to ALL the karaoke participants: we have some real talent among our data streaming enthusiasts!

Up Next: Flink Forward Heads to Shanghai and Jakarta!

Don’t miss the next Flink Forward events, first in Shanghai November 29-30th, followed by Jakarta Dec 5th, 2024. Get all the upcoming details and register here.

Resources and Next Steps

The event team recorded the Opening Session and Breakout Sessions, and the on-demand videos will be available soon. Subscribe to be among the first notified once the videos go live.

If you’d like to hear more about Flink Forward, check out these resources:

• AI Journal: Ververica upholds data sovereignty with its new Bring Your Own Cloud (BYOC) deployment for data Read the article

• The Ravit Show: Origin of Apache Flink and it's impact on the market
  Watch the video

• Subscribe to Flink Forward Communications

On behalf of the entire event team here at Ververica, we extend our thanks to all the attendees, sponsors, Program Committee members and Apache Flink Community for helping make this an event to remember.

Until next year, keep Flinking!