Ververica is proud to host the Flink Forward conferences, uniting Apache Flink® and streaming data communities. Each year we nominate a Program Chair to select a broad range of Program Committee members to evaluate the talk submissions from the community for the Flink Forward program. Program Committee members come from various industries, all with deep knowledge of Apache Flink and streaming data technologies.
We are proud to have Erik de Nooij, Engineering lead Streaming Data Analytics at ING Bank, join us as one of our Program Committee members this year at Flink Forward Berlin 2024. Erik has worked on Apache Flink within ING since its inception in 2016, helping it grow toward the de facto standard for streaming at ING worldwide. With a wealth of experience in Apache Flink, Erik has witnessed firsthand the transformative impact of this powerful streaming data framework. I sat down with Erik to discuss his insights into Apache Flink, the future trends shaping the streaming data landscape, along with his tips for those interested in speaking at Flink Forward Berlin 2024!
Actually, there are two significant moments. The first was in 2016 when we started with Apache Flink. Apache Flink put us on a path of building use cases we could actually put into production. Before using Apache Flink that was not the case and the focus was more on learning and experimenting without having a clear path to production.
The second pivotal moment was when we adopted a SQL-first approach, or perhaps more accurately, a Flink SQL-only approach. By then, we had already invested substantial effort in custom code. Despite doubts among some in the organization about the feasibility of transitioning to Flink SQL, we successfully managed it. This achievement was enabled, in part, by our widespread adoption of User-Defined Functions (UDFs). Despite being custom code, UDFs provided a much lighter solution compared to our previous framework, effectively minimizing our IT footprint.
Being part of a community of passionate Flink enthusiasts and having the chance to evaluate abstracts promises a wealth of inspiration and intriguing discussions. With my extensive background in Flink, I believe I can contribute valuable insights and ideas to the mix.
One crucial challenge is ensuring compatibility, particularly as we transition to Flink 2.0. Guidance on how savepoints can be migrated would be immensely beneficial to many Flink users. Possibly, providing savepoints with a more defined structure could be part of the solution.
Maybe make it a bit easier to find people who have a common interest on certain topics. E.g. start with a survey up front and have badges with different colors corresponding to your interests.
Personally, I find talks and workshops most valuable when they address practical real-world problems. Then again, other perspectives on developing and operating streaming platforms are just as valuable.
The fact that it is getting more and more connected to other projects via connectors. Also, Flink is getting more attention on non-Flink conferences as well, which is a good indication of the impact it has.
Watch some videos from earlier conferences and you will see that, even though no single presentation is perfect, they don’t have to be: it’s the effort that counts, and the effort is always appreciated by the audience. Don’t worry about your presentation too much, enjoy the experience and keep it authentic.
The way I see it, the current collaboration within the community is mainly focused on Flink as a product or platform, if you will. Additionally, exploring joint implementations in specific industries could further enrich the conference’s content. For instance, developing real-time apps for scenarios like managing electricity grid fluctuations or providing city tours based on real-time locations could be interesting collaborative projects.
The long track record of a combination of both a strong product and a strong community.
Personally, I believe that streams will be used more and more to make intelligent decisions by continuously feeding (generative) AI models with up-to-date features and/or prompts. These feature updates can be provided by Flink jobs running close to the operational data, potentially following a Data Mesh architecture. These ideas still need to be fleshed out further and events like Flink Forward are instrumental in shaping them.
Let's maintain the tradition of physical gatherings, as they offer invaluable opportunities for collaboration and connection. I hope to meet you all in Berlin this October!
Erik de Nooij
Engineering lead Streaming Data Analytics
ING Bank
Speaking at Flink Forward Berlin 2024 is a great way to connect with hundreds of your peers, and broaden your knowledge on data streaming.
Be quick, Call For Presentation close 11:59 pm (CEST) on May 17th, 2024.