For more than three years, data Artisans has been working closely with the teams who manage some of the largest stream processing deployments in the world, including Apache Flink® deployments at global companies such as Alibaba, Netflix, Uber, ING, and King. These and many more organizations use Flink as the stream processing platform of choice for large-scale, stateful applications, including real-time analytics, search and content ranking, and fraud and anomaly detection.
By now, Flink has developed into one of the most battle-tested stream processing frameworks for stateful stream processing applications, able to deliver fault tolerant performance at massive scale. This is no small feat, and we along with the rest of the Apache Flink community have much to be proud of.
Why make stream processing with Apache Flink easier for everyone?
Today’s reality is dominated by data, be it geographical, machine-generated, or user-generated data--and data that is produced continuously in the form of data streams.
We believe that the only way for enterprises to gain a meaningful business advantage from this data is through an architecture where services run continuously and can react immediately to important events. Gone are the days of waiting for hours for a computation to finish while competitors are already taking advantage of insights derived from their real-time data.
Writing Apache Flink applications to model complex business logic has become relatively easy, as Flink’s APIs have evolved significantly since the project’s inception. But as we’ve mentioned before, we know that there’s still much work to be done to make stateful stream processing as easy as possible for anyone to deploy.
Working closely with a wide variety of Flink production users has convinced us that operationalizing and supporting 24/7 stateful stream processing applications is a process that can be made easier and is something that’s not addressed by existing operations technologies, which are generally designed for stateless applications.
And so today, we’re excited to announce dA Platform, a stream processing platform that includes open source Apache Flink and dA Application Manager, a new component from data Artisans that streamlines the workflows of companies who are operationalizing stateful stream processing in production. dA Platform makes it easier for anyone to build, deploy, and maintain Apache Flink stream processing applications.
dA Platform takes the very best of what we’ve learned when working with innovative organizations who have built stream processing platforms powered by Apache Flink and makes the same stream processing that powers Alibaba, Uber, Netflix, and more easily accessible to enterprises of all shapes and sizes.
What’s inside dA Platform?
dA Platform is comprised of three core components:
1) Open source Apache Flink
2) dA Application Manager
3) A reference architecture and optional components for logging, metrics, and more
.webp?width=579&height=242&name=application_manager_opt4@3x-1%20(1).webp)
Application Manager introduces the concept of--you guessed it--an application, which is how most production Flink users think of their long-running jobs. Instead of managing Flink jobs manually, Application Manager coordinates starting, savepointing and stopping Flink jobs behind the scenes. Common tasks such as upgrading to a newer application version or restoring to an older savepoint is just one click away.
Application Manager provides a holistic view of a company’s streaming applications at a high level while managing Flink jobs, savepoints, and other details transparently in the background. Users can monitor and manage their applications via a web UI or a REST API.
In this way, Application Manager makes stream processing application lifecycle and state management as easy as possible. Combined with cloud-ready components for integrated resource management and logging and metrics, dA Platform turns Flink into an all-in-one platform for stream processing.
By combining all the application lifecycle and metadata of streaming applications, Application Manager makes application upgrades, A/B tests with different Flink jobs, and CI/CD integration simple to implement. Application Manager stores all of the required metadata, such as savepoint locations or previous job versions, to facilitate these operations.
Common tasks such as debugging application issues or performance regressions become much easier, too, because Application Manager knows which job version has been running at any point in time. Answering questions such as, “Which code version was running 5 days ago?” or “Where can I find the metrics from when we deployed an improved implementation?” now becomes very straightforward.
We’ll be demoing dA Platform tomorrow at Flink Forward Berlin, and we’ll share a recording of the demo as soon as it’s ready.
When will dA Platform be available?
Starting today, an Early Access version of dA Platform is available via the Early Access program and will be generally available in Q1/2018. Sign up here to be eligible for the Early Access program or to be informed when the GA release is available.
What is the Early Access program for dA Platform?
Interested users can sign up to participate in an early access program working closely with the data Artisans engineering team. We’ll be able to work with a limited number of users to ensure a quality program and an experience that is mutually beneficial. After signing up, you’ll receive an email asking for a few details about your environment, and from there we’ll take the next steps.
Making it easier to run your business in real time
This product is one more step towards our vision at data Artisans: enabling the enterprise to run in real-time. With the second release of the dA Platform, we seek to bring the benefit of modern stream processing that is powering the world’s most innovative companies to organizations of all shapes and sizes. Apache Flink already delivers fast insights, and dA Platform will enable faster time to market, and therefore faster bottom-line impact, for Flink-powered stateful stream processing applications.
If you’re interested in the Early Access program, you can sign up now.
