Testing Flink SQL Scripts Made Simple for Non-Developers

Developing Flink SQL scripts can be challenging, especially for non-developers like data scientists. Simplifying the development process is essential to ensure correctness and reliability, with testing playing a crucial role. We introduce a Flink SQL Test Runner as a solution to streamline this process.

Key Points:1. Test Runner Architecture: Get a look at the inner workings of the SQL Test Runner which supports both unit and integration tests and generates detailed reports. It is designed to be used behind REST APIs and CI/CD Pipelines.

2. Unit Tests: Learn hands-on how to write a unit test using a SQL script and a Java unit test. In the background, Apache Paimon is used to mock sources and sinks. The unit test files are compiled at runtime, enabling quick execution over the network.

3. Integration Tests: Get to know the concepts used behind a SQL integration test using both the user’s SQL script and a testing SQL script - integrating the ideas of DB to test SQL with SQL. The Test Runner supports both negative and positive testing modes to assert the count of the retrieved results of a stream.

4. Deployment: The Test Runner is packaged as a Docker image, accepting file paths via environment variables, and facilitating integration into CI/CD pipelines behind REST APIs.

Conclusion:The Flink SQL Test Runner significantly enhances the testing process for Flink SQL scripts, supporting robust unit and integration tests. It simplifies the development process for complex Flink projects, meeting the needs of both developers and non-developers.