Example applications in Java, Python, Scala and SQL for Amazon Managed Service for Apache Flink (formerly known as Amazon Kinesis Data Analytics), illustrating various aspects of Apache Flink applications, and simple "getting started" base projects.
- Getting Started - DataStream API - Skeleton project for a basic Flink Java application using DataStream API
- Getting Started - Table API & SQL - Basic Flink Java application using Table API & SQL with DataStream API
- Kinesis Connectors - Examples of Flink Kinesis Connector source and sink (standard and EFO)
- Kinesis Source Deaggregation - Handling Kinesis record deaggregation in the Kinesis source
- Kafka Connectors - Examples of Flink Kafka Connector source and sink
- Kafka Config Providers - Examples of using Kafka Config Providers for secure configuration management
- DynamoDB Stream Source - Reading from DynamoDB Streams as a source
- Kinesis Firehose Sink - Writing data to Amazon Kinesis Data Firehose
- SQS Sink - Writing data to Amazon SQS
- Prometheus Sink - Sending metrics to Prometheus
- Flink CDC - Change Data Capture examples using Flink CDC
- JdbcSink - Writes to a relational database executing upsert statements
- Iceberg - Working with Apache Iceberg and Amazon S3 Tables
- S3 Sink - Writing JSON data to Amazon S3
- S3 Avro Sink - Writing Avro format data to Amazon S3
- S3 Avro Source - Reading Avro format data from Amazon S3
- S3 Parquet Sink - Writing Parquet format data to Amazon S3
- S3 Parquet Source - Reading Parquet format data from Amazon S3
- Avro with Glue Schema Registry - Kinesis - Using Avro format with AWS Glue Schema Registry and Kinesis
- Avro with Glue Schema Registry - Kafka - Using Avro format with AWS Glue Schema Registry and Kafka
- Serialization - Serialization of record and state
- Windowing - Time-based window aggregation examples
- Side Outputs - Using side outputs for data routing and filtering
- Async I/O - Asynchronous I/O patterns with retries for external API calls
- Custom Metrics - Creating and publishing custom application metrics
- Fetching credentials from Secrets Manager - Dynamically fetching credentials from AWS Secrets Manager
- Fink Data Generator (JSON) - How to use a Flink application as data generator, for functional and load testing.
- Getting Started - Basic PyFlink application Table API & SQL
- Python Dependencies - Managing Python dependencies in PyFlink applications using
requirements.txt - Packaged Python Dependencies - Managing Python dependencies packaged with the PyFlink application at build time
- Datastream Kafka Connector - Using Kafka connector with PyFlink DataStream API
- Kafka Config Providers - Secure configuration management for Kafka in PyFlink
- S3 Sink - Writing data to Amazon S3 using PyFlink
- Firehose Sink - Writing data to Amazon Kinesis Data Firehose
- Iceberg Sink - Writing data to Apache Iceberg tables
- Windowing - Time-based window aggregation examples with PyFlink/SQL
- User Defined Functions (UDF) - Creating and using custom functions in PyFlink
- Data Generator - Python script for generating sample data to Kinesis Data Streams
- Local Development on Apple Silicon - Setup guide for local development of Flink 1.15 on Apple Silicon Macs (not required with Flink 1.18 or later)
- Getting Started - DataStream API - Skeleton project for a basic Flink Scala application using DataStream API
- Auto Scaling - Custom autoscaler for Amazon Managed Service for Apache Flink
- Scheduled Scaling - Scale applications up and down based on daily time schedules
- Monitoring - Extended CloudWatch Dashboard examples for monitoring applications
- Scripts - Useful shell scripts for interacting with Amazon Managed Service for Apache Flink control plane API
See Contributing Guidelines for more information.
This sample code is made available under the MIT-0 license. See the LICENSE file.