7,907 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
-2
votes
0
answers
49
views
Kafka Connect Refuse to Flink SQL [closed]
I want to process data through flink SQL, here is my code
CREATE TABLE ohlc_source (
screener STRING,
symbol STRING,
`open` DOUBLE,
`high` DOUBLE,
`low` DOUBLE,
`close` DOUBLE,
`volume` ...
0
votes
1
answer
74
views
Flink 1.20: kafka sink job fails with endofinput exception when restoring from savepoint
i encountered one kafka sink exception when started from a savepoint. msg as below:
java.lang.IllegalStateException: Received element after endOfInput: Record @ (undef) : org.apache.flink.table.data....
0
votes
0
answers
61
views
Flink: Could not initialize class org.apache.flink.runtime.util.HadoopUtils (Error when creating catalog with iceberg type in Flink)
I am trying to run a very simple Flink (Java) job that:
Creates an Iceberg JDBC catalog backed by PostgreSQL
Sets the Iceberg warehouse to the Hadoop FileSystem
The job is built successfully with ...
Best practices
0
votes
2
replies
35
views
Which classes must be POJOs/serializable in Apache Flink? When should I use env.registerType()?
I'm mantaining a Flink application and I'm confused about which classes need to be POJOs (serializable) for Flink to reach the State compatibility between different versions of the app.
What I ...
1
vote
1
answer
50
views
Flink: Channels vs. Gates
The Flink docs mention channels and gates.
I am having difficulties inferring what a channel and what a gate is and how they differ. Are these merely logical abstractions or is there also a one-to-one ...
1
vote
2
answers
42
views
Does Flink strictly enforce CPU requests with fine-grained resource management?
Flink allows to define requirements for CPU cores using fine-grained resource management.
I am wondering if this CPU request is strictly enforced or best effort?
Example:
A task manager has 4 CPU ...
-3
votes
1
answer
160
views
Flink Job Manager Direct Buffer Memory gets exhausted when checkpointing enabled
Issue:
Flink application throws Thread 'jobmanager-io-thread-25' produced an uncaught exception. java.lang.OutOfMemoryError: Direct buffer memory and terminates after running for 2-3 days.
No matter ...
Advice
0
votes
0
replies
91
views
Flink Iceberg job loses authentication with REST catalog (Keycloak OAuth2) after short time
I’m running a Flink DataStream job that reads events from a Kafka topic and writes them into an Apache Iceberg table using the REST catalog (Lakekeeper).
Authentication to the REST catalog is ...
0
votes
2
answers
115
views
Is it safe to mutate, emit, and snapshot the same operator-state instance in Apache Flink?
I'm building a single global Topology object in a non-keyed ProcessFunction with parallelism = 1. I keep it as a local mutable object and update it for every input event using topology.apply(GnmiEvent)...
-1
votes
1
answer
47
views
Does Flink throughput decrease proportionally with the number of side outputs?
I have a Flink job with multiple downstream operators I want to route tuples to based on a condition. Side outputs are advertised for this use case in the Flink documentation.
However, when sending ...
0
votes
1
answer
56
views
How to emit keyed records for a compacted topic (SimpleStringSchema ClassCastException)?
I'm upgrading a PyFlink job to 2.0 and want to write to a Kafka compacted topic using the new KafkaSink. The stream produces (key, value) tuples (key is a string, value is a JSON payload). I configure ...
Advice
0
votes
1
replies
60
views
Flink - Async IO Threads required
we are using Flink's AsyncIO function with Futures to make external gRPC calls. Currently, we have set the async capacity to 1, and we are using a blocking stub to make those calls. For each event, we ...
0
votes
1
answer
70
views
Flink state processor api writes new savepoint failed because "All uid's/hashes must be unique"?
Flink Version:1.17.1
There is a KeyedBroadcastProcessFunction in my project like this:
public class MyOperator extends KeyedBroadcastProcessFunction<..> {
private MapState<String, String&...
0
votes
1
answer
49
views
Are Flink's timer service & Collector thread-safe?
Question
Async operation & Future callback was added as the State API was upgraded to v2. Will it be thread-safe to call the Timer service & Collector from that callback?
Example
final var ...
0
votes
1
answer
66
views
Does an uninitialized ValueState occupy memory in checkpoints in Flink?
I'm using a ValueState with TTL and I want to understand the difference (if any) in the checkpointed state size/memory between two scenarios:
First scenario
I create/obtain the ValueState but never ...