Identify and find specific message in Kafka topic

Question

I am trying to understand is it possible to find specific message in Kafka topic. Kafka message KEY was considered for means of solving the problem.

But I suspect, that key allow to identify only the partition inside topic, not the specific message inside partition.

Is there any way to identify specific message in Kafka partition, and finding this message for further using (for example, reading this specific message)?

It is about stream processing.

For example:

Partition include 4 messages: Message 1, Message 2, Message 3, Message 4. And I want to find only Message 1, relying on the fact that this Message has (possible!) the unique id. Like id in database.

• Michael Heil: You may find this helpful.

Answer 1

Kafka topics can be seen as an append-only log and not like a key/value store.

Therefore, concepts like a primary key in a relational database that you can look up do not apply here. However, Kafka has a rather technical unique identifier which is called the "offset" which is a incremental counter. A message within a Kafka is uniquely defined by its topic name, topic partition and offset.

So unless you are aware of these three values, you can not look up a specific message.

The alternative would be to consume the entire topic from beginning and implements a filter on your requirement. To avoid reading the topic from beginning you could also use the offsetsForTimes API which helps you identify the offsets based on a timestamp.

• Robin Moffatt: You can also use Kafka Streams or ksqlDB to consume the data and build K/V lookup stores within it.