Kafka Replication

Question

We are working on the project where we wish to use Kafka. Based on our learning we have few queries:

Reference URL: https://www.youtube.com/watch?v=BGhlHsFBhLE#t=40m53s

1. In multiple nodes multiple brokers architecture, can consumer read from in-sync follower?

2. Any Kafka documentation links that gives us a walk through around such an architecture?

3. Kafka says that "Producers and Consumers both write to and read from the LEADER replica and Follower replica is a High Availability solution and not meant to be read data from" In this case, how does a same TOPIC be read from multiple brokers? Any documentation / reference links that can help me how this can be achieved?

4. If the concept of "LEADER / FOLLOWER" is at the partition level and topics reside within a partition, then how can a topic be read from multiple brokers (as the replication on other brokers will be a FOLLOWER replica – from which data cannot be read)?

Answer 1

1. No. Consumers always read from leaders.
2. I guess there is bunch of material about Kafka -- just search the Internet. Also check out http://docs.confluent.io/3.0.1/
3. A topic consists of one or more partitions, and partitions are distributed over the brokers. (see https://kafka.apache.org/documentation.html#intro_topics) Thus, for a single topic you can use the (at max) the same number of broker are topic partitions, to read/write date into this topic.
4. It is the other way round (it is not correct that "topics reside within a partition"): a topic contains multiple partitions.

Also check out this blog post about partitions and replication in Kafka: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/

Answer 2

No consumers must read just from partition leader. Replication is just for fault tolerance.

Topic is divided to partitions. Partition is a basic unit of replication and distribution. Each partition has its own leader for read and writes. You can specify layout how those partitions should be distributed across brokers.

Check out following short blog describing basic concepts.