Apache Kafka🔗 i

1Client🔗 i

Clients transparently pool connections to brokers within a cluster. Connections are leased from the pool in order of least in-progress requests. Reconnections are handled transparently, and connection errors bubble up to the caller. Despite being thread-safe, clients may not be shared between consumers.

When a sasl-ctx is provided, it is used to authenticate the connection to the bootstrap host as well as any subsequent connections made to other nodes in the cluster.

When an ssl-ctx is provided, it is used to encrypt all connections.

2Consumer🔗 i

Consumers form consumer groups to subscribe to topics and retrieve records. As the name implies, consumer groups group consumers together so that topic partitions may be spread out across the members of the group.

Consumers are not thread-safe.

The #:reset-strategy argument controls what the consumer’s initial offsets for newly-assigned partitions are going to be. When this value is 'earliest, the consumer will receive records starting from the beginning of each partition. When this value is 'latest, it will receive records starting from the time that it subscribes to each topic.

When a consumer leaves or joins the consumer group, the event will synchronize to a 'rebalance result. In that case, the consumer will automatically re-join the group and discard any un-committed offsets. The associated data is a hash from topic names to hashes of partition ids to offsets. When a rebalance happens, you must take care not to commit any old offsets (i.e. you must issue a new consume-evt before making any calls to consumer-commit ).

When either the timeout passes or new records become available on the broker, the event will synchronize to a 'records result whose associated data will be a vector of records.

More result types may be added in the future.

The timeout-ms argument controls how long the server-side may wait before returning a response. If there are no records in between the time this function is called and when the timeout passes, an empty vector or records will be returned. The other end may not necessarily respect the timeout value, and may return immediately when there are no more records.

Call this function after you have successfully processed a batch of records received from consume-evt . If you forget to call this function, or if the consumer crashes in between calling consume-evt and calling this function, another consumer in the group will eventually receive that same batch again.

3Producer🔗 i

Producers publish data to one or more topics. They batch data internally by topic & partition, and they are thread-safe.

Data is batched internally by topic & partition. Within each batch, the data is compressed according to the #:compression method.

The producer automatically flushes its data every #:flush-interval milliseconds, whenever the total size of all its batches exceeds #:max-batch-bytes, or whenever the total number of records contained in all of its batches exceeds #:max-batch-size, whichever condition is true first.

During a flush, calling produce on a producer blocks until the flush completes.

Typically, you would call this function in a loop to produce a set of data, collect the results then sync them to ensure they’ve been written to the log.