Apache Kafka 101: Consumers (2023)

How do we scale(on what criteria) consumers in consumer group(considering we have 3 partition with 1 consumer to start with )? is it message count in topic and auto scaler decides to add in new consumer service ?

Complicated!!!

Where can I get hands on api examples

Wow - a totally perfect video, clearing most of my doubts in less than 7 minutes!
Thank You!

Can I have the consumer act as a producer(publisher) as well? I have a user case where a consumer (C1) polls a topic and pulls messages. after processing the message and performing a commit, it needs to notify another process to carry on remaining work. Given this use case is it a valid design for Consumer (C1) to publish a message to a different topic? i.e. C1 is also acting as a producer

We run Kafka consumer on Kubernetes. For one use case we use pod-name as the groupId to filter out specific events. Is having 1000s of groupId for a topic an issue? will the Kafka cluster cleanup unused groupid safely in the background (metadata and offsets)?

Very nice

Hi I have a doubt in this partition

Number of consumers is restricted to number of partitions?

Great videos!

where i can bought that t-shirt! it's so cool!

Each partition ensures the order of the messages contained therein. The producer can target messages to be produced onto a specific partition by the message key. Consumers can target a specific partition by specifying the key of the messages they're interested in. If the message order doesn't matter as in when transactions are idempotent(i.e. Order has shipped) then you can produce those messages in a round robin fashion to multiple partitions and have each partition assigned a consumer where each consumer belongs to the same consumer group, consume those messages at scale. 10 partitions then define 10 consumers within the same consumer group so that messages are consumed at scale of 10. Got it.

The MOST important thing to know is if msg Order IS important for message consumption(i.e. CRUD event notification). If it is then you can only have 1 partition and have multiple consumer groups each with 1 consumer reading from that partition. If msg order IS NOT important then you can scale.

Tim has explained kafka `simplified` exceptionally well.

Great video! Any tips of:
1) how to define and monitor a good number of partitions?
2) how to handle consumers failures? For example, how to not get stucked because of a miss-formed message?