Wednesday, December 31, 2014

Kafka Round Robin Partitioner

In this blog we will cover how we can write a custom partition in Kafka to distribute data uniformly between a topic partitions.

Let’s consider, we have a TopicA in Kafka which has partitions count 5 and replication factor 3 and we want to distribute data uniformly between all the partitions so that all the partitions contains same data size.
The Kafka uses the default partition mechanism to distribute data between partitions, but in case of default partition mechanism it might be possible that our some partitions size larger than others. Suppose we have inserted 40 GB data into Kafka, then the data size of each partition may look like:
Partition0 à 10 GB
Partition1 à 8 GB
Partition2 à 6 GB
Partition3 à 9 GB
Partition4 à 11 GB

Hence, to distribute a data uniformly, we would need to code a round robin custom partition class.
In Kafka, we can write custom partition class by implementing the Kafka kafka.producer.Partitioner interface. The implemented interface contain a method called partition which has two arguments, one is key that we provide from producer and use to partition the data and second one is number of partitions of a topic. The partition method contains logic to calculate the destination partition and return the target partition number.

import java.util.concurrent.atomic.AtomicInteger;
import org.apache.log4j.Logger;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

/**
 * Round robin partitioner using a simple thread safe AotmicInteger
 */
public class RoundRobinPartitioner implements Partitioner {
    private static final Logger log = Logger.getLogger(RoundRobinPartitioner.class);

    final AtomicInteger counter = new AtomicInteger(0);

    public RoundRobinPartitioner(VerifiableProperties props) {
        log.trace("Instatiated the Round Robin Partitioner class");
    }
    /**
     * Take key as value and return the partition number
     */
    public int partition(Object key, int partitions) {

    int partitionId = counter.incrementAndGet() % partitions;
if (counter.get() > 65536) {
           counter.set(0);
}
return partitionId; 
    }
}

Now, while writing a producer we can set “partitioner.class” property in the instance of kafka.producer.ProducerConfig.

Properties props = new Properties();
props.put("partitioner.class",”com.learnining.kafka.RoundRobinPartitioner");

After using the round robin partition class, the disk usage of each partition will be look like:
Partition0 à 8 GB
Partition1 à 8 GB
Partition2 à 8 GB
Partition3 à 8 GB
Partition4 à 8 GB



27 comments:

Andre Fucs said...

Anjit, would you mind if I use your code as a base for an improvement to Apache NiFi's PutKafka processor?

Asaf Mesika said...

Kafka uses round robin by deault

Saki R said...

this is really too useful and have more ideas from yours. keep sharing many techniques. eagerly waiting for your new blog and useful information. keep doing more.
PHP Training Institute in Chennai

deeksha said...

This is really awesome and it is very much interesting , thanks for sharing those information and it is very well nice as it is really good too.

Best Android Training Institute in Chennai

Xplorant Seo said...

Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..

iOS Training in Chennai

Sanjana sekar said...

This post is really nice and informative. The explanation given is really comprehensive and informative..
PHP Training in Chennai

Anonymous said...

Important note: with such partitioning, ordering guaranties are lost.

Shalini said...

Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.

digital marketing company in india

Abiya Carol said...

Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

Hadoop Training in Chennai

star said...

I would suggest to use counter.compareAndSet() to make it real atomic operation. Or with Java 8 it even may be one-liner


int partition = counter.accumulateAndGet(partiotionsCount, (c, x) -> (c + 1) % x);

john stany said...

This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...

Android App Development Company

john stany said...


Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well
iOS App Development Company
iOS App Development Company

Philips Huges said...


Its very useful to me. Wonderful blog.. Thanks for sharing informative Post.

Installment loans in alabama
Payday loans in alabama
Title loans in alabama
Cash Advances in alabama

louis philip said...

I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.
Fitness SMS
Fitness Text
Salon SMS
Salon Text
Investor Relation SMS
Investor Relation Text
Mobile Marketing Services
mobile marketing companies
Sms API

jessica said...

This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Mobile Marketing Service
Mobile Marketing Companies
Sms API
Texting API

louis philip said...

These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
Texting API
Text message marketing
Digital Mobile Marketing
Mobile Marketing Services
Mobile marketing companies
Fitness SMS

Ezhilarasu L said...

nice article

Java Training Institute in Chennai

nivedhitha reddy said...

very nice article apache kafka training

Govind Raj said...

Hey there, You have done a fantastic job. I’ll definitely digg it and personally suggest to my friends. I am confident they will be benefited from this web site.

PSD to Wordpress
wordpress website development

Ancy merina said...

The feeling was shared. This page is great. You can refer to this page. It also has similar content.
Web development company in bangalore
Outsource magento ecommerce services india
ECommerce Website developers in bangalore

chitra pragya said...

I'm here representing the visitors and readers of your own website say many thanks for many remarkable
angularjs Training in marathahalli

angularjs interview questions and answers

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in chennai

automation anywhere online Training

afiah ahamed said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
Java interview questions and answers

Java training in Chennai | Java training institute in Chennai | Java course in Chennai

Java training in Bangalore | Java training institute in Bangalore | Java course in Bangalore

Java interview questions and answers

sakthi said...

Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information.

rpa training in chennai
rpa training in bangalore
rpa course in bangalore
best rpa training in bangalore
rpa online training

shri yadav said...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
online Python training
python training in chennai

sudhagar raja said...

I am really happy with your blog because your article is very unique and powerful for new reader.
Click here:
selenium training in chennai | selenium course in chennai
selenium training in bangalore | selenium course in bangalore
selenium training in Pune | selenium course in pune | selenium class in pune
selenium training in Pune | selenium course in pune | selenium class in pune
selenium online training | selenium training online | online training on selenium

johnsy sai said...

Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…
Devops Training courses
Devops Training in Bangalore
Best Devops Training in pune
Devops interview questions and answers

dhanya sridhar said...

Well somehow I got to read lots of articles on your blog. It’s amazing how interesting it is for me to visit you very often.
Data Science training in kalyan nagar
Data Science training in OMR | Data science training in chennai
Data Science training in chennai | Best Data science Training in Chennai
Data science training in velachery | Data Science Training in Chennai
Data science training in tambaram | Data Science training in Chennai
Data science training in jaya nagar | Data science Training in Bangalore

Post a Comment