Wednesday, December 31, 2014

Kafka Round Robin Partitioner

In this blog we will cover how we can write a custom partition in Kafka to distribute data uniformly between a topic partitions.

Let’s consider, we have a TopicA in Kafka which has partitions count 5 and replication factor 3 and we want to distribute data uniformly between all the partitions so that all the partitions contains same data size.
The Kafka uses the default partition mechanism to distribute data between partitions, but in case of default partition mechanism it might be possible that our some partitions size larger than others. Suppose we have inserted 40 GB data into Kafka, then the data size of each partition may look like:
Partition0 à 10 GB
Partition1 à 8 GB
Partition2 à 6 GB
Partition3 à 9 GB
Partition4 à 11 GB

Hence, to distribute a data uniformly, we would need to code a round robin custom partition class.
In Kafka, we can write custom partition class by implementing the Kafka kafka.producer.Partitioner interface. The implemented interface contain a method called partition which has two arguments, one is key that we provide from producer and use to partition the data and second one is number of partitions of a topic. The partition method contains logic to calculate the destination partition and return the target partition number.

import java.util.concurrent.atomic.AtomicInteger;
import org.apache.log4j.Logger;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

/**
 * Round robin partitioner using a simple thread safe AotmicInteger
 */
public class RoundRobinPartitioner implements Partitioner {
    private static final Logger log = Logger.getLogger(RoundRobinPartitioner.class);

    final AtomicInteger counter = new AtomicInteger(0);

    public RoundRobinPartitioner(VerifiableProperties props) {
        log.trace("Instatiated the Round Robin Partitioner class");
    }
    /**
     * Take key as value and return the partition number
     */
    public int partition(Object key, int partitions) {

    int partitionId = counter.incrementAndGet() % partitions;
if (counter.get() > 65536) {
           counter.set(0);
}
return partitionId; 
    }
}

Now, while writing a producer we can set “partitioner.class” property in the instance of kafka.producer.ProducerConfig.

Properties props = new Properties();
props.put("partitioner.class",”com.learnining.kafka.RoundRobinPartitioner");

After using the round robin partition class, the disk usage of each partition will be look like:
Partition0 à 8 GB
Partition1 à 8 GB
Partition2 à 8 GB
Partition3 à 8 GB
Partition4 à 8 GB



20 comments:

Andre Fucs said...

Anjit, would you mind if I use your code as a base for an improvement to Apache NiFi's PutKafka processor?

Asaf Mesika said...

Kafka uses round robin by deault

Saki R said...

this is really too useful and have more ideas from yours. keep sharing many techniques. eagerly waiting for your new blog and useful information. keep doing more.
PHP Training Institute in Chennai

deeksha said...

This is really awesome and it is very much interesting , thanks for sharing those information and it is very well nice as it is really good too.

Best Android Training Institute in Chennai

Xplorant Seo said...

Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..

iOS Training in Chennai

Sanjana sekar said...

This post is really nice and informative. The explanation given is really comprehensive and informative..
PHP Training in Chennai

Anonymous said...

Important note: with such partitioning, ordering guaranties are lost.

Shalini said...

Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.

digital marketing company in india

Abiya Carol said...

Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

Hadoop Training in Chennai

star said...

I would suggest to use counter.compareAndSet() to make it real atomic operation. Or with Java 8 it even may be one-liner


int partition = counter.accumulateAndGet(partiotionsCount, (c, x) -> (c + 1) % x);

john stany said...

This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...

Android App Development Company

john stany said...


Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well
iOS App Development Company
iOS App Development Company

Philips Huges said...


Its very useful to me. Wonderful blog.. Thanks for sharing informative Post.

Installment loans in alabama
Payday loans in alabama
Title loans in alabama
Cash Advances in alabama

louis philip said...

I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.
Fitness SMS
Fitness Text
Salon SMS
Salon Text
Investor Relation SMS
Investor Relation Text
Mobile Marketing Services
mobile marketing companies
Sms API

jessica said...

This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Mobile Marketing Service
Mobile Marketing Companies
Sms API
Texting API

louis philip said...

These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
Texting API
Text message marketing
Digital Mobile Marketing
Mobile Marketing Services
Mobile marketing companies
Fitness SMS

Ezhilarasu L said...

nice article

Java Training Institute in Chennai

nivedhitha reddy said...

very nice article apache kafka training

Govind Raj said...

Hey there, You have done a fantastic job. I’ll definitely digg it and personally suggest to my friends. I am confident they will be benefited from this web site.

PSD to Wordpress
wordpress website development

Ancy merina said...

The feeling was shared. This page is great. You can refer to this page. It also has similar content.
Web development company in bangalore
Outsource magento ecommerce services india
ECommerce Website developers in bangalore

Post a Comment