Wednesday, December 31, 2014

Kafka Round Robin Partitioner

In this blog we will cover how we can write a custom partition in Kafka to distribute data uniformly between a topic partitions.

Let’s consider, we have a TopicA in Kafka which has partitions count 5 and replication factor 3 and we want to distribute data uniformly between all the partitions so that all the partitions contains same data size.
The Kafka uses the default partition mechanism to distribute data between partitions, but in case of default partition mechanism it might be possible that our some partitions size larger than others. Suppose we have inserted 40 GB data into Kafka, then the data size of each partition may look like:
Partition0 à 10 GB
Partition1 à 8 GB
Partition2 à 6 GB
Partition3 à 9 GB
Partition4 à 11 GB

Hence, to distribute a data uniformly, we would need to code a round robin custom partition class.
In Kafka, we can write custom partition class by implementing the Kafka kafka.producer.Partitioner interface. The implemented interface contain a method called partition which has two arguments, one is key that we provide from producer and use to partition the data and second one is number of partitions of a topic. The partition method contains logic to calculate the destination partition and return the target partition number.

import java.util.concurrent.atomic.AtomicInteger;
import org.apache.log4j.Logger;
import kafka.producer.Partitioner;
import kafka.utils.VerifiableProperties;

/**
 * Round robin partitioner using a simple thread safe AotmicInteger
 */
public class RoundRobinPartitioner implements Partitioner {
    private static final Logger log = Logger.getLogger(RoundRobinPartitioner.class);

    final AtomicInteger counter = new AtomicInteger(0);

    public RoundRobinPartitioner(VerifiableProperties props) {
        log.trace("Instatiated the Round Robin Partitioner class");
    }
    /**
     * Take key as value and return the partition number
     */
    public int partition(Object key, int partitions) {

    int partitionId = counter.incrementAndGet() % partitions;
if (counter.get() > 65536) {
           counter.set(0);
}
return partitionId; 
    }
}

Now, while writing a producer we can set “partitioner.class” property in the instance of kafka.producer.ProducerConfig.

Properties props = new Properties();
props.put("partitioner.class",”com.learnining.kafka.RoundRobinPartitioner");

After using the round robin partition class, the disk usage of each partition will be look like:
Partition0 à 8 GB
Partition1 à 8 GB
Partition2 à 8 GB
Partition3 à 8 GB
Partition4 à 8 GB



41 comments:

Andre Fucs said...

Anjit, would you mind if I use your code as a base for an improvement to Apache NiFi's PutKafka processor?

Asaf Mesika said...

Kafka uses round robin by deault

Saki R said...

this is really too useful and have more ideas from yours. keep sharing many techniques. eagerly waiting for your new blog and useful information. keep doing more.
PHP Training Institute in Chennai

deeksha said...

This is really awesome and it is very much interesting , thanks for sharing those information and it is very well nice as it is really good too.

Best Android Training Institute in Chennai

Xplorant Seo said...

Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..

iOS Training in Chennai

Sanjana sekar said...

This post is really nice and informative. The explanation given is really comprehensive and informative..
PHP Training in Chennai

Anonymous said...

Important note: with such partitioning, ordering guaranties are lost.

Shalini said...

Great site for these post and i am seeing the most of contents have useful for my Carrier.Thanks to such a useful information.Any information are commands like to share him.

digital marketing company in india

Abiya Carol said...

Very nice post here thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

Hadoop Training in Chennai

star said...

I would suggest to use counter.compareAndSet() to make it real atomic operation. Or with Java 8 it even may be one-liner


int partition = counter.accumulateAndGet(partiotionsCount, (c, x) -> (c + 1) % x);

john stany said...

This article is very much helpful and i hope this will be an useful information for the needed one. Keep on updating these kinds of informative things...

Android App Development Company

john stany said...


Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well
iOS App Development Company
iOS App Development Company

Philips Huges said...


Its very useful to me. Wonderful blog.. Thanks for sharing informative Post.

Installment loans in alabama
Payday loans in alabama
Title loans in alabama
Cash Advances in alabama

louis philip said...

I am expecting more interesting topics from you. And this was nice content and definitely it will be useful for many people.
Fitness SMS
Fitness Text
Salon SMS
Salon Text
Investor Relation SMS
Investor Relation Text
Mobile Marketing Services
mobile marketing companies
Sms API

jessica said...

This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
Mobile Marketing Service
Mobile Marketing Companies
Sms API
Texting API

louis philip said...

These ways are very simple and very much useful, as a beginner level these helped me a lot thanks fore sharing these kinds of useful and knowledgeable information.
Texting API
Text message marketing
Digital Mobile Marketing
Mobile Marketing Services
Mobile marketing companies
Fitness SMS

Ezhilarasu L said...

nice article

Java Training Institute in Chennai

nivedhitha reddy said...

very nice article apache kafka training

Govind Raj said...

Hey there, You have done a fantastic job. I’ll definitely digg it and personally suggest to my friends. I am confident they will be benefited from this web site.

PSD to Wordpress
wordpress website development

Ancy merina said...

The feeling was shared. This page is great. You can refer to this page. It also has similar content.
Web development company in bangalore
Outsource magento ecommerce services india
ECommerce Website developers in bangalore

chitra pragya said...

I'm here representing the visitors and readers of your own website say many thanks for many remarkable
angularjs Training in marathahalli

angularjs interview questions and answers

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in chennai

automation anywhere online Training

afiah ahamed said...

Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.
Java interview questions and answers

Java training in Chennai | Java training institute in Chennai | Java course in Chennai

Java training in Bangalore | Java training institute in Bangalore | Java course in Bangalore

Java interview questions and answers

sakthi said...

Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information.

rpa training in chennai
rpa training in bangalore
rpa course in bangalore
best rpa training in bangalore
rpa online training

shri yadav said...

This is a nice article here with some useful tips for those who are not used-to comment that frequently. Thanks for this helpful information I agree with all points you have given to us. I will follow all of them.
online Python training
python training in chennai

sudhagar raja said...

I am really happy with your blog because your article is very unique and powerful for new reader.
Click here:
selenium training in chennai | selenium course in chennai
selenium training in bangalore | selenium course in bangalore
selenium training in Pune | selenium course in pune | selenium class in pune
selenium training in Pune | selenium course in pune | selenium class in pune
selenium online training | selenium training online | online training on selenium

johnsy sai said...

Nice tutorial. Thanks for sharing the valuable information. it’s really helpful. Who want to learn this blog most helpful. Keep sharing on updated tutorials…
Devops Training courses
Devops Training in Bangalore
Best Devops Training in pune
Devops interview questions and answers

dhanya sridhar said...

Well somehow I got to read lots of articles on your blog. It’s amazing how interesting it is for me to visit you very often.
Data Science training in kalyan nagar
Data Science training in OMR | Data science training in chennai
Data Science training in chennai | Best Data science Training in Chennai
Data science training in velachery | Data Science Training in Chennai
Data science training in tambaram | Data Science training in Chennai
Data science training in jaya nagar | Data science Training in Bangalore

amsa leka said...

Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays. Well written article Thank You for Sharing with Us pmp training Chennai | pmp training centers in Chennai | pmp training institutes in Chennai | pmp training and certification in Chennai | pmp training in velachery

Saro said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
rpa training in chennai
rpa training in bangalore
rpa course in bangalore
best rpa training in bangalore
rpa online training

sandy star said...

Hmm, it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up what I had written and say, I’m thoroughly enjoying your blog. I as well as an aspiring blog writer, but I’m still new to the whole thing. Do you have any recommendations for newbie blog writers? I’d appreciate it.
Advanced AWS Interview Questions And Answers, Top 250+AWS Interviews Questions and Answers 2018
Advanced AWS Interview questions and answers | Top 110 AWS Interview Question and Answers
Advanced AWS Training in Bangalore | Best Amazon Web Services Training in Bangalore

DJ PRASATH said...

Thanks for your post. This is excellent information. The list of your blogs is very helpful for those who want to learn, It is amazing!!! You have been helping many application.
best selenium training in chennai | best selenium training institute in chennai selenium training in chennai | best selenium training in chennai | selenium training in Velachery | selenium training in chennai omr | quora selenium training in chennai | selenium testing course fees | java and selenium training in chennai | best selenium training institute in chennai | best selenium training center in chennai

DJ PRASATH said...

Thanks for your post. This is excellent information. The list of your blogs is very helpful for those who want to learn, It is amazing!!! You have been helping many application.
best selenium training in chennai | best selenium training institute in chennai selenium training in chennai | best selenium training in chennai | selenium training in Velachery | selenium training in chennai omr | quora selenium training in chennai | selenium testing course fees | java and selenium training in chennai | best selenium training institute in chennai | best selenium training center in chennai

amsa leka said...

Nice blog..! I really loved reading through this article. Thanks for sharing such an amazing post with us and keep blogging...Well written article Thank You for Sharing with Us pmp training institute in chennai | pmp training class in chennai | pmp training near me | pmp training courses online | pmp training fee

saranyaregan said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
best rpa training in bangalore
rpa training in bangalore | rpa course in bangalore
RPA training in bangalore
rpa training in chennai
rpa online training

Inigo joseph said...

I simply want to give you a huge thumbs up for the great info you have got here on this post.
Python training in bangalore
Python course in pune
Python training in bangalore

sarah blossom1234567 said...

A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.
AWS Online Training
AWS Training in Bangalore

OLIVIA H said...

I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
apple iphone service center in chennai | Mac book pro service center in chennai | ipod service center in chennai | apple ipad service center in chennai

Blogger said...

This is most informative and also this post most user friendly and super navigation to all posts... Thank you so much for giving this information to me..
Devops Training in Chennai | Devops Training Institute in Chennai

dịch thuật miền trung said...

Thank for Baì viết hữu ích. Mình cũng muốn giới thiệu về một Công ty dịch thuật uy tín - Công ty CP dịch thuật miền trung - MIDtrans địa chỉ 02 Hoàng Diệu, TP Đồng Hới, tỉnh Quảng Bình có Giấy phép kinh doanh số 3101023866 cấp ngày 9/12/2016 là đơn vị chuyên cung cấp dịch vụ dịch thuật, phiên dịch dành các cá nhân. Hệ thống thương hiệu và các Công ty dịch thuật con trực thuộc: văn phòng dịch thuật sài gòn 247 địa chỉ 47 Điện Biên Phủ, Phường Đakao, Quận 1 TP HCM, dịch thuật phan thiết, bình thuận : địa chỉ 100 , Lê lợi, TX Phan Thiết là nhà cung ứng dịch vụ dịch thuật uy tín hàng đầu tại Bình Thuận vietnamese translate : dịch vụ dịch thuật cho người nước ngoài có nhu cầu, giao diện tiếng Anh dễ sử dụng; dịch thuật công chứng quận 11 (mười một) : nhà cung ứng dịch vụ dịch vụ dịch thuật phiên dịch hàng đầu tại Quận 11 (mười một), TP HCM; dịch thuật đà nẵng midtrans : Địa chỉ 54 Đinh Tiên Hoàng, Quận Hải Châu, TP Đà Nẵng chuyên cung cấp dịch vụ dịch thuật công chứng, dịch thuật chuyên ngành tại Đà Nẵng; dịch thuật hà nội midtrans : địa chỉ 101 Láng Hạ, Đống Đa, Hà Nội là nhà cung ứng dịch vụ biên dịch, phiên dịch chuyên nghiệp tại địa bàn Hà Nội. Chúng tôi chuyên cung cấp các dịch vụ biên dịch và phiên dịch, dịch thuật công chứng chất lượng cao hơn 50 ngôn ngữ khác nhau như tiếng Anh, Nhật, Hàn, Trung, Pháp, Đức, Nga, Tây Ban Nha, Bồ Đào Nha, Ý, Ba Lan, Phần Lan, Thái Lan, Hà Lan, Rumani, Lào, Campuchia, Philippin, Indonesia, La Tinh, Thụy Điển, Malaysia, Thổ Nhĩ Kỳ..vv... Dịch thuật MIDtrans tự hào với đội ngũ lãnh đạo với niềm đam mê, khát khao vươn tầm cao trong lĩnh vực dịch thuật, đội ngũ nhân sự cống hiến và luôn sẵn sàng cháy hết mình. Chúng tôi phục vụ từ sự tậm tâm và cố gắng từ trái tim những người dịch giả.Tự hào là công ty cung cấp dịch thuật chuyên ngành hàng đầu với các đối tác lớn tại Việt nam trong các chuyên ngành hẹp như: y dược (bao gồm bệnh lý), xây dựng (kiến trúc), hóa chất, thủy nhiệt điện, ngân hàng, tài chính, kế toán. Các dự án đã triển khai của Công ty dịch thuật chuyên nghiệp MIDtrans đều được Khách hàng đánh giá cao và đạt được sự tín nhiệm về chất lượng biên phiên dịch đặc biệt đối với dịch hồ sơ thầu , dịch thuật tài liệu tài chính ngân hàng, dịch thuật tài liệu y khoa đa ngữ chuyên sâu. Đó là kết quả của một hệ thống quản lý chất lượng dịch thuật chuyên nghiệp, những tâm huyết và kinh nghiệm biên phiên dịch nhiều năm của đội ngũ dịch giả của chúng tôi. Hotline: 0947688883. email: info@dichthuatmientrung.com.vn . Các bạn ghé thăm site ủng hộ nhé. Cám ơn nhiều

rohini said...

I am really impressed with your efforts and really pleased to visit this post.
Laptop battery replacement in chennai
Laptop unlocking service in chennai
100% genuine laptop parts
Laptop service centre in chennai
Laptop display replacement in chennai

Sheran said...

Post for sneeze guard are available of CRL company. These posts are constructed with stainless steel tubing. Glass can be attached to the full length U-channels with clear silicone. In DK Hardware posts are available in three stock sizes and two finishes.

Post a Comment