https://fromkk.com/tags/kafka/Recent content in Kafka on KK's Blog (fromkk)Hugoenbebound@gmail.com (KK)bebound@gmail.com (KK)Sun, 10 Aug 2025 18:44:06 +0800https://fromkk.com/posts/improve-kafka-throughput/Fri, 28 May 2021 00:57:00 +0800bebound@gmail.com (KK)https://fromkk.com/posts/improve-kafka-throughput/<p>Kafka is a high-performance and scalable messaging system. Sometimes when handling big data. The default configuration may limit the maximum performance. In this article, I&rsquo;ll explain how messages are generate and saved in Kafka, and how to improve performance by changing configuration.</p> <h2 id="kafka-internals">Kafka Internals</h2> <h3 id="how-does-producer-send-messages">How does Producer Send Messages?</h3> <p>In short, messages will assembled into batches (named <code>RecordBatch</code>) and send to broker.</p> <p>The producer manages some internal queues, and each queue contains <code>RecordBatch</code> that will send to one broker. When calling <code>send</code> method, the producer will look into the internal queue and try to append this message to <code>RecordBatch</code> which is smaller than <code>batch.size</code> (default value is 16KB) or create new <code>RecordBatch</code>.</p>