Kafka Cluster Tutorial

kafka-cluster-tutorial

Create simple kafka cluster easily!

Download

Apache Kafka

my version is 2.12-2.3.0

Start Server

Before start kafka, we need to start zookeeper first.

config files

#zookeeper / kafka config files
$ vi config/zookeeper.properties

$ vi config/server.properties

start zookeeper & kafka

$ bin/zookeeper-server-start.sh
$ bin/kafka-server-start.sh

you can see messages

{: width="100%"}

Multi broker cluster

1. Create each properties

$ cp config/server.properties config/server1.properties
$ cp config/server.properties config/server2.properties

2. Modify each config files

#server1
config/server1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9094
    log.dirs=/tmp/kafka-logs-1

#server2
config/server2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9095
    log.dirs=/tmp/kafka-logs-2

3. Start Replicated nodes

$ bin/kafka-server-start.sh config/server1.properties &
$ bin/kafka-server-start.sh config/server2.properties &

4. Create new topic

$ bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 3 --partitions 1 --topic my-replicated-topic

kafka-topics.sh : topic execute file

`replication-factor` : can decide replica count
`partitions` : can decide partition count

5. Check topic

$ bin/kafka-topics.sh --describe --bootsrap-server localhost:9093 --topic my-replicated-topic

{: width="100%"}

[broker.id](http://broker.id) : cluster에서 각 노드의 식별 가능한 영구적인 id, broker가 여러개가 되면, leader node가 생성된다.

Replicas : 구성된 node 목록

Isr : 현재 살아있는 node

zookeeper port ⇒ 2182

Ref.

Apache Kafka

Useful Commands

##stop server
$ bin/kafka-server-stop.sh config/server.properties
$ bin/kafka-server-stop.sh config/server1.properties
$ bin/kafka-server-stop.sh config/server2.properties
$ bin/zookeeper-server-stop.sh config/zookeeper.properties

##start server
##before start kafka, you should run zookeeper first
$ bin/zookeeper-server-start.sh config/zookeeper.properties &
$ bin/kafka-server-start.sh config/server.properties &

##search topic
$ ./bin/kafka-topics.sh --list --bootstrap-server localhost:9093