Develop
Kafka Cluster Tutorial
kimyuuum
2020. 11. 11. 13:58
kafka-cluster-tutorial
Create simple kafka cluster easily!
Download
my version is 2.12-2.3.0
Start Server
Before start kafka, we need to start zookeeper first.
config files
#zookeeper / kafka config files
$ vi config/zookeeper.properties
$ vi config/server.properties
start zookeeper & kafka
$ bin/zookeeper-server-start.sh
$ bin/kafka-server-start.sh
you can see messages
{: width="100%"}
Multi broker cluster
1. Create each properties
$ cp config/server.properties config/server1.properties
$ cp config/server.properties config/server2.properties
2. Modify each config files
#server1
config/server1.properties:
broker.id=1
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-1
#server2
config/server2.properties:
broker.id=2
listeners=PLAINTEXT://:9095
log.dirs=/tmp/kafka-logs-2
3. Start Replicated nodes
$ bin/kafka-server-start.sh config/server1.properties &
$ bin/kafka-server-start.sh config/server2.properties &
4. Create new topic
$ bin/kafka-topics.sh --create --bootstrap-server localhost:9093 --replication-factor 3 --partitions 1 --topic my-replicated-topic
kafka-topics.sh
: topic execute file
`replication-factor` : can decide replica count
`partitions` : can decide partition count
5. Check topic
$ bin/kafka-topics.sh --describe --bootsrap-server localhost:9093 --topic my-replicated-topic
{: width="100%"}
[broker.id](http://broker.id)
: cluster에서 각 노드의 식별 가능한 영구적인 id, broker가 여러개가 되면, leader node가 생성된다.
Replicas
: 구성된 node 목록
Isr
: 현재 살아있는 node
zookeeper port
⇒ 2182
Ref.
Useful Commands
##stop server
$ bin/kafka-server-stop.sh config/server.properties
$ bin/kafka-server-stop.sh config/server1.properties
$ bin/kafka-server-stop.sh config/server2.properties
$ bin/zookeeper-server-stop.sh config/zookeeper.properties
##start server
##before start kafka, you should run zookeeper first
$ bin/zookeeper-server-start.sh config/zookeeper.properties &
$ bin/kafka-server-start.sh config/server.properties &
##search topic
$ ./bin/kafka-topics.sh --list --bootstrap-server localhost:9093