docker搭建kafka叢集的方法實現

2022-05-17 19:00:24

一、原生Docker命令

1. 刪除所有dangling資料卷（即無用的Volume，殭屍檔案）

docker volume rm $(docker volume ls -qf dangling=true)

2. 刪除所有dangling映象（即無tag的映象）

docker rmi $(docker images | grep "^<none>" | awk "{print $3}"

3. 刪除所有關閉的容器

docker ps -a | grep Exit | cut -d ' ' -f 1 | xargs docker rm

二、映象選擇

環境為M1版本的mbp:

Zookeeper採用zookeeper
Kafka採用wurstmeister/kafka
Kafka-Manager採用scjtqs/kafka-manager
Mysql採用mysql/mysql-server

三、叢集規劃

1. 新建docker網路

docker network create docker-net --subnet 172.20.10.0/16

docker network ls

2. 叢集規劃

hostname	Ip addr	port	listener
zook1	172.20.10.11	2184:2181
zook2	172.20.10.12	2185:2181
zook3	172.20.10.13	2186:2181
kafka1	172.20.10.14	內部9092:9092，外部9192:9192	kafka1
kafka2	172.20.10.15	內部9093:9093，外部9193:9193	kafka2
Kafka3	172.20.10.16	內部9094:9094，外部9194:9194	Kafka3
本機（宿主機Mbp）	172.20.10.2
kafka manager	172.20.10.10	9000:9000

四、Zookeeper叢集安裝

1. 新建檔案zk-docker-compose.yml

version: '3.4'

services:
  zook1:
    image: zookeeper:latest
    restart: always
    hostname: zook1
    container_name: zook1 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2183:2181 #將本容器的zookeeper預設埠號對映出去
    volumes: # 掛載資料卷
    - "/Users/konsy/Development/volume/zkcluster/zook1/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs"
    environment:
        ZOO_MY_ID: 1  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
    networks:
        docker-net:
            ipv4_address: 172.20.10.11

  zook2:   
    image: zookeeper:latest
    restart: always
    hostname: zook2
    container_name: zook2 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2184:2181 #將本容器的zookeeper預設埠號對映出去
    volumes:
    - "/Users/konsy/Development/volume/zkcluster/zook2/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs"
    environment:
        ZOO_MY_ID: 2  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
    networks:
        docker-net:
            ipv4_address: 172.20.10.12
            
  zook3:   
    image: zookeeper:latest
    restart: always
    hostname: zook3
    container_name: zook3 #容器名稱，方便在rancher中顯示有意義的名稱
    ports:
    - 2185:2181 #將本容器的zookeeper預設埠號對映出去
    volumes:
    - "/Users/konsy/Development/volume/zkcluster/zook3/data:/data"
    - "/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog"
    - "/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs"
    environment:
        ZOO_MY_ID: 3  #即是zookeeper的節點值，也是kafka的brokerid值
        ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181
    networks:
        docker-net:
            ipv4_address: 172.20.10.13
networks:
  docker-net:
    external:
      name: docker-net

2. 執行指令碼部署zookeeper至Docker:

docker compose -f ./zk-docker-compose.yml up -d

五、Kafka叢集安裝

1. 新建檔案kafka-docker-compose.yml

version: '2'

services:
  kafka1:
    image: docker.io/wurstmeister/kafka
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
      - 9093:9093
      - 9193:9193
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193
      #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
      SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控埠，來監測叢集資料
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      docker-net:
        ipv4_address: 172.20.10.14

  kafka2:
    image: docker.io/wurstmeister/kafka
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
      - 9094:9094
      - 9194:9194
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194
      #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控埠，來監測叢集資料
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      docker-net:
        ipv4_address: 172.20.10.15

  kafka3:
    image: docker.io/wurstmeister/kafka
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
      - 9095:9095
      - 9195:9195
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195
      #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181
      ALLOW_PLAINTEXT_LISTENER : 'yes'
      JMX_PORT: 9999 #開放JMX監控埠，來監測叢集資料
    volumes:
      - /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka
      - /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka
    external_links:
      - zook1
      - zook2
      - zook3
    networks:
      docker-net:
        ipv4_address: 172.20.10.16
networks:
  docker-net:
    external:
      name: docker-net

2. 執行指令碼部署kafka至Docker:

docker compose -f ./kafka-docker-compose.yml up -d

3. listeners 和 advertised.listeners

listeners: 學名叫監聽器，其實就是告訴外部連線者要通過什麼協定存取指定主機名和埠開放的 Kafka 服務。
advertised.listeners：和 listeners 相比多了個 advertised。Advertised 的含義表示宣稱的、公佈的，就是說這組監聽器是 Broker 用於對外發布的。

比如說：

   listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094
   advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公網 ip>:埠
   kafka_listener_security_protocol_map: "INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT"
   kafka_inter_broker_listener_name: "INSIDE"

advertised_listeners 監聽器會註冊在 zookeeper 中；

當我們對 172.17.0.10:9092 請求建立連線，kafka 伺服器會通過 zookeeper 中註冊的監聽器，找到 INSIDE 監聽器，然後通過 listeners 中找到對應的通訊 ip 和埠；

同理，當我們對 <公網 ip>:埠請求建立連線，kafka 伺服器會通過 zookeeper 中註冊的監聽器，找到 OUTSIDE 監聽器，然後通過 listeners 中找到對應的通訊 ip 和埠 172.17.0.10:9094；

總結：advertised_listeners 是對外暴露的伺服器埠，真正建立連線用的是 listeners。

4.安裝kafka-manager

4.1 新建檔案kafka-manager-docker-compose.yml

version: '2'

services:
  kafka-manager:
    image: scjtqs/kafka-manager:latest
    restart: always
    hostname: kafka-manager
    container_name: kafka-manager
    ports:
      - 9000:9000
    external_links:  # 連線本compose檔案以外的container
      - zook1
      - zook2
      - zook3
      - kafka1
      - kafka2
      - kafka3
    environment:
      ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181
      KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095
      APPLICATION_SECRET: letmein
      KM_ARGS: -Djava.net.preferIPv4Stack=true
    networks:
      docker-net:
        ipv4_address: 172.20.10.10
networks:
  docker-net:
    external:
      name: docker-net

4.2 執行指令碼部署kafka-manager至Docker:

docker compose -f ./kafka-manager-docker-compose.yml up -d

4.3 設定Cluster

5. 測試kafka

5.1連線容器

進入kafka容器的命令列：

docker exec -ti kafka /bin/bash

進入kafka所在目錄：

cd opt/kafka_2.13-2.8.1/

5.2 建立Replication為2，Partition為2的topic

bin/kafka-topics.sh --create --zookeeper zook1:2181 --replication-factor 2 --partitions 2 --topic partopic

5.3 檢視topic的狀態

在kafka容器中的 opt/kafka_2.12-1.1.0/ 目錄下輸入

bin/kafka-topics.sh --describe --zookeeper zook1:2181 --topic partopic

到此這篇關於docker搭建kafka叢集的方法實現的文章就介紹到這了,更多相關docker搭建kafka叢集內容請搜尋it145.com以前的文章或繼續瀏覽下面的相關文章希望大家以後多多支援it145.com！