Apacke Kafka Tutorial: Create Topic using Docker Compose

 

Apache Kafka stands as a robust distributed streaming platform, empowering users to construct real-time data pipelines and applications. While setting up Kafka traditionally entails complexities, Docker Compose offers a streamlined approach by defining and orchestrating multi-container Docker applications. This guide aims to demystify the process, providing a step-by-step tutorial on creating a Kafka topic using Docker Compose, catering to developers and DevOps professionals alike.

Prerequisites:

Before embarking on the journey, ensure your system meets the following prerequisites:

• Docker: Enables creation, deployment, and execution of applications via containers.
• Docker Compose: Facilitates the definition and orchestration of multi-container Docker applications.

Step 1: Crafting a Docker Compose File

The first step involves crafting a docker-compose.yml file, defining the Kafka and Zookeeper services essential for running Kafka instances. Zookeeper serves as a centralized service, maintaining configuration information, providing distributed synchronization, and offering group services.

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    networks:
      - kafka-net
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "YourTopicName:1:1"
    networks:
      - kafka-net
networks:
  kafka-net:
    driver: bridge

Replace "YourTopicName" with the desired topic name, adhering to the format TopicName:NumberOfPartitions:ReplicationFactor for the KAFKA_CREATE_TOPICS environment variable.

Step 2: Executing Docker Compose

Navigate to the directory containing your docker-compose.yml file and execute the following command in your terminal:

docker-compose up -d

This command fetches necessary Docker images for Kafka and Zookeeper, subsequently launching containers in detached mode.

Step 3: Validating Topic Creation

To ensure successful creation of your Kafka topic, leverage the Kafka topics command-line tool provided with Kafka. Execute the following command to list topics and verify your topic's existence:

docker-compose exec kafka kafka-topics.sh --list --zookeeper zookeeper:2181 

Step 4: Producing and Consuming Messages

To validate your setup further, utilize Kafka's console producer and consumer scripts to produce and consume messages.

Producing Messages:

Within the Kafka container's bash, execute:

docker-compose exec kafka kafka-console-producer.sh --broker-list localhost:9092 --topic YourTopicName 

Press Ctrl+D after entering messages to send them.

Consuming Messages:

In a separate terminal session, access the Kafka container and execute:

docker-compose exec kafka kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic YourTopicName --from-beginning 

You should observe the messages you produced earlier.

Step 5: Optional Topic Creation

Although Docker Compose initializes topics defined in the docker-compose.yaml file, you can manually create new Kafka topics if needed:

docker-compose exec kafka kafka-topics.sh --create --topic NewTopicName --partitions 1 --replication-factor 1 --bootstrap-server kafka:9092 

Replace "NewTopicName" with your desired topic name.

Conclusion

Congratulations! You've successfully created a Kafka topic using Docker Compose and verified its functionality by producing and consuming messages. This setup not only simplifies Kafka management but also furnishes a scalable and reproducible environment for streaming applications. Whether in local development or production deployment, Docker Compose with Kafka equips you with a robust toolkit to streamline data streaming pipelines.