Showing posts with label Kafka. Show all posts
Showing posts with label Kafka. Show all posts

Monday, March 13, 2017

Apache Kafka Installation on Windows

  1. Make sure JDK and JRE are installed and path is set in environment variable
  2. Download Apache Kafka from https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.10-0.10.2.0.tgz
  3. Unpack kafka_2.10-0.10.2.0.tgz
  4. Copy extracted kafka_2.10-0.8.2.1 directory to C:\
  5. Open C:\kafka_2.10-0.10.2.0\config\server.properties and update existing log.dir entry as mentioned below :
    dirs=c:/kafka_2.10-0.10.2.0/kafka-logs
  6. Open C:\kafka_2.10-0.10.2.0\config\zookeeper.properties and update existing dataDir entry as mentioned below :
    dataDir=c:/kafka_2.10-0.10.2.0/zookeeper-data(Note : Make sure you use forward slash as depicted for step-5 and step-6)
  7. Installation of Single Node Single Broker Kafka is done on Windows
  8. Starting Zookeeper and Kafka server
    Open a command prompt and start Zookeeper server using following command :
    cd c:\kafka_2.10-0.10.2.0
    bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
  9. Now open another command prompt and start Kafka server using following command
    .\bin\windows\kafka-server-start.bat .\config\server.properties
    I hope both Zookeeper and Kafka server are executed successfully for you.
  10. Now let's create Kafka Topic using following command
    cd C:\kafka_2.10-0.10.2.0
    bin\windows\kafka-topics.bat ––create ––zookeeper localhost:2181 ––replication-factor 1 ––partition 1 ––topic test
  11. List the topics using following command
    cd C:\kafka_2.10-0.10.2.0
    bin\windows\kafka-topics.bat ––list  ––zookeeper localhost:2181
  12. Send message using Kafka console producer
    cd C:\kafka_2.10-0.10.2.0
    bin\windows\kafka-console-producer.bat ––broker-list localhost:9092 ––topic test
  13. Consume the above message using Kafka console consumer
    cd C:\kafka_2.10-0.10.2.0
    bin\windows\kafka-console-consumer.bat ––zookeeper localhost:2181 ––topic test ––from-beginning

Apache Spark – Catalyst Optimizer

Optimizer is the one that automatically finds out the most efficient plan to execute data operations specified in the user’s program. In...