FATAL Fatal error during KafkaServer startup, NumberFormatException

This might have taken me a long time to figure out, but fortunately my super awesome colleague (nicknamed Mr.T, he also pities fools) showed me the solution.

We ran into a Kafka Broker which was not starting, and gave the exception that is pasted in the bottom of this post.

FATAL Fatal error during KafkaServer startup. [..] java.lang.NumberFormatException

The solution is that the string (in this example “hs_err_pid19313”) is actually an error log which exists in a topic partition directory. (Re)move this file and Kafka will start without a problem.

(Tip: use find and grep to quickly find the file, go to your Kafka storage directory and run the following command;

find . |grep hs_err_pid19313

Continue reading “FATAL Fatal error during KafkaServer startup, NumberFormatException”

This ZooKeeper instance is not currently serving requests

When one of your zookeeper nodes is sending you this message, that means that your Zookeeper cluster hasn’t started in the right order.

 

Solution: Restart your cluster (node per node), starting from node 1 (as stated in zoo.conf)

 

This problem is easy to diagnose. When the order was wrong you will get this output:

[myserver:myuser] ~: echo stat | nc localhost 2181
This ZooKeeper instance is not currently serving requests

 

After you’ve restarted all nodes (in the correct order), you will get this output:

[myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode
Mode: follower

[myserver:myuser] ~: echo stat | nc localhost 2181 |grep Mode
Mode: leader

 

 

Hope this will help you out!

Why no SSL!? Port is open!

Okay, this has taken me too long to not post.. So here it is..:

When your firewall is blocking SSL traffic but allowing HTTP traffic, openssl s_client will show this:

my_host:joris [/etc/stores] openssl s_client -host external_host -port 12345
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 247 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Continue reading “Why no SSL!? Port is open!”