7

A recommended setting for JVM looks like following

-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

my question is - How do I set the above Java options for Kafka?

I know for sure that we can set 

export KAFKA_HEAP_OPTS="-Xmx8G -Xms8G"

but not sure if we can append the whole line 

"-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

to KAFKA_HEAP_OPTS variable

reference - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

Improve this question
1
  • yes you can, have a look on how it is done in /bin/kafka-server-start.sh
    – Paizo
    Commented May 25, 2018 at 8:46

2 Answers 2

Reset to default
6

you can check kafka-run-class.sh, here you can see what env variables kafka uses to start the java process:

  • $KAFKA_HEAP_OPTS
  • $KAFKA_JVM_PERFORMANCE_OPTS
  • $KAFKA_GC_LOG_OPTS
  • $KAFKA_JMX_OPTS
  • $KAFKA_LOG4J_OPTS

and then it run the java application passing them:

nohup $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS....

basically the content of the env variables are just appended as is to the command, it does not really matter where you put the settings as soon as the jvm parameters are in the correct order.

So, you can simply change $KAFKA_HEAP_OPTS however to keep the variable names consistent with their content I would put -Xmx8g -Xms8g in KAFKA_HEAP_OPTS and the remaining optimization to KAFKA_JVM_PERFORMANCE_OPTS.

Improve this answer
5
  • 1
    what you think about to set that ( in kafka-env template ) , export KAFKA_JVM_PERFORMANCE_OPTS="-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80" , is it also right syntax instead to separate the values in two variables
    – enodmilvado
    Commented May 25, 2018 at 12:08
  • not really because if empty default will apply, see if [ -z "$KAFKA_HEAP_OPTS" ]; then KAFKA_HEAP_OPTS="-Xmx256M" fi
    – Paizo
    Commented May 25, 2018 at 12:13
  • in that case I will prefer to do the following - export KAFKA_HEAP_OPTS="-Xmx8g -Xms8g" and export KAFKA_JVM_PERFORMANCE_OPTS=" -XX:MetaspaceSize=96m -XX:+UseG1GC-XX:MaxGCPauseMillis=20 - XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M-XX:MinMetaspaceFreeRatio=50 - XX:MaxMetaspaceFreeRatio=80"
    – enodmilvado
    Commented May 25, 2018 at 12:26
  • do you except this above ? ( since I use ambari I will set both variable in kafka-env template )
    – enodmilvado
    Commented May 25, 2018 at 12:28
  • 1
    should be ok, you can then verify the actual invocation with ps -aux | grep java
    – Paizo
    Commented May 25, 2018 at 12:55
4

As you mentioned, -Xmx8G -Xms8G should be set using KAFKA_HEAP_OPTS.

For the other configurations you listed, you should probably use KAFKA_JVM_PERFORMANCE_OPTS.

I'm not aware of a place where all the supported environment variables are clearly described. The best is to check the kafka-run-class.sh tool, as it is called by all the tools, including kafka-server-start.sh.

For example:

Improve this answer
1
  • @Michael , can you answer please to Paizo , seems he not agree with you
    – enodmilvado
    Commented May 25, 2018 at 11:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.