3

When using spark-submit to submit a Spark app to Yarn, I can pass java options to the driver via the --driver-java-options, for example: 

spark-submit --driver-java-options "-Dlog4j.configuration=file:///conf/log4j.properties" ...

How do I achieve the same when submitting via SparkLauncher? In particular, is there a way to achieve it with Spark 1.4?

Improve this question

1 Answer 1

Reset to default
6

Not familiar with SparkLauncher but from looking at the code it appears you can pass configuration with setConf(). In this if you add the property SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS this should have the same effect.

For example

Process spark = new SparkLauncher()
     .setAppResource("/my/app.jar")
     .setMainClass("my.spark.app.Main")
     .setMaster("local[*]")
     .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, "-Dmy.property=someval")
     .launch();
Improve this answer
2
  • 2
    Are you sure ? From spark.apache.org/docs/latest/configuration.html "Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file. "
    – mathieu
    Commented Jun 9, 2017 at 12:33
  • I don't think the question is referring to client mode, tbh functionality of SparkLauncher has probably changed between version 1.4 and 2.1 (the version linked in your comment)
    – Jem Tucker
    Commented Jun 9, 2017 at 12:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.