Limit memory usage of Java 8 application (SceneBuilder)

Question

I'm experiencing trouble with (what appears to be) memory leaks in JavaFX Scene Builder. I have filed a bug report, but in the meantime my system sometimes gets unbearably slow as Scene Builder uses 10+ GB.

As a temporary fix I want to limit the amount of memory it can use. I have tried setting the following in SceneBuilder.cfg under JVMOptions:

-XX:InitialHeapSize=256m
-XX:MaxHeapSize=256m
-XX:MaxMetaspaceSize=256m

jps confirms the arguments are passed to the process:

3052  -Djava.library.path=/opt/SceneBuilder/app:/opt/SceneBuilder -Djava.launcher.path=/opt/SceneBuilder -Dapp.preferences.id=com/oracle/javafx/scenebuilder/app -XX:InitialHeapSize=256m -XX:MaxHeapSize=256m -XX:MaxMetaspaceSize=256m

However, the process still climbs up in memory usage, going way over 256MB, quickly surpassing 6GB and over.

So is there a way to truly limit the amount of memory used by a Java process?

Edit: I have tried using ulimit:

ulimit -d 1024000 -v 1024000 -m 1024000

but then I'm getting an error when trying to launch:

Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

which suggests Java indeed ignores the MetaspaceSize parameter. Any other way to limit memory in Java 8?

Answer 1

Assuming You're on Unix-based operating system.

1. You can run the program as different user and enforce limits defined in /etc/security/limits.conf on this account. This should also allow you to reserve more resources for your normal user account.

2. You can try caging the process with tool like timeout.

But since ulimit doesn't help, it'll probably end up just with the process crashing (with cannot allocate memory) or being killed by the system.

Also check-out https://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process since this is similar question.

Edit: As for metaspace limit being ignored. Answer is here https://stackoverflow.com/questions/31075761/java-8-reserves-minimum-1g-for-metaspace-despite-maxmetaspacesize