Memory for a VMware machine (Workstation or Player) is set in the VM Settings with the machine OFF. Memory is not set within the Guest machine.
General rule of thumb is to limit the amount of memory allocated to the machine to not more than half the host memory. Allocating 10 GB requires you have 20 GB of memory in your host machine.
A VM set this way can use all the memory allocated without impacting the Host machine.
whenever I'm trying to start something that needs a lot of RAM - it kills the entire host system failing to free that standby memory.
This is the result of allocating too much memory to the Guest machine. 10 GB is likely more than half your host memory.
It is not a problem with VMware as that is the way VMware allocates memory.
Memory swapping may be occurring in the guest. According to memory settings, swapping can occur. But excessive swapping may be a guest issue.
Make sure you have adequate Host memory to run Virtual Machines - minimum (in my view) is 16 GB.
Note:
I have 3 VMware Guests running here on a Windows 11 Host and everything co-exists well by keep machine and guest memory well managed.