You are entirely correct in your assumption.
Memory management algorithms are very complex and by any means not perfect. So swapping occurs even when there is plenty of spare RAM. On some systems, like Linux
, you can control swappiness, on others you can't. By swapping out data when there is still plenty of RAM, system in its own way prepares for the situation when it might run out of RAM.
So disabling swapping functionality might give you the improvement in performance because you will only be using RAM which is faster as you already said.
One thing to consider (and you mentioned it already) - you need to have enough RAM to accommodate all the programs you are executing, otherwise you are risking to run out of memory. In this case the performance will drop, some processes may be terminated by OS and system may experience crash/freeze. (read more about it here)
On some machines, especially ones that keep swap file on HDD not SSD, the effect from disabling swapping is very noticeable. On others it is not so obvious. But even if you don't get obvious improvement, think of it in another way, by disabling swapping you will save yourself some disk space on your SSD.
By disabling swapping, you will also prevent memory algorithms from doing unnecessary operation - moving data from RAM to swap and vice versa - in case of SSD this will prevent excessive wear. And in any case this will improve the performance by eliminating unnecessary operations.
Also, read: