Generally-speaking, pages are not swapped back into RAM unless they are needed. That is, the page would only be loaded back into RAM if the page is actually accessed by something (causing a page fault exception in the MMU which the OS would handle by loading the page back into RAM before allowing the accessing thread to continue). Hard-killing the process (with SIGKILL) does not run any of the process's threads, so the killed process can't try to access any of its pages, and those pages should not get loaded into RAM.

In fact, even if you don't kill the process and it does resume running, it will still swap in (to RAM) only the pages it actually accesses. Furthermore, if all RAM is full and it needs to access another page that is not in RAM, the OS will select some other page of RAM to swap out (to disk) to make room in RAM for the page your program needs to access. As long as there's some space in the swap partition on disk, this swapping of pages can continue indefinitely, keeping everything alive, without triggering the OOM killer. Of course, if this happens a lot, it can really slow down your machine because it's spending most of its time moving memory between RAM and disk instead of executing program instructions. This is called "thrashing".

Generally-speaking, pages are not swapped back into RAM unless they are needed. That is, the page would only be loaded back into RAM if the page is actually accessed by something (causing a page fault exception in the MMU which the OS would handle by loading the page back into RAM before allowing the accessing thread to continue). Hard-killing the process (with SIGKILL) does not run any of the process's threads, so the killed process can't try to access any of its pages, and those pages should not get loaded into RAM.

In fact, even if you kill the process "gently" by sending it a signal that it can catch and run a graceful shutdown, the OS will still only swap in the pages the process actually accesses. Furthermore, even if all RAM is full and your process needs to access a swapped page as part of its shutdown procedures, the OS will select some other page of RAM to swap out to make room in RAM for the page your program needs. As long as there's some space in the swap partition on disk, this swapping of pages can continue indefinitely, keeping everything alive, without triggering the OOM killer. So, in the scenario you describe, you should be safe to gracefully terminate your process.

One option for tracking and killing processes in a process tree on Linux is to use cgroups (short for Control Groups). If you start your top level process in a particular cgroup, all of its children will be in the same cgroup. When you decide you want to kill all of those processes, you can just tell the cgroup to kill everything in it (see cgroup.kill). This will send a SIGKILL to everything in the cgroup, hard-killing it.

Cgroups can also help you prevent your processes from consuming too much memory (among other things) in the first place by setting a memory.max limit on your cgroup. If the memory usage of all the processes in the cgroup combined exceeds the cgroup's memory.max limit, the OOM Killer will kill something in the cgroup, leaving the rest of your system alone.