As I understand it, setting /proc/sys/vm/overcommit_memory to 1 is supposed to make sure malloc always succeeds, and set the OOM killer loose if there's an actual memory problem.
I'm wondering what happens when you've malloc'd so much memory that you've exhausted the address space for your process? Does it return NULL despite the overcommit_memory setting, or does your process get a signal? Or something else entirely?