By default, Linux follows an optimistic memory allocation strategy. This means that when malloc() returns non-NULL there is no guarantee that the memory really is available. This is a really bad bug. In case it turns out that the system is out of memory, one or more processes will be killed by the infamous OOM killer. In case Linux is employed under circumstances where it would be less desirable to suddenly lose some randomly picked processes, and moreover the kernel version is sufficiently recent, one can switch off this overcommitting behavior using a command like
# echo 2 > /proc/sys/vm/overcommit_memory
See also the kernel Documentation directory, files vm/overcommit-accounting and sysctl/vm.txt.
– the malloc manual page, emphasis (italics) added
Yes, kiddies, malloc() returns non-null results when you have no memory left. You can even start to use that memory, but at some point you hit the barrier between what you requested and what was really available, and things go explodey and the kernel goes on a random killing spree. I quote, “this is a really bad bug.”