{"id":87,"date":"2013-05-21T19:22:41","date_gmt":"2013-05-21T17:22:41","guid":{"rendered":"http:\/\/blog.ramses-pyramidenbau.de\/?p=87"},"modified":"2013-05-21T19:38:04","modified_gmt":"2013-05-21T17:38:04","slug":"linuxs-strange-memory-allocation-strategy","status":"publish","type":"post","link":"https:\/\/blog.vmexit.de\/?p=87","title":{"rendered":"Linux’s strange memory allocation strategy"},"content":{"rendered":"

As I learned today<\/a>, Linux has a real strange memory allocation strategy. Have a look at this piece of code:<\/p>\n

\r\n#include <stdio.h>\r\n#include <stdlib.h>\r\n\r\nint main(void) {\r\n  while(1) {\r\n    char *foo = (char*)malloc(1024);\r\n    if(foo == NULL) {\r\n      printf("Couldn't alloc\\n");\r\n      fflush(stdout);\r\n      return 0;\r\n    }\r\n  }\r\n  return 0;\r\n}\r\n<\/pre>\n
According to the malloc reference<\/a> it should return NULL<\/strong> if it is not able to allocate memory:<\/p>\n
“If the function failed to allocate the requested block of memory, a null pointer<\/i> is returned.”<\/p><\/blockquote>\n
So I thought my process would write Couldn’t alloc<\/strong> to stdout<\/em> and exit in a proper way. But it didn’t act that way. When my system ran out of memory the process got killed by the kernel with signal SIGKILL. So why does it act like this? Wikipedia writes about Out of memory<\/a>:<\/p>\n
“A process which exceeds its per-process limit and then attempts to allocate further memory – with malloc<\/a>()<\/code>, for example – will return failure. A well-behaved application should handle this situation gracefully; however, many do not. An attempt to allocate memory without checking the result is known as an “unchecked malloc”.”<\/p><\/blockquote>\n
Well… Yes… Of course, you always should check malloc()<\/strong> if it returned NULL<\/strong>, but under normal conditions it never will return NULL<\/strong> because of Linux’s memory overcommitting<\/a><\/em>. By default Linux has an optimistic memory allocation strategy. When allocating memory, malloc()<\/strong> will return a pointer. The space on the heap which is needed for that pointer gets allocated at the first read\/write operation to that address. In my opinion, this is a really strange behaviour, because when memory gets allocated, it will actually be used. Here’s a short example to see the effects of that strange behaviour:<\/p>\n
\r\n#include <stdio.h>\r\n#include <stdlib.h>\r\n#include <string.h>\r\n\r\n#define MALLOC_SIZE (1024*1024*10)\r\n\r\nint main(void) {\r\n  int i = 0;\r\n  while(1) {\r\n    char *foo = (char*)malloc(MALLOC_SIZE);\r\n    \/\/memset(foo, 0xaa, MALLOC_SIZE);\r\n    printf("Allocation No. %d\\n", i++);\r\n    if(foo == NULL) {\r\n      printf("Couldn't alloc\\n");\r\n      fflush(stdout);\r\n      return 0;\r\n    }\r\n  }\r\n  return 0;\r\n}\r\n<\/pre>\n
With the memset<\/strong> line commented out it results to:<\/p>\n
\r\nralf@Pegasus:C$ .\/test|tail\r\nAllocation No. 1841101\r\nAllocation No. 1841102\r\nAllocation No. 1841103\r\nAllocation No. 1841104\r\nAllocation No. 1841105\r\nAllocation No. 1841106\r\nAllocation No. 1841107\r\nAllocation No. 1841108\r\nAllocation No. 1841109\r\nAllocation Nzsh: killed     .\/test |\r\nzsh: done       tail\r\n<\/pre>\n
And with the memset<\/strong> commented in it results to:<\/p>\n
\r\nralf@Pegasus:C$ .\/test|tail\r\nAllocation No. 1275\r\nAllocation No. 1276\r\nAllocation No. 1277\r\nAllocation No. 1278\r\nAllocation No. 1279\r\nAllocation No. 1280\r\nAllocation No. 1281\r\nAllocation No. 1282\r\nAllocation No. 1283\r\nAllocazsh: killed     .\/test |\r\nzsh: done       tail\r\n<\/pre>\n
This is really funny. You can allocate tons of space you don’t have. That’s creepy. But this behaviour can be switched by<\/p>\n
\r\necho 2 > \/proc\/sys\/vm\/overcommit_memory\r\n<\/pre>\n
“Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit – never commit a virtual address space larger than swap space plus a fraction overcommit_ratio of the physical memory. Here \/proc\/sys\/vm\/overcommit_ratio (by default 50) is another user-settable parameter. It is possible to set overcommit_ratio to values larger than 100. (See also Documentation\/vm\/overcommit-accounting.)” Source<\/a><\/p><\/blockquote>\n
Then, both results (with and without memset<\/strong>) lead to:<\/p>\n
\r\nralf@Pegasus:C$ .\/test|tail\r\nAllocation No. 433\r\nAllocation No. 434\r\nAllocation No. 435\r\nAllocation No. 436\r\nAllocation No. 437\r\nAllocation No. 438\r\nAllocation No. 439\r\nAllocation No. 440\r\nAllocation No. 441\r\nCouldn't alloc\r\n<\/pre>\n
And this is exactly what I originally expected to be the normal way<\/em>. Without this bypass, malloc<\/strong> will never return NULL<\/strong> and a process will not be able to shut down properly if the system runs out of memory.<\/p>\n
Here<\/a>‘s a nice anecdote on that topic.<\/p>\n","protected":false},"excerpt":{"rendered":"
As I learned today, Linux has a real strange memory allocation strategy. Have a look at this piece of code: #include <stdio.h> #include <stdlib.h> int main(void) { while(1) { char *foo = (char*)malloc(1024); if(foo == NULL) { printf("Couldn’t alloc\\n"); fflush(stdout); return 0; } } return 0; } According to the malloc reference it should return … Continue reading Linux’s strange memory allocation strategy<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,7,6],"tags":[],"class_list":["post-87","post","type-post","status-publish","format-standard","hentry","category-c","category-kernel","category-linux"],"_links":{"self":[{"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/posts\/87","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=87"}],"version-history":[{"count":38,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/posts\/87\/revisions"}],"predecessor-version":[{"id":123,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=\/wp\/v2\/posts\/87\/revisions\/123"}],"wp:attachment":[{"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=87"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=87"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.vmexit.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=87"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}