Linux:由于虚拟内存限制,不能在单个进程中分配超过32 GB / 64 GB的内存

我有一个128 GB的RAM,运行Linux(3.19.5-200.fc21.x86_64)的计算机。 但是,我不能在一个进程中分配超过30GB的内存。 除此之外, malloc失败:

 #include <stdlib.h> #include <iostream> int main() { size_t gb_in_bytes = size_t(1)<<size_t(30); // 1 GB in bytes (2^30). // try to allocate 1 block of 'i' GB. for (size_t i = 25; i < 35; ++ i) { size_t n = i * gb_in_bytes; void *p = ::malloc(n); std::cout << "allocation of 1 x " << (n/double(gb_in_bytes)) << " GB of data. Ok? " << ((p==0)? "nope" : "yes") << std::endl; ::free(p); } } 

这会产生以下输出:

 /tmp> c++ mem_alloc.cpp && a.out allocation of 1 x 25 GB of data. Ok? yes allocation of 1 x 26 GB of data. Ok? yes allocation of 1 x 27 GB of data. Ok? yes allocation of 1 x 28 GB of data. Ok? yes allocation of 1 x 29 GB of data. Ok? yes allocation of 1 x 30 GB of data. Ok? yes allocation of 1 x 31 GB of data. Ok? nope allocation of 1 x 32 GB of data. Ok? nope allocation of 1 x 33 GB of data. Ok? nope allocation of 1 x 34 GB of data. Ok? nope 

我search了相当一段时间,发现这与最大的虚拟内存大小有关:

 ~> ulimit -all [...] max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited virtual memory (kbytes, -v) 32505856 [...] 

我可以通过ulimit -v 64000000将此限制增加到ulimit -v 64000000 GB,但不能进一步。 除此之外,我得到了operation not permitted错误:

 ~> ulimit -v 64000000 ~> ulimit -v 65000000 bash: ulimit: virtual memory: cannot modify limit: Operation not permitted ~> ulimit -v unlimited bash: ulimit: virtual memory: cannot modify limit: Operation not permitted 

一些更多的search表明,原则上应该可以通过/etc/security/limits.conf的“as”(地址空间)条目设置这些限制。 但是,通过这样做,我只能减less虚拟内存的最大数量,而不能增加它。

有没有办法完全提升每个进程的虚拟内存限制,或将其提高到64GB以上? 我想在单个应用程序中使用所有的物理内存。

编辑:

  • 在Ingo Leonhardt之后,我以root用户的身份尝试了ulimits -v unlimited ,而不是标准用户。 这样做解决了root问题(程序可以以root身份login时分配所有的物理内存)。 但是这只适用于root,不适用于其他用户。 但是,至less这意味着内核原则上可以很好地处理这个问题,而且只有一个configuration问题。

  • 关于limits.conf :我试过明确添加

    • 努力无限
    • 软无限

    /etc/security/limits.conf ,然后重启。 这没有效果。 以标准用户身份login后, ulimit -v仍返回大约32 GB,并且ulimit -v 65000000仍然表示permission denied (而ulimit -v 64000000起作用)。 其余的limits.conf被注释掉了,在/etc/security/limits.d中只有一个不相关的条目(非root用户限制nproc为4096)。 也就是说,虚拟内存限制必须来自其他地方而不是limits.conf 。 任何想法还有什么可以导致ulimits -v不是“无限”?

编辑/解决scheme:

  • 这是我自己的愚蠢造成的。 我在用户设置中有一个(被遗忘的)程序,它使用setrlimit限制每个进程的内存量,以防止Linux交换死亡。 它被无意中从32 GB的机器复制到128 GB的机器。 感谢Paul和Andrew Janke以及其他人帮助我们追踪。 对不起大家:/。

  • 如果有其他人遇到这种情况:在bash和configuration文件设置中searchulimit/setrlimit ,以及潜在调用这些设置(包括你自己的和系统级的/ etc设置)的程序,并确保/security/limits.conf不包括这个限制…(或至less尝试创build一个新的用户,看看这是否发生在您的用户或系统设置)

Solutions Collecting From Web of "Linux:由于虚拟内存限制,不能在单个进程中分配超过32 GB / 64 GB的内存"

这是一个ulimit和系统设置问题,而不是一个c + +的问题。

我可以在Amazon EC2实例类型r3.4xlarge上运行适当修改的代码,没有任何问题。 现货市场价格低于每小时0.20美元,所以我建议你租一个,也许在/ etc中看一看,然后比较你自己的设置…或者你需要重新编译一个Linux内核来使用它很多内存……但它不是一个C ++或gcc的问题。

EC2机器上的Ubuntu已经被设置为无限的进程内存。

 $ sudo su # ulimit -u --> unlimited 

这个有125GB的内存

 # free total used free shared buffers cached Mem: 125903992 1371828 124532164 344 22156 502248 -/+ buffers/cache: 847424 125056568 Swap: 0 0 0 

我修改了程序的限制,最高可达149GB。

这是输出。 看起来不错,高达118GB。

 root@ip-10-203-193-204:/home/ubuntu# ./memtest allocation of 1 x 25 GB of data. Ok? yes allocation of 1 x 26 GB of data. Ok? yes allocation of 1 x 27 GB of data. Ok? yes allocation of 1 x 28 GB of data. Ok? yes allocation of 1 x 29 GB of data. Ok? yes allocation of 1 x 30 GB of data. Ok? yes allocation of 1 x 31 GB of data. Ok? yes allocation of 1 x 32 GB of data. Ok? yes allocation of 1 x 33 GB of data. Ok? yes allocation of 1 x 34 GB of data. Ok? yes allocation of 1 x 35 GB of data. Ok? yes allocation of 1 x 36 GB of data. Ok? yes allocation of 1 x 37 GB of data. Ok? yes allocation of 1 x 38 GB of data. Ok? yes allocation of 1 x 39 GB of data. Ok? yes allocation of 1 x 40 GB of data. Ok? yes allocation of 1 x 41 GB of data. Ok? yes allocation of 1 x 42 GB of data. Ok? yes allocation of 1 x 43 GB of data. Ok? yes allocation of 1 x 44 GB of data. Ok? yes allocation of 1 x 45 GB of data. Ok? yes allocation of 1 x 46 GB of data. Ok? yes allocation of 1 x 47 GB of data. Ok? yes allocation of 1 x 48 GB of data. Ok? yes allocation of 1 x 49 GB of data. Ok? yes allocation of 1 x 50 GB of data. Ok? yes allocation of 1 x 51 GB of data. Ok? yes allocation of 1 x 52 GB of data. Ok? yes allocation of 1 x 53 GB of data. Ok? yes allocation of 1 x 54 GB of data. Ok? yes allocation of 1 x 55 GB of data. Ok? yes allocation of 1 x 56 GB of data. Ok? yes allocation of 1 x 57 GB of data. Ok? yes allocation of 1 x 58 GB of data. Ok? yes allocation of 1 x 59 GB of data. Ok? yes allocation of 1 x 60 GB of data. Ok? yes allocation of 1 x 61 GB of data. Ok? yes allocation of 1 x 62 GB of data. Ok? yes allocation of 1 x 63 GB of data. Ok? yes allocation of 1 x 64 GB of data. Ok? yes allocation of 1 x 65 GB of data. Ok? yes allocation of 1 x 66 GB of data. Ok? yes allocation of 1 x 67 GB of data. Ok? yes allocation of 1 x 68 GB of data. Ok? yes allocation of 1 x 69 GB of data. Ok? yes allocation of 1 x 70 GB of data. Ok? yes allocation of 1 x 71 GB of data. Ok? yes allocation of 1 x 72 GB of data. Ok? yes allocation of 1 x 73 GB of data. Ok? yes allocation of 1 x 74 GB of data. Ok? yes allocation of 1 x 75 GB of data. Ok? yes allocation of 1 x 76 GB of data. Ok? yes allocation of 1 x 77 GB of data. Ok? yes allocation of 1 x 78 GB of data. Ok? yes allocation of 1 x 79 GB of data. Ok? yes allocation of 1 x 80 GB of data. Ok? yes allocation of 1 x 81 GB of data. Ok? yes allocation of 1 x 82 GB of data. Ok? yes allocation of 1 x 83 GB of data. Ok? yes allocation of 1 x 84 GB of data. Ok? yes allocation of 1 x 85 GB of data. Ok? yes allocation of 1 x 86 GB of data. Ok? yes allocation of 1 x 87 GB of data. Ok? yes allocation of 1 x 88 GB of data. Ok? yes allocation of 1 x 89 GB of data. Ok? yes allocation of 1 x 90 GB of data. Ok? yes allocation of 1 x 91 GB of data. Ok? yes allocation of 1 x 92 GB of data. Ok? yes allocation of 1 x 93 GB of data. Ok? yes allocation of 1 x 94 GB of data. Ok? yes allocation of 1 x 95 GB of data. Ok? yes allocation of 1 x 96 GB of data. Ok? yes allocation of 1 x 97 GB of data. Ok? yes allocation of 1 x 98 GB of data. Ok? yes allocation of 1 x 99 GB of data. Ok? yes allocation of 1 x 100 GB of data. Ok? yes allocation of 1 x 101 GB of data. Ok? yes allocation of 1 x 102 GB of data. Ok? yes allocation of 1 x 103 GB of data. Ok? yes allocation of 1 x 104 GB of data. Ok? yes allocation of 1 x 105 GB of data. Ok? yes allocation of 1 x 106 GB of data. Ok? yes allocation of 1 x 107 GB of data. Ok? yes allocation of 1 x 108 GB of data. Ok? yes allocation of 1 x 109 GB of data. Ok? yes allocation of 1 x 110 GB of data. Ok? yes allocation of 1 x 111 GB of data. Ok? yes allocation of 1 x 112 GB of data. Ok? yes allocation of 1 x 113 GB of data. Ok? yes allocation of 1 x 114 GB of data. Ok? yes allocation of 1 x 115 GB of data. Ok? yes allocation of 1 x 116 GB of data. Ok? yes allocation of 1 x 117 GB of data. Ok? yes allocation of 1 x 118 GB of data. Ok? yes allocation of 1 x 119 GB of data. Ok? nope allocation of 1 x 120 GB of data. Ok? nope allocation of 1 x 121 GB of data. Ok? nope allocation of 1 x 122 GB of data. Ok? nope allocation of 1 x 123 GB of data. Ok? nope allocation of 1 x 124 GB of data. Ok? nope allocation of 1 x 125 GB of data. Ok? nope allocation of 1 x 126 GB of data. Ok? nope allocation of 1 x 127 GB of data. Ok? nope allocation of 1 x 128 GB of data. Ok? nope allocation of 1 x 129 GB of data. Ok? nope allocation of 1 x 130 GB of data. Ok? nope allocation of 1 x 131 GB of data. Ok? nope allocation of 1 x 132 GB of data. Ok? nope allocation of 1 x 133 GB of data. Ok? nope allocation of 1 x 134 GB of data. Ok? nope allocation of 1 x 135 GB of data. Ok? nope allocation of 1 x 136 GB of data. Ok? nope allocation of 1 x 137 GB of data. Ok? nope allocation of 1 x 138 GB of data. Ok? nope allocation of 1 x 139 GB of data. Ok? nope allocation of 1 x 140 GB of data. Ok? nope allocation of 1 x 141 GB of data. Ok? nope allocation of 1 x 142 GB of data. Ok? nope allocation of 1 x 143 GB of data. Ok? nope allocation of 1 x 144 GB of data. Ok? nope allocation of 1 x 145 GB of data. Ok? nope allocation of 1 x 146 GB of data. Ok? nope allocation of 1 x 147 GB of data. Ok? nope allocation of 1 x 148 GB of data. Ok? nope allocation of 1 x 149 GB of data. Ok? nope 

现在,我花了这个0.17美元