当加载/清除大量的数据时,std :: vector会变得越来越慢

问题

我有一个相当复杂的image processing应用程序,其中一个子模块需要将巨大的二进制位图加载到内存中。 实际上高达96 GB(意思是888 888 x 888 888像素的图像)。 磁盘是2xSSD raid0,读写速度约为1 GB / s。 它将图像加载到智能指针向量(每个元素表示8个像素)的向量中(每个元素表示一个位图中的一行)。 这里的一个奇怪的问题是,在重复加载和清除向量之后(我发现内存实际上是没有内存泄漏的填充和清除),每次迭代似乎花费的时间越来越长。 特别清理记忆需要很长时间。

testing

我做了一些简单的testing应用程序来testing这个孤立的,从不同的angular度。 用原始指针代替智能指针给了同样奇怪的行为。 然后,我试图使用本地数组而不是vector,并做了诀窍。 使用向量后,100次迭代的载入/清除24 GB时间急剧增加,而arrays实现时间稳定。 下面是testing应用程序填充内存与24 GB的垃圾,而不是加载一个实际的图像,具有相同的结果。 在Windows 10 Pro上使用128 GB RAM进行testing,并使用Visual Studio 2013 Update 5进行构build。

这个函数使用vector来加载/清除:

void SimpleLoadAndClear_Vector(int width, int height) { time_t start_time, end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto width_bytes = width / 8; auto image = new vector<vector<unsigned char>*>(height); for (auto y = 0; y < height; y++) { (*image)[y] = new vector<unsigned char>(width_bytes); auto row_ptr = (*image)[y]; for (auto b = 0; b < width_bytes; b++) { (*row_ptr)[b] = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time, start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete (*image)[y]; } delete image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time, start_time); cout << to_string(mem_clear) + " sec" << endl; } 

该函数使用数组进行加载清除:

 void SimpleLoadAndClear_Array(int width, int height) { time_t start_time, end_time; // Load memory time(&start_time); cout << "Loading image into memory..."; auto width_bytes = width / 8; auto image = new unsigned char*[height]; for (auto y = 0; y < height; y++) { image[y] = new unsigned char[width_bytes]; auto row_ptr = image[y]; for (auto b = 0; b < width_bytes; b++) { row_ptr[b] = 0xFF; } } cout << "DONE: "; time(&end_time); auto mem_load = (int)difftime(end_time, start_time); cout << to_string(mem_load) << " sec" << endl; // Clear memory time(&start_time); cout << "Clearing memory..."; for (auto y = 0; y < height; y++) { delete[] image[y]; } delete[] image; cout << "DONE: "; time(&end_time); auto mem_clear = (int)difftime(end_time, start_time); cout << to_string(mem_clear) + " sec" << endl; } 

这是调用上述加载/清除function的主要function:

 void main() { auto width = 455960; auto height = 453994; auto i_max = 50; for (auto i = 0; i < i_max; i++){ SimpleLoadAndClear_Vector(width, height); } } 

在50次迭代之后,vector版本的testing输出如下所示(显然,加载/清除时间越来越多):

 Loading image into memory...DONE: 19 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 20 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 39 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 24 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 34 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 29 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 33 sec Loading image into memory...DONE: 28 sec Clearing memory...DONE: 37 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 35 sec Loading image into memory...DONE: 30 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 38 sec Loading image into memory...DONE: 31 sec Clearing memory...DONE: 41 sec Loading image into memory...DONE: 32 sec Clearing memory...DONE: 40 sec Loading image into memory...DONE: 33 sec Clearing memory...DONE: 42 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 43 sec Loading image into memory...DONE: 34 sec Clearing memory...DONE: 46 sec Loading image into memory...DONE: 36 sec Clearing memory...DONE: 47 sec Loading image into memory...DONE: 35 sec Clearing memory...DONE: 49 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 50 sec Loading image into memory...DONE: 37 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 51 sec Loading image into memory...DONE: 39 sec Clearing memory...DONE: 53 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 52 sec Loading image into memory...DONE: 40 sec Clearing memory...DONE: 55 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 56 sec Loading image into memory...DONE: 41 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 59 sec Loading image into memory...DONE: 42 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 60 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 44 sec Clearing memory...DONE: 63 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 64 sec Loading image into memory...DONE: 46 sec Clearing memory...DONE: 65 sec Loading image into memory...DONE: 45 sec Clearing memory...DONE: 67 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 69 sec Loading image into memory...DONE: 47 sec Clearing memory...DONE: 70 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 72 sec Loading image into memory...DONE: 48 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 49 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 74 sec Loading image into memory...DONE: 50 sec Clearing memory...DONE: 76 sec Loading image into memory...DONE: 51 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 78 sec Loading image into memory...DONE: 53 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 80 sec Loading image into memory...DONE: 54 sec Clearing memory...DONE: 82 sec Loading image into memory...DONE: 55 sec Clearing memory...DONE: 91 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 84 sec Loading image into memory...DONE: 56 sec Clearing memory...DONE: 88 sec 

arrays版本的testing输出在50次迭代后显示如下(显然,加载/清除时间是稳定的,并且不会越来越多地增加):

 Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 27 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 17 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 17 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 26 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 25 sec Clearing memory...DONE: 19 sec Loading image into memory...DONE: 18 sec Clearing memory...DONE: 25 sec Loading image into memory...DONE: 26 sec Clearing memory...DONE: 18 sec 

问题

  1. 当处理巨大的std :: vectors时,这个Windows处理内存操作的方式不好吗?
  2. 它是性病::vector,只是执行蹩脚的庞大的数据,通过devise?
  3. 我完全错过了什么吗?
  4. 是否有任何其他明显的std容器,我应该用(而不是从不同的线程在X和Y索引访问图像数据)?
  5. 任何其他好的解释和build议解决scheme?

Solutions Collecting From Web of "当加载/清除大量的数据时,std :: vector会变得越来越慢"

我做错了,我正在为图像中的每一行(数千次)调用矢量分配器。 首先将整个事物作为一个向量进行分配,然后将不同的行映射到大向量中的正确位置,即可解决问题。

感谢@PaulMcKenzie的回答,指出我正确的方向。