MYSQL巨大的SQL文件插入| MyISAM速度突然减慢插入(奇怪的问题)

我正面临着非常奇怪的问题,我在这里提出了关于加快在MYSQL中插入的问题,特别是关于插入大小为几GB的巨大SQL文件的问题。 他们build议我使用MyISAM引擎。 我做了以下几点:

  • ALTER TABLE revision ENGINE=MyISAM;
  • 使用ALTER TABLE .. DISABLE KEYS
  • (仅限MyISAM)将bulk_insert_buffer_size设置为500M。
  • (仅限MyISAM)设置unique_checks = 0 。 没有检查。
  • SET autocommit=0; ... SQL import statements ... COMMIT;
  • SET foreign_key_checks=0;

这加快了5分钟,过去花了2个小时的过程,我留下了深刻的印象。 但现在当我尝试与其他表相同的事情,那么没有加速,并需要几个小时了 (…

当我最初获得成功比我的CPU使用率大约是90% ,插入只需要5分钟,但现在按照相同的程序,我的CPU使用率是最多5%左右 。 它显示错误..

我也通过以下方法validation了我的表引擎是MyISAM:

 SHOW TABLE STATUS WHERE Name = 'xxx'; 

注意:我正在使用维基百科数据库schema.I在Wikipedia数据集的Categorylinks表上获得成功。 我在修订 , 页面和文本表没有成功(非常慢的插入速度)。

请帮我解决这个奇怪的问题。

我还没有找到理想的原因,背后的问题。 但是,当我钻了下来,我发现它与表格有很丰富的关系。 虽然我已经禁用了键..但是我仍然认为以下设置对于大文件插入是最好的,因为以下设置可以在大多数表中提供最佳性能:

  - ALTER TABLE revision ENGINE=MyISAM; - Use ALTER TABLE .. DISABLE KEYS . - (MyISAM only) Set bulk_insert_buffer_size to 500M. - (MyISAM only) Set unique_checks = 0 . not checked. - SET autocommit=0; ... SQL import - statements ... COMMIT; - SET foreign_key_checks=0; 

其他调整性能的调整如下:请参阅完整的my.cnf文件..

 # # The MySQL database server configuration file. # # You can copy this to one of: # - "/etc/mysql/my.cnf" to set global options, # - "~/.my.cnf" to set user-specific options. # # One can use all long options that the program supports. # Run program with --help to get a list of available options and with # --print-defaults to see which it would actually understand and use. # # For explanations see # http://dev.mysql.com/doc/mysql/en/server-system-variables.html # This will be passed to all mysql clients # It has been reported that passwords should be enclosed with ticks/quotes # escpecially if they contain "#" chars... # Remember to edit /etc/mysql/debian.cnf when changing the socket location. [client] port = 3306 socket = /var/run/mysqld/mysqld.sock # Here is entries for some specific programs # The following values assume you have at least 32M ram # This was formally known as [safe_mysqld]. Both versions are currently parsed. [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] # # * Basic Settings # user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmp bulk_insert_buffer_size = 1G lc-messages-dir = /usr/share/mysql skip-grant-tables skip-networking skip-external-locking init_connect='SET autocommit=0' innodb_buffer_pool_size = 1G innodb_flush_log_at_trx_commit = 0 max_allowed_packet = 500M table_open_cache = 512 max_connections=100 query_cache_size=32M table_cache=512 tmp_table_size=64M thread_cache_size=8 myisam_max_sort_file_size=100G myisam_sort_buffer_size=256M key_buffer_size=512M # # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 127.0.0.1 # # * Fine Tuning # key_buffer = 16M max_allowed_packet = 16M thread_stack = 192K thread_cache_size = 8 # This replaces the startup script and checks MyISAM tables if needed # the first time they are touched myisam-recover = BACKUP #max_connections = 100 #table_cache = 64 #thread_concurrency = 10 # # * Query Cache Configuration # query_cache_limit = 1M query_cache_size = 16M # # * Logging and Replication # # Both location gets rotated by the cronjob. # Be aware that this log type is a performance killer. # As of 5.1 you can enable the log at runtime! #general_log_file = /var/log/mysql/mysql.log general_log=0 # # Error log - should be very few entries. # log_error = /var/log/mysql/error.log # # Here you can see queries with especially long duration #log_slow_queries = /var/log/mysql/mysql-slow.log #long_query_time = 2 #log-queries-not-using-indexes # # The following can be used as easy to replay backup logs or for replication. # note: if you are setting up a replication slave, see README.Debian about # other settings you may need to change. #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 1G #binlog_do_db = include_database_name #binlog_ignore_db = include_database_name # # * InnoDB # # InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/. # Read the manual for more InnoDB related options. There are many! # # * Security Features # # Read the manual, too, if you want chroot! # chroot = /var/lib/mysql/ # # For generating SSL certificates I recommend the OpenSSL GUI "tinyca". # # ssl-ca=/etc/mysql/cacert.pem # ssl-cert=/etc/mysql/server-cert.pem # ssl-key=/etc/mysql/server-key.pem [mysqldump] quick quote-names max_allowed_packet = 16M [mysql] #no-auto-rehash # faster start of mysql but no tab completition [isamchk] key_buffer = 16M # # * IMPORTANT: Additional settings that can override those from this file! # The files must end with '.cnf', otherwise they'll be ignored. # !includedir /etc/mysql/conf.d/ 

所以总的来说这些设置是推荐的,你会得到相当的性能优势。