在Linux / Python2.7的终止()后,多处理器。池中产生新的childern?

我有一个可执行文件,我需要经常运行,使用不同的参数。 为此,我使用处理模块编写了一个小Python(2.7)包装器,遵循这里给出的模式。

我的代码如下所示:

try: logging.info("starting pool runs") pool.map(run_nlin, params) pool.close() except KeyboardInterrupt: logging.info("^C pressed") pool.terminate() except Exception, e: logging.info("exception caught: ", e) pool.terminate() finally: time.sleep(5) pool.join() logging.info("done") 

我的工作人员function在这里:

 class KeyboardInterruptError(Exception): pass def run_nlin((path_config, path_log, path_nlin, update_method)): try: with open(path_log, "w") as log_: cmdline = [path_nlin, path_config] if update_method: cmdline += [update_method, ] sp.call(cmdline, stdout=log_, stderr=log_) except KeyboardInterrupt: time.sleep(5) raise KeyboardInterruptError() except: raise 

path_config是二进制程序configuration文件的path; 在那里有例如运行程序的date。

当我开始包装,一切都很好。 但是,当我按^C ,包装脚本似乎在终止之前从池中启动一个额外的numproc进程。 例如,当我启动第1-10天的脚本时,可以在ps aux输出中看到二进制程序的两个实例正在运行(通常为第1天和第3天)。 现在,当我按^C ,包装脚本退出,天1和3的二进制程序已经消失,但有一些新的二进制程序在第5天和第7天运行。

所以对我来说,似乎Pool在最后死亡之前启动另一个numproc进程。

有什么想法发生在这里,我能做些什么呢?

Solutions Collecting From Web of "在Linux / Python2.7的终止()后,多处理器。池中产生新的childern?"

在这个页面上 ,多处理模块的作者Jesse Noller展示了处理KeyboardInterrupt的正确方法是让子进程返回 – 而不是重新规定异常。 这允许主进程终止池。

但是,如下面的代码所示,主程序所有由pool.map生成的任务都运行之后才会到达except KeyboardInterrupt区块except KeyboardInterrupt区块。 这就是为什么(我相信)在Ctrl-C被按下之后,你会看到额外的对你的工作函数run_nlin调用。

一种可能的解决方法是如果已经设置了multiprocessing.Event那么让所有的worker函数都测试一下。 如果事件已经确定,那么让工人提前退出,否则,长期计算。


 import logging import multiprocessing as mp import time logger = mp.log_to_stderr(logging.WARNING) def worker(x): try: if not terminating.is_set(): logger.warn("Running worker({x!r})".format(x = x)) time.sleep(3) else: logger.warn("got the message... we're terminating!") except KeyboardInterrupt: logger.warn("terminating is set") terminating.set() return x def initializer(terminating_): # This places terminating in the global namespace of the worker subprocesses. # This allows the worker function to access `terminating` even though it is # not passed as an argument to the function. global terminating terminating = terminating_ def main(): terminating = mp.Event() result = [] pool = mp.Pool(initializer=initializer, initargs=(terminating, )) params = range(12) try: logger.warn("starting pool runs") result = pool.map(worker, params) pool.close() except KeyboardInterrupt: logger.warn("^C pressed") pool.terminate() finally: pool.join() logger.warn('done: {r}'.format(r = result)) if __name__ == '__main__': main() 

运行脚本产生:

 % test.py [WARNING/MainProcess] starting pool runs [WARNING/PoolWorker-1] Running worker(0) [WARNING/PoolWorker-2] Running worker(1) [WARNING/PoolWorker-3] Running worker(2) [WARNING/PoolWorker-4] Running worker(3) 

这里Ctrl-C被按下; 每个工人都设置terminating事件。 我们真的只需要一个来设置它,但是尽管效率不高,它仍然有效。

  Cc Cc[WARNING/PoolWorker-4] terminating is set [WARNING/PoolWorker-2] terminating is set [WARNING/PoolWorker-3] terminating is set [WARNING/PoolWorker-1] terminating is set 

现在,运行由pool.map排队的所有其他任务:

 [WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-3] got the message... we're terminating! 

最后主进程到达except KeyboardInterrupt块外。

 [WARNING/MainProcess] ^C pressed [WARNING/MainProcess] done: []