Keyboard Interruptions Handling in multiprocessing Pools with Python

Handling KeyboardInterrupts is crucial for managing parallel processes effectively. The multiprocessing library provides a way to create a pool of worker processes that can execute tasks concurrently. However, an issue arises when a KeyboardInterrupt is initiated during the execution of these processes.

Problem:

In the given Python code, a Pool is created and an attempt is made to handle KeyboardInterrupts using a try-except block. However, the corresponding code within the except block never executes, leaving the program hanging.

try:
    results = pool.map(slowly_square, range(40))
except KeyboardInterrupt:
    pool.terminate()
    print("You cancelled the program!")
    sys.exit(1)

Cause:

This problem is caused by a bug in Python where KeyboardInterrupts are not sent when waiting for a condition in threading.Condition.wait(). In the multiprocessing library, the Pool uses a condition variable to wait for the results of the map() operation. When a KeyboardInterrupt occurs, the condition variable wait() does not return, thus preventing the interrupt from being handled.

Solution:

To resolve this issue, a workaround is to specify a timeout when waiting for the results. This can be achieved by replacing the map() call with map_async().get(timeout), where timeout is set to a large value.

results = pool.map_async(slowly_square, range(40)).get(9999999)

By specifying a timeout, the condition variable wait() will return even if the KeyboardInterrupt has not been processed. This allows the except block to execute, enabling the termination of the pool and the graceful handling of the interrupt.