Pools (nornir_pools)

The build manager and image registration packages make every attempt to scale well. To enable this the pool package is used to dispatch jobs to threads, processes, and other computers in a cluster.

nornir_pools aims to provide a consistent interface around four different multi-threading and clustering libraries available to Python.

The use pattern for pools is:

  1. Create a pool
  2. add a task or process to the pool
  3. save the task object returned
  4. call wait or wait_return on the task object to fetch the output or raise exceptions

Steps 3 and 4 can be skipped if output is not required. In this case wait_completion can be called on the pool to delay until all tasks have completed. Note that in this pattern exceptions may be lost.

Pool Creation

Pool creation functions share a common signature

Get<X>Pool([Poolname=None, num_threads=None)

Return a pool of X type, listed below. Repeated calls using the same name returns the same pool

  • Poolname (str) – Name of the pool to get or create. Passing “None” returns the global pool
  • num_threads (int) – Number of tasks allowed to execute concurrently. Not honored by all pools at this time

object derived from PoolBase

Return type:


nornir_pools.GetThreadPool(Poolname=None, num_threads=None)

Get or create a specific thread pool using vanilla python threads

nornir_pools.GetMultithreadingPool(Poolname=None, num_threads=None)

Get or create a specific thread pool to execute threads in other processes on the same computer using the multiprocessing library

nornir_pools.GetProcessPool(Poolname=None, num_threads=None)

Get or create a specific pool to invoke shell command processes on the same computer using the subprocess module

nornir_pools.GetParallelPythonPool(Poolname=None, num_threads=None)

Get or create a specific pool to invoke functions or shell command processes on a cluster using parallel python

Global pools

Most callers will not care about getting a specific pool. These functions always return the same pool.


Common pool for thread based tasks


Common pool for multithreading module tasks, threads run in different python processes to work around the global interpreter lock


Common pool for processes on the local machine


Get the common pool for placing tasks on the cluster

Pool Objects

class nornir_pools.poolbase.LocalThreadPoolBase(*args, **kwargs)

Base class for pools that rely on local threads and a queue to dispatch jobs


Wait for completion of all the tasks in the queue

class nornir_pools.poolbase.PoolBase(*args, **kwargs)

Pool objects provide the interface to create tasks on the pool.

add_process(name, func, *args, **kwargs)

Invoke a process on the pool. This function creates a task using name and then invokes pythons subprocess

  • name (str) – Friendly name of the task. Non-unique
  • func (function) – Process name to invoke using subprocess

task object

Return type:


add_task(name, func, *args, **kwargs)

Call a python function on the pool

  • name (str) – Friendly name of the task. Non-unique
  • func (function) – Python function pointer to invoke on the pool

task object

Return type:



The pool waits for all tasks to complete and frees any resources such as threads in a thread pool


Blocks until all tasks have completed

Task Objects

class nornir_pools.task.Task(name, *args, **kwargs)

Represents a task assigned to a pool. Responsible for allowing the caller to wait for task completion, raising any exceptions, and returning data from the call. Task objects are created by adding tasks or processes to the pools. They are not intended to be created directly by callers.


Non-blocking test to determine if task has completed. No exception is raised if the task raised an exception during execution until wait or wait_return is called.

Returns:True if the task is completed, otherwise False
Return type:bool

Wait for task to complete, does not return a value

Raises Exception:
 Exceptions raised during task execution are re-raised on the thread calling wait

Wait for task to complete and return the value

Returns:The output of the task function or the stdout text of a called process
Raises Exception:
 Exceptions raised during task execution are re-raised on the thread calling wait_return

Pool Destruction

It is not necessary to perform any cleanup. Functions to delete pools would not be hard to add. ClosePools is called automatically at script termination by atexit


Shutdown all pools.


On windows there is significant overhead to passing parameters to multiprocessing jobs. To address this I added pickle overrides to objects being marshalled. I also removed as many global initializations as I could from modules loaded by the tasks.