Pools (nornir_pools)

The build manager and image registration packages make every attempt to scale well. To enable this the pool package is used to dispatch jobs to threads, processes, and other computers in a cluster.

nornir_pools aims to provide a consistent interface around four different multi-threading and clustering libraries available to Python.

The use pattern for pools is:

  1. Create a pool

  2. add a task or process to the pool

  3. save the task object returned

  4. call wait or wait_return on the task object to fetch the output or raise exceptions

Steps 3 and 4 can be skipped if output is not required. In this case wait_completion can be called on the pool to delay until all tasks have completed. Note that in this pattern exceptions may be lost.

Pool Creation

Pool creation functions share a common signature

Get<X>Pool([Poolname=None, num_threads=None)

Return a pool of X type, listed below. Repeated calls using the same name returns the same pool

Parameters

  • Poolname (str) – Name of the pool to get or create. Passing “None” returns the global pool

  • num_threads (int) – Number of tasks allowed to execute concurrently. Not honored by all pools at this time

Returns

object derived from PoolBase

Return type

PoolBase

nornir_pools.GetThreadPool(Poolname=None, num_threads=None)

Get or create a specific thread pool using vanilla python threads

nornir_pools.GetMultithreadingPool(Poolname=None, num_threads=None)

Get or create a specific thread pool to execute threads in other processes on the same computer using the multiprocessing library

nornir_pools.GetProcessPool(Poolname=None, num_threads=None)

Get or create a specific pool to invoke shell command processes on the same computer using the subprocess module

nornir_pools.GetParallelPythonPool(Poolname=None, num_threads=None)

Get or create a specific pool to invoke functions or shell command processes on a cluster using parallel python

Global pools

Most callers will not care about getting a specific pool. These functions always return the same pool.

nornir_pools.GetGlobalThreadPool()

Common pool for thread based tasks

nornir_pools.GetGlobalMultithreadingPool()

Common pool for multithreading module tasks, threads run in different python processes to work around the global interpreter lock

nornir_pools.GetGlobalProcessPool()

Common pool for processes on the local machine

nornir_pools.GetGlobalClusterPool()

Get the common pool for placing tasks on the cluster

Pool Objects

class nornir_pools.poolbase.LocalThreadPoolBase(*args, **kwargs)

Base class for pools that rely on local threads and a queue to dispatch jobs

shutdown()

The pool waits for all tasks to complete and frees any resources such as threads in a thread pool

wait_completion()

Wait for completion of all the tasks in the queue

class nornir_pools.poolbase.PoolBase(*args, **kwargs)

Pool objects provide the interface to create tasks on the pool.

TryReportActiveTaskCount()

Report the current job count if we haven’t reported it recently

add_process(name, func, *args, **kwargs)

Invoke a process on the pool. This function creates a task using name and then invokes pythons subprocess

Parameters

  • name (str) – Friendly name of the task. Non-unique

  • func (function) – Process name to invoke using subprocess

Returns

task object

Return type

task

add_task(name, func, *args, **kwargs)

Call a python function on the pool

Parameters

  • name (str) – Friendly name of the task. Non-unique

  • func (function) – Python function pointer to invoke on the pool

Returns

task object

Return type

task

shutdown()

The pool waits for all tasks to complete and frees any resources such as threads in a thread pool

wait_completion()

Blocks until all tasks have completed

Task Objects

class nornir_pools.task.Task(name, *args, **kwargs)

Represents a task assigned to a pool. Responsible for allowing the caller to wait for task completion, raising any exceptions, and returning data from the call. Task objects are created by adding tasks or processes to the pools. They are not intended to be created directly by callers.

property iscompleted

Non-blocking test to determine if task has completed. No exception is raised if the task raised an exception during execution until wait or wait_return is called.

Returns

True if the task is completed, otherwise False

Return type

bool

wait()

Wait for task to complete, does not return a value

Raises

Exception – Exceptions raised during task execution are re-raised on the thread calling wait

wait_return()

Wait for task to complete and return the value

Returns

The output of the task function or the stdout text of a called process

Raises

Exception – Exceptions raised during task execution are re-raised on the thread calling wait_return

Pool Destruction

It is not necessary to perform any cleanup. Functions to delete pools would not be hard to add. ClosePools is called automatically at script termination by atexit

nornir_pools.ClosePools()

Shutdown all pools.

Optimization

On windows there is significant overhead to passing parameters to multiprocessing jobs. To address this I added pickle overrides to objects being marshalled. I also removed as many global initializations as I could from modules loaded by the tasks.