Pool

Module to handle a pool.

class qarnot.pool.Pool(connection, name, profile, instancecount=1, shortname=None, scheduling_type: SchedulingType | None = None)[source]

Bases: object

Represents a Qarnot pool.

Note

A Pool must be created with qarnot.connection.Connection.create_pool() or retrieved with qarnot.connection.Connection.pools() or qarnot.connection.Connection.retrieve_pool().

__init__(connection, name, profile, instancecount=1, shortname=None, scheduling_type: SchedulingType | None = None)[source]

Create a new Pool.

Parameters:

  • connection (Connection) – the cluster on which to send the pool

  • name (str) – given name of the pool

  • profile (str) – which profile to use with this task

  • instancecount (int or str) – number of instances or ranges on which to run pool

  • shortname (str) – userfriendly pool name

  • logger (logging.Logger) – which job to attach the task to

classmethod from_json(connection, json_pool, is_summary=False)[source]

Create a Pool object from a json pool.

Parameters:
Returns:

The created Pool.

submit()[source]

Submit pool to the cluster if it is not already submitted.

Raises:

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

update(flushcache=False)[source]

Update the pool object from the REST Api. The flushcache parameter can be used to force the update, otherwise a cached version of the object will be served when accessing properties of the object. Cache behavior is configurable with auto_update and update_cache_time.

Raises:
commit()[source]

Replicate local changes on the current object instance to the REST API

Raises:

This function need to be call to apply the local elastic pool setting modifications. .. note:: When updating buckets’ properties, auto update will be disabled until commit is called.

update_resources()[source]

Update resources for a running pool.

The typical workflow is as follows:

  1. Upload new files on your resource bucket,

  2. Call this method,

  3. The new files will appear on all the compute nodes in the same resources folder as original resources

Note: There is no way to know when the files are effectively transfered. This information is available on the compute node only. Note: The update is additive only: files deleted from the bucket will NOT be deleted from the pool’s resources directory.

Raises:
setup_elastic(minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90)[source]

Setup the pool elastic properties

Parameters:

  • minimum_total_slots (int) – Minimum slot number for the pool in elastic mode. Defaults to 0.

  • maximum_total_slots (int) – Maximum slot number for the pool in elastic mode. Defaults to 1.

  • minimum_idle_slots (int) – Minimum idling slot number. Defaults to 0.

  • minimum_idle_time_seconds (int) – Wait time in seconds before closing an unused slot if the number of unused slots are upper than the minimum_idle_slots. Defaults to 0.

  • resize_factor (float) – Growing factor of the pool. It must be a number between 0 and 1. Defaults to 1.

  • resize_period (int) – Refresh rate of resizing the pool in elastic mode. Defaults to 90.

delete(purge_resources=False)[source]

Delete this pool on the server.

Parameters:

purge_resources (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

Raises:
close()[source]

Close this pool if running.

Raises:
stdout(instanceId: int | None = None)[source]

Get the standard output of the pool, or of a specific instance of the pool, since the submission of the pool.

Return type:

str

Returns:

The standard output.

Raises:

Note

The buffer is circular, if stdout is too big, prefer calling fresh_stdout() regularly.

fresh_stdout(instanceId: int | None = None)[source]

Get what has been written on the standard output since last time the output of the pool or of the instance was retrieved.

Return type:

str

Returns:

The new output since last call.

Raises:
stderr(instanceId: int | None = None)[source]

Get the standard error of the pool, or of a specific instance of the pool, since the submission of the pool.

Return type:

str

Returns:

The standard error.

Raises:

Note

The buffer is circular, if stderr is too big, prefer calling fresh_stderr() regularly.

fresh_stderr(instanceId: int | None = None)[source]

Get what has been written on the standard error since last time the standard error of the pool or of its instance was retrieved.

Return type:

str

Returns:

The new error messages since last call.

Raises:
property uuid
Type:

str

Getter:

Returns this pool’s uuid

The pool’s uuid.

Automatically set when a pool is submitted.

property state
Type:

str

Getter:

return this pool’s state

State of the pool.

Value is in

  • UnSubmitted

  • Submitted

  • PartiallyDispatched

  • FullyDispatched

  • PartiallyExecuting

  • FullyExecuting

  • Closing

  • Closed

  • Failure

  • PendingDelete

Warning

this is the state of the pool when the object was retrieved, call update() for up to date value.

property resources
Type:

list(Bucket)

Getter:

Returns this pool’s resources bucket

Setter:

Sets this pool’s resources bucket

Represents resource files.

property name
Type:

str

Getter:

Returns this pool’s name

Setter:

Sets this pool’s name

The pool’s name.

Can be set until pool is submitted.

property shortname
Type:

str

Getter:

Returns this pool’s shortname

Setter:

Sets this pool’s shortname

The pool’s shortname, must be DNS compliant and unique, if not provided, will default to uuid.

Can be set until pool is submitted.

property tags
Type:

:class:list(str)

Getter:

Returns this pool’s tags

Setter:

Sets this pool’s tags

Custom tags.

property profile
Type:

str

Getter:

Returns this pool’s profile

Setter:

Sets this pool’s profile

The profile to run the pool with.

Can be set until submit() is called.

property instancecount
Type:

int

Getter:

Returns this pool’s instance count

Setter:

Sets this pool’s instance count

Number of instances needed for the pool.

Can be set until submit() is called.

property running_core_count
Type:

int

Getter:

Returns this pool’s running core count

Number of core running inside the pool.

property running_instance_count
Type:

int

Getter:

Returns this pool’s running instance count

Number of instances running inside the pool.

property errors
Type:

list(str)

Getter:

Returns this pool’s error list

Error reason if any, empty string if none

property creation_date
Type:

str

Getter:

Returns this pool’s creation date

Creation date of the pool (UTC Time)

property status
Type:

Status

Getter:

Returns this pool’s status

Status of the pool

property auto_update
Type:

bool

Getter:

Returns this pool’s auto update state

Setter:

Sets this pool’s auto update state

Auto update state, default to True When auto update is disabled properties will always return cached value for the object and a call to update() will be required to get latest values from the REST Api.

property update_cache_time
Type:

int

Getter:

Returns this pool’s auto update state

Setter:

Sets this pool’s auto update state

Cache expiration time, default to 5s

property is_elastic
Type:

bool

Getter:

Returns this pool’s is_elastic

Setter:

Sets this pool’s is_elastic

Define if you use a static or an elastic pool.

property elastic_minimum_slots
Type:

int

Getter:

Returns this pool’s elastic_minimum_slots

Setter:

Sets this pool’s elastic_minimum_slots

The minimum slot number of the elastic pool. Define the minimum number of pool instances stay open during the pool execution.

property elastic_maximum_slots
Type:

int

Getter:

Returns this pool’s elastic_maximum_slots

Setter:

Sets this pool’s elastic_maximum_slots

The maximum slot number of the elastic pool. Define the maximum number of pool instances opened during the pool execution.

property elastic_minimum_idle_slots
Type:

int

Getter:

Returns this pool’s elastic_minimum_idle_slots

Setter:

Sets this pool’s elastic_minimum_idle_slots

The minimum idle number of the elastic pool. Define the minimum number of the idle pool instances stay opened during the pool execution. It should be lower to elastic_minimum_slots to be usefull

property elastic_minimum_idle_time
Type:

int

Getter:

Returns this pool’s elastic_minimum_idle_time

Setter:

Sets this pool’s elastic_minimum_idle_time

Wait time in seconds before closing an unused slot if the number of unused slots are upper than the minimum_idle_slots.

property elastic_resize_factor
Type:

float

Getter:

Returns this pool’s elastic_resize_factor

Setter:

Sets this pool’s elastic_resize_factor

The resize factor of the pool. It represent the resize factor of the slots. It’s a decimal number upper than 0 and and equal or lower the 1

property elastic_resize_period
Type:

int

Getter:

Returns this pool’s elastic_resize_period

Setter:

Sets this pool’s elastic_resize_period

The resize period of the elastic pool in second. This is the refresh rate of resizing the elastic pool.

property preparation_command_line
Type:

str:

Getter:

Returns this pool’s command line.

Setter:

set the pool’s command line.

Update the pool command line if needed The command line is a command executed on the node before any task is executed.

property constants
Type:

dictionary{str : str}

Getter:

Returns this pool’s constants dictionary.

Setter:

set the pool’s constants dictionary.

Update the constants if needed. Constants are used to configure the profiles, set them to change your profile’s parameters.

property constraints
Type:

dictionary{str : str}

Getter:

Returns this pool’s constraints dictionary.

Setter:

set the pool’s constraints dictionary.

Update the constraints if needed advance usage

property secrets_access_rights
Type:

SecretsAccessRights

Getter:

Returns the description of the secrets the tasks in this pool will have access to when running.

Setter:

set the secrets this pool will have access to when running.

Secrets can be accessible either by exact match on the key or by using a prefix in order to match all the secrets starting with said prefix.

property forced_network_rules
Type:

list{ForcedNetworkRule}

Getter:

Returns this pool’s forced network rules list.

Setter:

set the pool’s forced network rules list.

Update the forced network rules if needed. Forced network rules are reserved for internal use.

property labels
Type:

dictionary{str : str}

Getter:

Return this pool’s labels dictionary.

Setter:

set the pool’s labels constraints dictionary.

Labels are used to attach arbitrary key / value pairs to a pool in order to find them later with greater ease. They do not affect the execution of a pool.

property tasks_default_wait_for_pool_resources_synchronization
Type:

bool

Getter:

Returns this task’s tasks_default_wait_for_pool_resources_synchronization.

Setter:

set the task’s tasks_default_wait_for_pool_resources_synchronization.

Raises:

AttributeError – can’t set this attribute on a launched task

property auto_delete

Autodelete this pool if it is finished and your max number of pool is reach

Can be set until submit() is called.

Type:

bool

Getter:

Returns is this pool must autodelete

Setter:

Sets this pool’s autodelete

Default_value:

“False”

Raises:

AttributeError – if you try to reset the auto_delete after the pool is submit

property completion_ttl

The pool will be auto delete completion_ttl after it is finished

Can be set until submit() is called.

Getter:

Returns this pool’s completed time to live.

Type:

str

Setter:

Sets this pool’s this pool’s completed time to live.

Type:

str or datetime.timedelta

Default_value:

“”

Raises:

AttributeError – if you try to set it after the pool is submitted

The completion_ttl must be a timedelta or a time span format string (example: d.hh:mm:ss or hh:mm:ss )

property previous_state
Type:

str

Getter:

Returns the running pool’s previous state

property state_transition_time
Type:

str

Getter:

Returns the running pool’s transition state time

pool state transition time (UTC Time)

property previous_state_transition_time
Type:

str

Getter:

Returns the running pool’s previous transition state time

pool previous state transition time (UTC Time)

property last_modified
Type:

str

Getter:

Returns the running pool’s last modification time

pool’s last modified time (UTC Time)

property execution_time
Type:

str

Getter:

Returns the running pool’s total CPU execution time.

pool’s execution time of all it’s instances.

property end_date
Type:

str

Getter:

Returns the finished pool’s end date.

pool’s end date (UTC Time)

property pool_usage
Type:

float

Getter:

Returns the pool usage

property total_slot_capacity
Type:

int

Getter:

Returns the pool slot capacity

property queued_or_running_task_instances_count
Type:

int

Getter:

Returns count of task instances dispatched in the pool

property hardware_constraints
Type:

list, optional

Getter:

setup the hardware constraints

Setter:

Set up specific hardware constraints.

Raises:

AttributeError – trying to set this after the task is submitted

property default_resources_cache_ttl_sec: int | None
Type:

int, optional

Getter:

The default time to live used for all the pool resources cache

Raises:

AttributeError – trying to set this after the task is submitted

property privileges: Privileges
Type:

Privileges

Getter:

The privileges granted to the pool

Raises:

AttributeError – trying to set this after the pool is submitted

allow_credentials_to_be_exported_to_pool_environment()[source]

Grant privilege to export api and storage credentials to the pool environment

property default_retry_settings: RetrySettings
Type:

RetrySettings

Getter:

The default retry settings applied to the pool’s tasks

Raises:

AttributeError – trying to set this after the task is submitted

property scheduling_type: SchedulingType
Type:

SchedulingType

Getter:

The scheduling type for the pool

Raises:

AttributeError – trying to set this after the pool is submitted

property targeted_reserved_machine_key: str
Type:

str

Getter:

The reserved machine key when using the “reserved” scheduling type

Raises:

AttributeError – trying to set this after the pool is submitted