Pool¶

Module to handle a pool.

class qarnot.pool.Pool(connection, name, profile, instancecount=1, shortname=None, scheduling_type: Optional[qarnot.scheduling_type.SchedulingType] = None)[source]¶

Bases: object

Represents a Qarnot pool.

Note

A Pool must be created with qarnot.connection.Connection.create_pool() or retrieved with qarnot.connection.Connection.pools() or qarnot.connection.Connection.retrieve_pool().

__init__(connection, name, profile, instancecount=1, shortname=None, scheduling_type: Optional[qarnot.scheduling_type.SchedulingType] = None)[source]¶

Create a new Pool.

Parameters

connection (Connection) – the cluster on which to send the pool
name (str) – given name of the pool
profile (str) – which profile to use with this task
instancecount (int or str) – number of instances or ranges on which to run pool
shortname (str) – userfriendly pool name
logger (logging.Logger) – which job to attach the task to

classmethod from_json(connection, json_pool, is_summary=False)[source]¶

Create a Pool object from a json pool.

Parameters

connection (qarnot.connection.Connection) – the cluster connection
json_pool (dict) – Dictionary representing the pool

Returns

The created Pool.

submit()[source]¶

Submit pool to the cluster if it is not already submitted.

Raises

QarnotGenericException – API general error, see message for details
MaxPoolException – Pool quota reached
NotEnoughCreditsException – Not enough credits
UnauthorizedException – invalid credentials
MissingBucketException – resource bucket not found

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

update(flushcache=False)[source]¶

Update the pool object from the REST Api. The flushcache parameter can be used to force the update, otherwise a cached version of the object will be served when accessing properties of the object. Cache behavior is configurable with auto_update and update_cache_time.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not represent a valid one

commit()[source]¶

Replicate local changes on the current object instance to the REST API

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist

This function need to be call to apply the local elastic pool setting modifications. .. note:: When updating buckets’ properties, auto update will be disabled until commit is called.

update_resources()[source]¶

Update resources for a running pool.

The typical workflow is as follows:

Upload new files on your resource bucket,
Call this method,
The new files will appear on all the compute nodes in the same resources folder as original resources

Note: There is no way to know when the files are effectively transfered. This information is available on the compute node only. Note: The update is additive only: files deleted from the bucket will NOT be deleted from the pool’s resources directory.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist

setup_elastic(minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90)[source]¶

Setup the pool elastic properties

Parameters

minimum_total_slots (int) – Minimum slot number for the pool in elastic mode. Defaults to 0.
maximum_total_slots (int) – Maximum slot number for the pool in elastic mode. Defaults to 1.
minimum_idle_slots (int) – Minimum idling slot number. Defaults to 0.
minimum_idle_time_seconds (int) – Wait time in seconds before closing an unused slot if the number of unused slots are upper than the minimum_idle_slots. Defaults to 0.
resize_factor (float) – Growing factor of the pool. It must be a number between 0 and 1. Defaults to 1.
resize_period (int) – Refresh rate of resizing the pool in elastic mode. Defaults to 90.

delete(purge_resources=False)[source]¶

Delete this pool on the server.

Parameters

purge_resources (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist

close()[source]¶

Close this pool if running.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
UnauthorizedException – invalid operation on non running pool
MissingPoolException – pool does not exist

stdout(instanceId: Optional[int] = None)[source]¶

Get the standard output of the pool, or of a specific instance of the pool, since the submission of the pool.

Return type

str

Returns

The standard output.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist
MissingPoolInstanceException – pool instance does not exist

Note

The buffer is circular, if stdout is too big, prefer calling fresh_stdout() regularly.

fresh_stdout(instanceId: Optional[int] = None)[source]¶

Get what has been written on the standard output since last time the output of the pool or of the instance was retrieved.

Return type

str

Returns

The new output since last call.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist
MissingPoolInstanceException – pool instance does not exist

stderr(instanceId: Optional[int] = None)[source]¶

Get the standard error of the pool, or of a specific instance of the pool, since the submission of the pool.

Return type

str

Returns

The standard error.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist
MissingPoolInstanceException – pool instance does not exist

Note

The buffer is circular, if stderr is too big, prefer calling fresh_stderr() regularly.

fresh_stderr(instanceId: Optional[int] = None)[source]¶

Get what has been written on the standard error since last time the standard error of the pool or of its instance was retrieved.

Return type

str

Returns

The new error messages since last call.

Raises

QarnotGenericException – API general error, see message for details
UnauthorizedException – invalid credentials
MissingPoolException – pool does not exist
MissingPoolInstanceException – pool instance does not exist

property uuid¶

Type: str
Getter: Returns this pool’s uuid

The pool’s uuid.

Automatically set when a pool is submitted.

property state¶

Type: str
Getter: return this pool’s state

State of the pool.

Value is in

UnSubmitted
Submitted
PartiallyDispatched
FullyDispatched
PartiallyExecuting
FullyExecuting
Closing
Closed
Failure
PendingDelete

Warning

this is the state of the pool when the object was retrieved, call update() for up to date value.

property resources¶

Type: list(Bucket)
Getter: Returns this pool’s resources bucket
Setter: Sets this pool’s resources bucket

Represents resource files.

property name¶

Type: str
Getter: Returns this pool’s name
Setter: Sets this pool’s name

The pool’s name.

Can be set until pool is submitted.

property shortname¶

Type: str
Getter: Returns this pool’s shortname
Setter: Sets this pool’s shortname

The pool’s shortname, must be DNS compliant and unique, if not provided, will default to uuid.

Can be set until pool is submitted.

property tags¶

Type: :class:list(str)
Getter: Returns this pool’s tags
Setter: Sets this pool’s tags

Custom tags.

property profile¶

Type: str
Getter: Returns this pool’s profile
Setter: Sets this pool’s profile

The profile to run the pool with.

Can be set until submit() is called.

property instancecount¶

Type: int
Getter: Returns this pool’s instance count
Setter: Sets this pool’s instance count

Number of instances needed for the pool.

Can be set until submit() is called.

property running_core_count¶

Type: int
Getter: Returns this pool’s running core count

Number of core running inside the pool.

property running_instance_count¶

Type: int
Getter: Returns this pool’s running instance count

Number of instances running inside the pool.

property errors¶

Type: list(str)
Getter: Returns this pool’s error list

Error reason if any, empty string if none

property creation_date¶

Type: str
Getter: Returns this pool’s creation date

Creation date of the pool (UTC Time)

property status¶

Type: Status
Getter: Returns this pool’s status

Status of the pool

property auto_update¶

Type: bool
Getter: Returns this pool’s auto update state
Setter: Sets this pool’s auto update state

Auto update state, default to True When auto update is disabled properties will always return cached value for the object and a call to update() will be required to get latest values from the REST Api.

property update_cache_time¶

Type: int
Getter: Returns this pool’s auto update state
Setter: Sets this pool’s auto update state

Cache expiration time, default to 5s

property is_elastic¶

Type: bool
Getter: Returns this pool’s is_elastic
Setter: Sets this pool’s is_elastic

Define if you use a static or an elastic pool.

property elastic_minimum_slots¶

Type: int
Getter: Returns this pool’s elastic_minimum_slots
Setter: Sets this pool’s elastic_minimum_slots

The minimum slot number of the elastic pool. Define the minimum number of pool instances stay open during the pool execution.

property elastic_maximum_slots¶

Type: int
Getter: Returns this pool’s elastic_maximum_slots
Setter: Sets this pool’s elastic_maximum_slots

The maximum slot number of the elastic pool. Define the maximum number of pool instances opened during the pool execution.

property elastic_minimum_idle_slots¶

Type: int
Getter: Returns this pool’s elastic_minimum_idle_slots
Setter: Sets this pool’s elastic_minimum_idle_slots

The minimum idle number of the elastic pool. Define the minimum number of the idle pool instances stay opened during the pool execution. It should be lower to elastic_minimum_slots to be usefull

property elastic_minimum_idle_time¶

Type: int
Getter: Returns this pool’s elastic_minimum_idle_time
Setter: Sets this pool’s elastic_minimum_idle_time

Wait time in seconds before closing an unused slot if the number of unused slots are upper than the minimum_idle_slots.

property elastic_resize_factor¶

Type: float
Getter: Returns this pool’s elastic_resize_factor
Setter: Sets this pool’s elastic_resize_factor

The resize factor of the pool. It represent the resize factor of the slots. It’s a decimal number upper than 0 and and equal or lower the 1

property elastic_resize_period¶

Type: int
Getter: Returns this pool’s elastic_resize_period
Setter: Sets this pool’s elastic_resize_period

The resize period of the elastic pool in second. This is the refresh rate of resizing the elastic pool.

property preparation_command_line¶

Type: str:
Getter: Returns this pool’s command line.
Setter: set the pool’s command line.

Update the pool command line if needed The command line is a command executed on the node before any task is executed.

property constants¶

Type: dictionary{str : str}
Getter: Returns this pool’s constants dictionary.
Setter: set the pool’s constants dictionary.

Update the constants if needed. Constants are used to configure the profiles, set them to change your profile’s parameters.

property constraints¶

Type: dictionary{str : str}
Getter: Returns this pool’s constraints dictionary.
Setter: set the pool’s constraints dictionary.

Update the constraints if needed advance usage

property forced_network_rules¶

Type: list{ForcedNetworkRule}
Getter: Returns this pool’s forced network rules list.
Setter: set the pool’s forced network rules list.

Update the forced network rules if needed. Forced network rules are reserved for internal use.

property labels¶

Type: dictionary{str : str}
Getter: Return this pool’s labels dictionary.
Setter: set the pool’s labels constraints dictionary.

Labels are used to attach arbitrary key / value pairs to a pool in order to find them later with greater ease. They do not affect the execution of a pool.

property tasks_default_wait_for_pool_resources_synchronization¶

Type: bool
Getter: Returns this task’s tasks_default_wait_for_pool_resources_synchronization.
Setter: set the task’s tasks_default_wait_for_pool_resources_synchronization.
Raises: AttributeError – can’t set this attribute on a launched task

property auto_delete¶

Autodelete this pool if it is finished and your max number of pool is reach

Can be set until submit() is called.

Type: bool
Getter: Returns is this pool must autodelete
Setter: Sets this pool’s autodelete
Default_value: “False”
Raises: AttributeError – if you try to reset the auto_delete after the pool is submit

property completion_ttl¶

The pool will be auto delete completion_ttl after it is finished

Can be set until submit() is called.

Getter: Returns this pool’s completed time to live.
Type: str
Setter: Sets this pool’s this pool’s completed time to live.
Type: str or datetime.timedelta
Default_value: “”
Raises: AttributeError – if you try to set it after the pool is submitted

The completion_ttl must be a timedelta or a time span format string (example: d.hh:mm:ss or hh:mm:ss )

property previous_state¶

Type: str
Getter: Returns the running pool’s previous state

property state_transition_time¶

Type: str
Getter: Returns the running pool’s transition state time

pool state transition time (UTC Time)

property previous_state_transition_time¶

Type: str
Getter: Returns the running pool’s previous transition state time

pool previous state transition time (UTC Time)

property last_modified¶

Type: str
Getter: Returns the running pool’s last modification time

pool’s last modified time (UTC Time)

property execution_time¶

Type: str
Getter: Returns the running pool’s total CPU execution time.

pool’s execution time of all it’s instances.

property end_date¶

Type: str
Getter: Returns the finished pool’s end date.

pool’s end date (UTC Time)

property pool_usage¶

Type: float
Getter: Returns the pool usage

property total_slot_capacity¶

Type: int
Getter: Returns the pool slot capacity

property queued_or_running_task_instances_count¶

Type: int
Getter: Returns count of task instances dispatched in the pool

property hardware_constraints¶

Type: list, optional
Getter: setup the hardware constraints
Setter: Set up specific hardware constraints.
Raises: AttributeError – trying to set this after the task is submitted

property default_resources_cache_ttl_sec: Optional[int]¶

Type: int, optional
Getter: The default time to live used for all the pool resources cache
Raises: AttributeError – trying to set this after the task is submitted

property privileges: qarnot.privileges.Privileges¶

Type: Privileges
Getter: The privileges granted to the pool
Raises: AttributeError – trying to set this after the pool is submitted

allow_credentials_to_be_exported_to_pool_environment()[source]¶: Grant privilege to export api and storage credentials to the pool environment

property default_retry_settings: qarnot.retry_settings.RetrySettings¶

Type: RetrySettings
Getter: The default retry settings applied to the pool’s tasks
Raises: AttributeError – trying to set this after the task is submitted

property scheduling_type: qarnot.scheduling_type.SchedulingType¶

Type: SchedulingType
Getter: The scheduling type for the pool
Raises: AttributeError – trying to set this after the pool is submitted

property targeted_reserved_machine_key: str¶

Type: str
Getter: The reserved machine key when using the “reserved” scheduling type
Raises: AttributeError – trying to set this after the pool is submitted