Task

Module to handle a task.

class qarnot.task.Task(connection: Any, name: str, profile_or_pool: str | Pool | None = None, instancecount_or_range: int | str = 1, shortname: str | None = None, job: Any | None = None, scheduling_type: SchedulingType | None = None)[source]

Bases: object

Represents a Qarnot task.

Note

A Task must be created with qarnot.connection.Connection.create_task() or retrieved with qarnot.connection.Connection.tasks() or qarnot.connection.Connection.retrieve_task().

__init__(connection: Any, name: str, profile_or_pool: str | Pool | None = None, instancecount_or_range: int | str = 1, shortname: str | None = None, job: Any | None = None, scheduling_type: SchedulingType | None = None)[source]

Create a new Task.

Parameters:

  • connection (Connection) – the cluster on which to send the task

  • name (str) – given name of the task

  • profile_or_pool (str or Pool or None) – which profile to use with this task, or which Pool to run task,

  • instancecount_or_range (int or str) – number of instances or ranges on which to run task

  • shortname (str) – userfriendly task name

  • job (Job) – which job to attach the task to

  • logger (logging.Logger) – which job to attach the task to

run(output_dir: str | None = None, job_timeout: float | None = None, live_progress: bool = False, results_progress: bool | None = None, follow_state: bool = False, follow_stdout: bool = False, follow_stderr: bool = False) → None[source]

Submit a task, wait for the results and download them if required.

Parameters:

  • output_dir (str) – (optional) path to a directory that will contain the results

  • job_timeout (float) – (optional) Number of seconds before the task abort() if it is not already finished

  • live_progress (bool) – (optional) display a live progress

  • results_progress (bool or function(float, float, str)) – (optional) can be a callback (read,total,filename) or True to display a progress bar

  • follow_state (bool) – (optional) print the task’s state every time it changes

  • follow_stdout (bool) – (optional) refresh and print the task’s stdout at every iteration

  • follow_stderr (bool) – (optional) refresh and print the task’s stderr at every iteration

Raises:

Note

Will ensure all added file are on the resource bucket regardless of their uploading mode.

Note

If this function is interrupted (script killed for example), but the task is submitted, the task will still be executed remotely (results will not be downloaded)

Warning

Will override output_dir content.

resume(output_dir: str, job_timeout: float | None = None, live_progress: bool = False, results_progress: bool | None = None) → None[source]

Resume waiting for this task if it is still in submitted mode. Equivalent to wait() + download_results().

Parameters:

  • output_dir (str) – path to a directory that will contain the results

  • job_timeout (float) – Number of seconds before the task abort() if it is not already finished

  • live_progress (bool) – display a live progress

  • results_progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises:

Note

Do nothing if the task has not been submitted.

Warning

Will override output_dir content.

submit() → None[source]

Submit task to the cluster if it is not already submitted.

Raises:

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

Note

To get the results, call download_results() once the job is done.

abort() → None[source]

Abort this task if running.

Raises:
update_resources() → None[source]

Update resources for a running task.

The typical workflow is as follows:

  1. Upload new files on your resource bucket,

  2. Call this method,

  3. The new files will appear on all the compute nodes in the same resources folder as original resources

Note: There is no way to know when the files are effectively transfered. This information is available on the compute node only. Note: The update is additive only: files deleted from the bucket will NOT be deleted from the task’s resources directory.

Raises:
delete(purge_resources: bool = False, purge_results: bool = False) → None[source]

Delete this task on the server.

Parameters:

  • purge_resources (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

  • purge_results (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

Raises:
update(flushcache: bool = False) → None[source]

Update the task object from the REST Api. The flushcache parameter can be used to force the update, otherwise a cached version of the object will be served when accessing properties of the object. Some methods will flush the cache, like submit(), abort(), wait() and instant(). Cache behavior is configurable with auto_update and update_cache_time.

Raises:
classmethod from_json(connection: Any, json_task: Dict, is_summary: bool = False) → Any[source]

Create a Task object from a json task.

Parameters:
Returns:

The created Task.

commit() → None[source]

Replicate local changes on the current object instance to the REST API

Raises:

Note

When updating buckets’ properties, auto update will be disabled until commit is called.

wait(timeout: float | None = None, live_progress: bool = False, follow_state: bool = False, follow_stdout: bool = False, follow_stderr: bool = False) → bool[source]

Wait for this task until it is completed.

Parameters:

  • timeout (float) – maximum time (in seconds) to wait before returning (None => no timeout)

  • live_progress (bool) – display a live progress

  • follow_state (bool) – (optional) print the task’s state every time it changes

  • follow_stdout (bool) – (optional) refresh and print the task’s stdout at every iteration

  • follow_stderr (bool) – (optional) refresh and print the task’s stderr at every iteration

Return type:

bool

Returns:

Is the task finished

Raises:
snapshot(interval: int) → None[source]

Start snapshooting results. If called, this task’s results will be periodically updated, instead of only being available at the end.

Snapshots will be taken every interval second from the time the task is submitted.

Parameters:

interval (int) – the interval in seconds at which to take snapshots

Raises:

Note

To get the temporary results, call download_results().

instant() → None[source]

Make a snapshot of the current task.

Raises:

Note

To get the temporary results, call download_results().

property state
Type:

str

Getter:

return this task’s state

State of the task.

Value is in

  • UnSubmitted

  • Submitted

  • PartiallyDispatched

  • FullyDispatched

  • PartiallyExecuting

  • FullyExecuting

  • UploadingResults

  • DownloadingResults

  • PendingCancel

  • Cancelled

  • Success

  • Failure

  • PendingDelete

Warning

this is the state of the task when the object was retrieved, call update() for up to date value.

property resources
Type:

list(Bucket)

Getter:

Returns this task’s resources bucket

Setter:

Sets this task’s resources bucket

Represents resource files.

property results
Type:

Bucket

Getter:

Returns this task’s results bucket

Setter:

Sets this task’s results bucket

Represents results files.

download_results(output_dir: str, progress: bool | Callable[[float, float, str], None] | None = None) → None[source]

Download results in given output_dir.

Parameters:

  • output_dir (str) – local directory for the retrieved files.

  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises:

Warning

Will override output_dir content.

stdout(instanceId: int | None = None)[source]

Get the standard output of the task, or of a specific instance of the task, since its submission.

Return type:

str

Returns:

The standard output.

Raises:

Note

The buffer is circular, if stdout is too big, prefer calling fresh_stdout() regularly.

fresh_stdout(instanceId: int | None = None)[source]

Get what has been written on the standard output since last time the output of the task or of the instance was retrieved.

Return type:

str

Returns:

The new output since last call.

Raises:
stderr(instanceId: int | None = None)[source]

Get the standard error of the task, or of a specific instance of the task, since its submission.

Return type:

str

Returns:

The standard error.

Raises:

Note

The buffer is circular, if stderr is too big, prefer calling fresh_stderr() regularly.

fresh_stderr(instanceId: int | None = None)[source]

Get what has been written on the standard error since last time the standard error of the task or of its instance was retrieved.

Return type:

str

Returns:

The new error messages since last call.

Raises:
carbon_facts(datacenter_name: str | None = None) → CarbonFacts[source]

Get the carbon facts of the task

Parameters:

datacenter_name (str) – the name of the datacenter used as reference to compare carbon facts.

Return type:

CarbonFacts

Returns:

The carbon facts of the task.

Raises:
property uuid
Type:

str

Getter:

Returns this task’s uuid

The task’s uuid.

Automatically set when a task is submitted.

property name
Type:

str

Getter:

Returns this task’s name

Setter:

Sets this task’s name

The task’s name.

Can be set until task is submitted.

property shortname
Type:

str

Getter:

Returns this task’s shortname

Setter:

Sets this task’s shortname

The task’s shortname, must be DNS compliant and unique, if not provided, will default to uuid.

Can be set until task is submitted.

property tags
Type:

:class:list(str)

Getter:

Returns this task’s tags

Setter:

Sets this task’s tags

Custom tags.

property pool
Type:

Pool

Getter:

Returns this task’s pool

Setter:

Sets this task’s pool

The pool to run the task in.

Can be set until run() is called.

Warning

This property is mutually exclusive with profile

property profile
Type:

str

Getter:

Returns this task’s profile

Setter:

Sets this task’s profile

The profile to run the task with.

Can be set until run() or submit() is called.

Warning

This property is mutually exclusive with pool

property instancecount
Type:

int

Getter:

Returns this task’s instance count

Setter:

Sets this task’s instance count

Number of instances needed for the task.

Can be set until run() or submit() is called.

Raises:

AttributeError – if advanced_range is not None when setting this property

Warning

This property is mutually exclusive with advanced_range

property running_core_count
Type:

int

Getter:

Returns this task’s running core count

Number of core running inside the task.

property running_instance_count
Type:

int

Getter:

Returns this task’s running instance count

Number of instances running inside the task.

property advanced_range
Type:

str

Getter:

Returns this task’s advanced range

Setter:

Sets this task’s advanced range

Advanced instances range selection.

Allows to select which instances will be computed. Should be None or match the following extended regular expression “([0-9]+|[0-9]+-[0-9]+)(,([0-9]+|[0-9]+-[0-9]+))+” eg: 1,3-8,9,12-19

Can be set until run() is called.

Raises:

AttributeError – if instancecount is not 0 when setting this property

Warning

This property is mutually exclusive with instancecount

property snapshot_whitelist
Type:

str

Getter:

Returns this task’s snapshot whitelist

Setter:

Sets this task’s snapshot whitelist

Snapshot white list (regex) for snapshot() and instant()

Can be set until task is submitted.

property snapshot_blacklist
Type:

str

Getter:

Returns this task’s snapshot blacklist

Setter:

Sets this task’s snapshot blacklist

Snapshot black list (regex) for snapshot() instant()

Can be set until task is submitted.

property results_whitelist
Type:

str

Getter:

Returns this task’s results whitelist

Setter:

Sets this task’s results whitelist

Results whitelist (regex)

Can be set until task is submitted.

property results_blacklist
Type:

str

Getter:

Returns this task’s results blacklist

Setter:

Sets this task’s results blacklist

Results blacklist (regex)

Can be set until task is submitted.

property status
Type:

Status

Getter:

Returns this task’s status

Status of the task

property completed_instances
Type:

list(CompletedInstance)

Getter:

Return this task’s completed instances

property creation_date
Type:

datetime

Getter:

Returns this task’s creation date

Creation date of the task (UTC Time)

property errors
Type:

list(Error)

Getter:

Returns this task’s errors if any.

Error reason if any, empty string if none

property constants
Type:

dictionary{str : str}

Getter:

Returns this task’s constants dictionary.

Setter:

set the task’s constants dictionary.

Update the constants if needed. Constants are used to configure the profiles, set them to change your profile’s parameters.

property secrets_access_rights
Type:

SecretsAccessRights

Getter:

Returns the description of the secrets this task will have access to when running.

Setter:

set the secrets this task will have access to when running.

Secrets can be accessible either by exact match on the key or by using a prefix in order to match all the secrets starting with said prefix.

property constraints
Type:

dictionary{str : str}

Getter:

Returns this task’s constraints dictionary.

Setter:

set the task’s constraints dictionary.

Update the constraints if needed. advance usage

property forced_constants
Type:

dictionary{str : ForcedConstant}

Getter:

Returns this task’s forced constants dictionary.

Setter:

set the task’s forced constants dictionary.

Update the forced constants if needed. Forced constants are reserved for internal use.

property forced_network_rules
Type:

list{ForcedNetworkRule}

Getter:

Returns this task’s forced network rules list.

Setter:

set the task’s forced network rules list.

Update the forced network rules if needed. Forced network rules are reserved for internal use.

property labels
Type:

dictionary{str : str}

Getter:

Returns this task’s labels dictionary.

Setter:

Set the task’s labels dictionary.

Update the labels if needed. Labels are used to attach arbitrary key / value pairs to a task in order to find them later with greater ease. They do not affect the execution of a task.

property wait_for_pool_resources_synchronization
Type:

bool or None

Getter:

Returns this task’s wait_for_pool_resources_synchronization.

Setter:

set the task’s wait_for_pool_resources_synchronization.

Raises:

AttributeError – can’t set this attribute on a launched task

property auto_update
Type:

bool

Getter:

Returns this task’s auto update state

Setter:

Sets this task’s auto update state

Auto update state, default to True When auto update is disabled properties will always return cached value for the object and a call to update() will be required to get latest values from the REST Api.

property update_cache_time
Type:

int

Getter:

Returns this task’s auto update state

Setter:

Sets this task’s auto update state

Cache expiration time, default to 5s

property upload_results_on_cancellation: bool | None
Type:

bool, optional

Getter:

Whether task results will be uploaded upon task cancellation

Setter:

Set to True to upload results on task cancellation, False to make sure they won’t be uploaded, and keep to None to get the default behavior.

Raises:

AttributeError – trying to set this after the task is submitted

set_task_dependencies_from_uuids(uuids: str)[source]

Setter for the task dependencies using uuid

set_task_dependencies_from_tasks(tasks: List[Any])[source]

Setter for the task dependencies using tasks

property hardware_constraints
Type:

list, optional

Getter:

setup the hardware constraints

Setter:

Set up specific hardware constraints.

Raises:

AttributeError – trying to set this after the task is submitted

property default_resources_cache_ttl_sec: int | None
Type:

int, optional

Getter:

The default time to live used for all the task resources cache

Raises:

AttributeError – trying to set this after the task is submitted

property privileges: Privileges
Type:

Privileges

Getter:

The privileges granted to the task

Raises:

AttributeError – trying to set this after the task is submitted

allow_credentials_to_be_exported_to_task_environment()[source]

Grant privilege to export api and storage credentials to the task environment

property retry_settings: RetrySettings
Type:

RetrySettings

Getter:

The retry settings for the task instances

Raises:

AttributeError – trying to set this after the task is submitted

property scheduling_type: SchedulingType
Type:

SchedulingType

Getter:

The scheduling type for the task

Raises:

AttributeError – trying to set this after the task is submitted

property targeted_reserved_machine_key: str
Type:

str

Getter:

The reserved machine key when using the “reserved” scheduling type

Raises:

AttributeError – trying to set this after the task is submitted

property auto_delete

Autodelete this Task if it is finished and your max number of task is reach

Can be set until run() or submit() is called.

Type:

bool

Getter:

Returns is this task must autodelete

Setter:

Sets this task’s autodelete

Default_value:

“False”

Raises:

AttributeError – if you try to reset the auto_delete after the task is submit

property completion_ttl

The task will be auto delete completion_ttl after it is finished

Can be set until run() or submit() is called.

Getter:

Returns this task’s completed time to live.

Type:

str

Setter:

Sets this task’s this task’s completed time to live.

Type:

str or datetime.timedelta

Default_value:

“”

Raises:

AttributeError – if you try to set it after the task is submitted

The completion_ttl must be a timedelta or a time span format string (example: d.hh:mm:ss or hh:mm:ss )

property previous_state
Type:

str

Getter:

Returns the running task’s previous state

property state_transition_time
Type:

str

Getter:

Returns the running task’s transition state time

task state transition time (UTC Time)

property previous_state_transition_time
Type:

str

Getter:

Returns the running task’s previous transition state time

task previous state transition time (UTC Time)

property last_modified
Type:

str

Getter:

Returns the running task’s last modification time

task’s last modified time (UTC Time)

property execution_time
Type:

str

Getter:

Returns the running task’s total CPU execution time.

task’s execution time of all it’s instances.

property end_date
Type:

str

Getter:

Returns the finished task’s end date.

task’s end date (UTC Time)

property wall_time
Type:

str

Getter:

task’s wall time.

task’s wall time

property snapshot_interval
Type:

int

Getter:

task’s snapshot interval in seconds.

task’s snapshot interval in seconds.

property progress
Type:

float

Getter:

task’s progress in %.

task’s progress in %.

class qarnot.task.CompletedInstance(json: Dict[str, Any])[source]

Bases: object

Completed Instance Information

Note

Read-only class

__init__(json: Dict[str, Any])[source]
instance_id
Type:

int

Instance number.

state
Type:

str

Instance final state.

wall_time_sec
Type:

float

Instance wall time in seconds.

exec_time_sec
Type:

float

Execution time in seconds.

exec_time_sec_ghz
Type:

float

Execution time in seconds GHz.

peak_memory_mb
Type:

int

Peak memory size in MB.

average_ghz
Type:

float

Instance execution time GHz

results
Type:

:class:list(str)

Instance produced results

execution_attempt_count
Type:

int

Number of execution attempt of an instance, (manly in case of preemption).

class qarnot.task.BulkTaskResponse(json: Dict[str, Any])[source]

Bases: object

Bulk Task Response Information

Note

Read-only class

__init__(json: Dict[str, Any])[source]
status_code
Type:

int

Status code.

uuid
Type:

str

Created Task Uuid.

message
Type:

str

User friendly error message.

is_success()[source]

Check that the task submit has been successful.

Return type:

bool

Returns:

The task creation success(depending on received uuid and the status code).