Task

Module to handle a task.

class qarnot.task.Task(connection: Any, name: str, profile_or_pool: Optional[Union[str, qarnot.pool.Pool]] = None, instancecount_or_range: Union[int, str] = 1, shortname: Optional[str] = None, job: Optional[Any] = None, scheduling_type: Optional[qarnot.scheduling_type.SchedulingType] = None)[source]

Bases: object

Represents a Qarnot task.

Note

A Task must be created with qarnot.connection.Connection.create_task() or retrieved with qarnot.connection.Connection.tasks() or qarnot.connection.Connection.retrieve_task().

__init__(connection: Any, name: str, profile_or_pool: Optional[Union[str, qarnot.pool.Pool]] = None, instancecount_or_range: Union[int, str] = 1, shortname: Optional[str] = None, job: Optional[Any] = None, scheduling_type: Optional[qarnot.scheduling_type.SchedulingType] = None)[source]

Create a new Task.

Parameters

  • connection (Connection) – the cluster on which to send the task

  • name (str) – given name of the task

  • profile_or_pool (str or Pool or None) – which profile to use with this task, or which Pool to run task,

  • instancecount_or_range (int or str) – number of instances or ranges on which to run task

  • shortname (str) – userfriendly task name

  • job (Job) – which job to attach the task to

  • logger (logging.Logger) – which job to attach the task to

run(output_dir: Optional[str] = None, job_timeout: Optional[float] = None, live_progress: bool = False, results_progress: Optional[bool] = None, follow_state: bool = False, follow_stdout: bool = False, follow_stderr: bool = False) → None[source]

Submit a task, wait for the results and download them if required.

Parameters

  • output_dir (str) – (optional) path to a directory that will contain the results

  • job_timeout (float) – (optional) Number of seconds before the task abort() if it is not already finished

  • live_progress (bool) – (optional) display a live progress

  • results_progress (bool or function(float, float, str)) – (optional) can be a callback (read,total,filename) or True to display a progress bar

  • follow_state (bool) – (optional) print the task’s state every time it changes

  • follow_stdout (bool) – (optional) refresh and print the task’s stdout at every iteration

  • follow_stderr (bool) – (optional) refresh and print the task’s stderr at every iteration

Raises

Note

Will ensure all added file are on the resource bucket regardless of their uploading mode.

Note

If this function is interrupted (script killed for example), but the task is submitted, the task will still be executed remotely (results will not be downloaded)

Warning

Will override output_dir content.

resume(output_dir: str, job_timeout: Optional[float] = None, live_progress: bool = False, results_progress: Optional[bool] = None) → None[source]

Resume waiting for this task if it is still in submitted mode. Equivalent to wait() + download_results().

Parameters

  • output_dir (str) – path to a directory that will contain the results

  • job_timeout (float) – Number of seconds before the task abort() if it is not already finished

  • live_progress (bool) – display a live progress

  • results_progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises

Note

Do nothing if the task has not been submitted.

Warning

Will override output_dir content.

submit() → None[source]

Submit task to the cluster if it is not already submitted.

Raises

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

Note

To get the results, call download_results() once the job is done.

abort() → None[source]

Abort this task if running.

Raises
update_resources() → None[source]

Update resources for a running task.

The typical workflow is as follows:

  1. Upload new files on your resource bucket,

  2. Call this method,

  3. The new files will appear on all the compute nodes in the same resources folder as original resources

Note: There is no way to know when the files are effectively transfered. This information is available on the compute node only. Note: The update is additive only: files deleted from the bucket will NOT be deleted from the task’s resources directory.

Raises
delete(purge_resources: bool = False, purge_results: bool = False) → None[source]

Delete this task on the server.

Parameters

  • purge_resources (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

  • purge_results (bool) – parameter value is used to determine if the bucket is also deleted. Defaults to False.

Raises
update(flushcache: bool = False) → None[source]

Update the task object from the REST Api. The flushcache parameter can be used to force the update, otherwise a cached version of the object will be served when accessing properties of the object. Some methods will flush the cache, like submit(), abort(), wait() and instant(). Cache behavior is configurable with auto_update and update_cache_time.

Raises
classmethod from_json(connection: Any, json_task: Dict, is_summary: bool = False) → Any[source]

Create a Task object from a json task.

Parameters
Returns

The created Task.

commit() → None[source]

Replicate local changes on the current object instance to the REST API

Raises

Note

When updating buckets’ properties, auto update will be disabled until commit is called.

wait(timeout: Optional[float] = None, live_progress: bool = False, follow_state: bool = False, follow_stdout: bool = False, follow_stderr: bool = False) → bool[source]

Wait for this task until it is completed.

Parameters

  • timeout (float) – maximum time (in seconds) to wait before returning (None => no timeout)

  • live_progress (bool) – display a live progress

  • follow_state (bool) – (optional) print the task’s state every time it changes

  • follow_stdout (bool) – (optional) refresh and print the task’s stdout at every iteration

  • follow_stderr (bool) – (optional) refresh and print the task’s stderr at every iteration

Return type

bool

Returns

Is the task finished

Raises
snapshot(interval: int) → None[source]

Start snapshooting results. If called, this task’s results will be periodically updated, instead of only being available at the end.

Snapshots will be taken every interval second from the time the task is submitted.

Parameters

interval (int) – the interval in seconds at which to take snapshots

Raises

Note

To get the temporary results, call download_results().

instant() → None[source]

Make a snapshot of the current task.

Raises

Note

To get the temporary results, call download_results().

property state
Type

str

Getter

return this task’s state

State of the task.

Value is in

  • UnSubmitted

  • Submitted

  • PartiallyDispatched

  • FullyDispatched

  • PartiallyExecuting

  • FullyExecuting

  • UploadingResults

  • DownloadingResults

  • PendingCancel

  • Cancelled

  • Success

  • Failure

  • PendingDelete

Warning

this is the state of the task when the object was retrieved, call update() for up to date value.

property resources
Type

list(Bucket)

Getter

Returns this task’s resources bucket

Setter

Sets this task’s resources bucket

Represents resource files.

property results
Type

Bucket

Getter

Returns this task’s results bucket

Setter

Sets this task’s results bucket

Represents results files.

download_results(output_dir: str, progress: Optional[Union[bool, Callable[[float, float, str], None]]] = None) → None[source]

Download results in given output_dir.

Parameters

  • output_dir (str) – local directory for the retrieved files.

  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises

Warning

Will override output_dir content.

stdout(instanceId: Optional[int] = None)[source]

Get the standard output of the task, or of a specific instance of the task, since its submission.

Return type

str

Returns

The standard output.

Raises

Note

The buffer is circular, if stdout is too big, prefer calling fresh_stdout() regularly.

fresh_stdout(instanceId: Optional[int] = None)[source]

Get what has been written on the standard output since last time the output of the task or of the instance was retrieved.

Return type

str

Returns

The new output since last call.

Raises
stderr(instanceId: Optional[int] = None)[source]

Get the standard error of the task, or of a specific instance of the task, since its submission.

Return type

str

Returns

The standard error.

Raises

Note

The buffer is circular, if stderr is too big, prefer calling fresh_stderr() regularly.

fresh_stderr(instanceId: Optional[int] = None)[source]

Get what has been written on the standard error since last time the standard error of the task or of its instance was retrieved.

Return type

str

Returns

The new error messages since last call.

Raises
property uuid
Type

str

Getter

Returns this task’s uuid

The task’s uuid.

Automatically set when a task is submitted.

property name
Type

str

Getter

Returns this task’s name

Setter

Sets this task’s name

The task’s name.

Can be set until task is submitted.

property shortname
Type

str

Getter

Returns this task’s shortname

Setter

Sets this task’s shortname

The task’s shortname, must be DNS compliant and unique, if not provided, will default to uuid.

Can be set until task is submitted.

property tags
Type

:class:list(str)

Getter

Returns this task’s tags

Setter

Sets this task’s tags

Custom tags.

property pool
Type

Pool

Getter

Returns this task’s pool

Setter

Sets this task’s pool

The pool to run the task in.

Can be set until run() is called.

Warning

This property is mutually exclusive with profile

property profile
Type

str

Getter

Returns this task’s profile

Setter

Sets this task’s profile

The profile to run the task with.

Can be set until run() or submit() is called.

Warning

This property is mutually exclusive with pool

property instancecount
Type

int

Getter

Returns this task’s instance count

Setter

Sets this task’s instance count

Number of instances needed for the task.

Can be set until run() or submit() is called.

Raises

AttributeError – if advanced_range is not None when setting this property

Warning

This property is mutually exclusive with advanced_range

property running_core_count
Type

int

Getter

Returns this task’s running core count

Number of core running inside the task.

property running_instance_count
Type

int

Getter

Returns this task’s running instance count

Number of instances running inside the task.

property advanced_range
Type

str

Getter

Returns this task’s advanced range

Setter

Sets this task’s advanced range

Advanced instances range selection.

Allows to select which instances will be computed. Should be None or match the following extended regular expression “([0-9]+|[0-9]+-[0-9]+)(,([0-9]+|[0-9]+-[0-9]+))+” eg: 1,3-8,9,12-19

Can be set until run() is called.

Raises

AttributeError – if instancecount is not 0 when setting this property

Warning

This property is mutually exclusive with instancecount

property snapshot_whitelist
Type

str

Getter

Returns this task’s snapshot whitelist

Setter

Sets this task’s snapshot whitelist

Snapshot white list (regex) for snapshot() and instant()

Can be set until task is submitted.

property snapshot_blacklist
Type

str

Getter

Returns this task’s snapshot blacklist

Setter

Sets this task’s snapshot blacklist

Snapshot black list (regex) for snapshot() instant()

Can be set until task is submitted.

property results_whitelist
Type

str

Getter

Returns this task’s results whitelist

Setter

Sets this task’s results whitelist

Results whitelist (regex)

Can be set until task is submitted.

property results_blacklist
Type

str

Getter

Returns this task’s results blacklist

Setter

Sets this task’s results blacklist

Results blacklist (regex)

Can be set until task is submitted.

property status
Type

Status

Getter

Returns this task’s status

Status of the task

property completed_instances
Type

list(CompletedInstance)

Getter

Return this task’s completed instances

property creation_date
Type

datetime

Getter

Returns this task’s creation date

Creation date of the task (UTC Time)

property errors
Type

list(Error)

Getter

Returns this task’s errors if any.

Error reason if any, empty string if none

property constants
Type

dictionary{str : str}

Getter

Returns this task’s constants dictionary.

Setter

set the task’s constants dictionary.

Update the constants if needed. Constants are used to configure the profiles, set them to change your profile’s parameters.

property constraints
Type

dictionary{str : str}

Getter

Returns this task’s constraints dictionary.

Setter

set the task’s constraints dictionary.

Update the constraints if needed. advance usage

property forced_constants
Type

dictionary{str : ForcedConstant}

Getter

Returns this task’s forced constants dictionary.

Setter

set the task’s forced constants dictionary.

Update the forced constants if needed. Forced constants are reserved for internal use.

property forced_network_rules
Type

list{ForcedNetworkRule}

Getter

Returns this task’s forced network rules list.

Setter

set the task’s forced network rules list.

Update the forced network rules if needed. Forced network rules are reserved for internal use.

property labels
Type

dictionary{str : str}

Getter

Returns this task’s labels dictionary.

Setter

Set the task’s labels dictionary.

Update the labels if needed. Labels are used to attach arbitrary key / value pairs to a task in order to find them later with greater ease. They do not affect the execution of a task.

property wait_for_pool_resources_synchronization
Type

bool or None

Getter

Returns this task’s wait_for_pool_resources_synchronization.

Setter

set the task’s wait_for_pool_resources_synchronization.

Raises

AttributeError – can’t set this attribute on a launched task

property auto_update
Type

bool

Getter

Returns this task’s auto update state

Setter

Sets this task’s auto update state

Auto update state, default to True When auto update is disabled properties will always return cached value for the object and a call to update() will be required to get latest values from the REST Api.

property update_cache_time
Type

int

Getter

Returns this task’s auto update state

Setter

Sets this task’s auto update state

Cache expiration time, default to 5s

property upload_results_on_cancellation: Optional[bool]
Type

bool, optional

Getter

Whether task results will be uploaded upon task cancellation

Setter

Set to True to upload results on task cancellation, False to make sure they won’t be uploaded, and keep to None to get the default behavior.

Raises

AttributeError – trying to set this after the task is submitted

set_task_dependencies_from_uuids(uuids: str)[source]

Setter for the task dependencies using uuid

set_task_dependencies_from_tasks(tasks: List[Any])[source]

Setter for the task dependencies using tasks

property hardware_constraints
Type

list, optional

Getter

setup the hardware constraints

Setter

Set up specific hardware constraints.

Raises

AttributeError – trying to set this after the task is submitted

property default_resources_cache_ttl_sec: Optional[int]
Type

int, optional

Getter

The default time to live used for all the task resources cache

Raises

AttributeError – trying to set this after the task is submitted

property privileges: qarnot.privileges.Privileges
Type

Privileges

Getter

The privileges granted to the task

Raises

AttributeError – trying to set this after the task is submitted

allow_credentials_to_be_exported_to_task_environment()[source]

Grant privilege to export api and storage credentials to the task environment

property retry_settings: qarnot.retry_settings.RetrySettings
Type

RetrySettings

Getter

The retry settings for the task instances

Raises

AttributeError – trying to set this after the task is submitted

property scheduling_type: qarnot.scheduling_type.SchedulingType
Type

SchedulingType

Getter

The scheduling type for the task

Raises

AttributeError – trying to set this after the task is submitted

property targeted_reserved_machine_key: str
Type

str

Getter

The reserved machine key when using the “reserved” scheduling type

Raises

AttributeError – trying to set this after the task is submitted

property auto_delete

Autodelete this Task if it is finished and your max number of task is reach

Can be set until run() or submit() is called.

Type

bool

Getter

Returns is this task must autodelete

Setter

Sets this task’s autodelete

Default_value

“False”

Raises

AttributeError – if you try to reset the auto_delete after the task is submit

property completion_ttl

The task will be auto delete completion_ttl after it is finished

Can be set until run() or submit() is called.

Getter

Returns this task’s completed time to live.

Type

str

Setter

Sets this task’s this task’s completed time to live.

Type

str or datetime.timedelta

Default_value

“”

Raises

AttributeError – if you try to set it after the task is submitted

The completion_ttl must be a timedelta or a time span format string (example: d.hh:mm:ss or hh:mm:ss )

property previous_state
Type

str

Getter

Returns the running task’s previous state

property state_transition_time
Type

str

Getter

Returns the running task’s transition state time

task state transition time (UTC Time)

property previous_state_transition_time
Type

str

Getter

Returns the running task’s previous transition state time

task previous state transition time (UTC Time)

property last_modified
Type

str

Getter

Returns the running task’s last modification time

task’s last modified time (UTC Time)

property execution_time
Type

str

Getter

Returns the running task’s total CPU execution time.

task’s execution time of all it’s instances.

property end_date
Type

str

Getter

Returns the finished task’s end date.

task’s end date (UTC Time)

property wall_time
Type

str

Getter

task’s wall time.

task’s wall time

property snapshot_interval
Type

int

Getter

task’s snapshot interval in seconds.

task’s snapshot interval in seconds.

property progress
Type

float

Getter

task’s progress in %.

task’s progress in %.

class qarnot.task.CompletedInstance(json: Dict[str, Any])[source]

Bases: object

Completed Instance Information

Note

Read-only class

__init__(json: Dict[str, Any])[source]
instance_id
Type

int

Instance number.

state
Type

str

Instance final state.

wall_time_sec
Type

float

Instance wall time in seconds.

exec_time_sec
Type

float

Execution time in seconds.

exec_time_sec_ghz
Type

float

Execution time in seconds GHz.

peak_memory_mb
Type

int

Peak memory size in MB.

average_ghz
Type

float

Instance execution time GHz

results
Type

:class:list(str)

Instance produced results

execution_attempt_count
Type

int

Number of execution attempt of an instance, (manly in case of preemption).

class qarnot.task.BulkTaskResponse(json: Dict[str, Any])[source]

Bases: object

Bulk Task Response Information

Note

Read-only class

__init__(json: Dict[str, Any])[source]
status_code
Type

int

Status code.

uuid
Type

str

Created Task Uuid.

message
Type

str

User friendly error message.

is_success()[source]

Check that the task submit has been successful.

Return type

bool

Returns

The task creation success(depending on received uuid and the status code).

class qarnot.task.ForcedConstantAccess(value)[source]

Bases: enum.Enum

An enumeration.

class qarnot.task.ForcedConstant(forced_value: str, force_export_in_environment: Optional[bool] = None, access: Optional[qarnot.task.ForcedConstantAccess] = None)[source]

Bases: object

Forced Constant Information

Note

For internal usage only

__init__(forced_value: str, force_export_in_environment: Optional[bool] = None, access: Optional[qarnot.task.ForcedConstantAccess] = None)[source]
forced_value
Type

str

Forced value for the constant.

force_export_in_environment
Type

bool

Whether the constant should be forced in the execution environment or not.

access
Type

ForcedConstantAccess

The access level of the constant: ReadOnly or ReadWrite.