Connection

Module describing a connection.

class qarnot.connection.Connection(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Logger | None = None)[source]

Bases: object

Represents the couple cluster/user to which submit tasks.

__init__(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Logger | None = None)[source]

Create a connection to a cluster with given config file, options or environment variables. Available environment variable are QARNOT_CLUSTER_URL, QARNOT_CLUSTER_UNSAFE, QARNOT_CLUSTER_TIMEOUT and QARNOT_CLIENT_TOKEN.

Parameters:

  • fileconf (str or dict) – path to a qarnot configuration file or a corresponding dict

  • client_token (str) – API Token

  • cluster_url (str) – (optional) Cluster url.

  • cluster_unsafe (bool) – (optional) Disable certificate check

  • cluster_timeout (int) – (optional) Timeout value for every request

  • storage_url (str) – (optional) Storage service url.

  • storage_unsafe (bool) – (optional) Disable certificate check

  • retry_count (int) – (optional) ConnectionError retry count. Default to 5.

  • retry_wait (float) – (optional) Retry on error wait time, progressive. (wait * (retry_count - retry_num). Default to 1s

  • sanitize_bucket_paths (bool) – (optional) Flag to automatically sanitize bucket paths (remove extra slashes). Default to true

  • show_bucket_warnings (bool) – (optional) Flag to show warnings of bucket paths sanitization. Default to true

  • logger (logging.Logger) – which job to attach the task to

Configuration sample:

[cluster]
# url of the REST API
url=https://localhost
# No SSL verification ?
unsafe=False
[client]
# auth string of the client
token=login
[storage]
url=https://storage
unsafe=False
property s3client

Pre-configured s3 client object.

Return type:

list(S3.Client)

Returns:

A list of ObjectSummary resources

property s3resource

Pre-configured s3 resource object.

Return type:

list(S3.ServiceResource)

Returns:

A list of ObjectSummary resources

property user_info

Get information of the current user on the cluster.

Return type:

UserInfo

Returns:

Requested information.

Raises:
buckets()[source]

Get the list of buckets.

Return type:

list(Bucket).

Returns:

List of buckets

Raises:

BucketStorageUnavailableException – the bucket storage engine is not available

pools(summary=True, tags_intersect=None, tags=None)[source]

Get the list of pools stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.

Parameters:

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

  • tags (list of str, optional) – Desired filtering tags, any of them

Return type:

List of Pool.

Returns:

Pools stored on the cluster owned by the user.

Raises:

Deprecated since version 2.5.0: This function can be inefficient, Use all_pools() instead.

tasks(tags=None, summary=True, tags_intersect=None)[source]

Get the list of tasks stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.

Parameters:

  • tags (list of str, optional) – Desired filtering tags, any of them

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type:

List of Task.

Returns:

Tasks stored on the cluster owned by the user.

Raises:

Deprecated since version 2.5.0: This function can be inefficient, Use all_tasks() instead.

jobs(tags=None, tags_intersect=None)[source]

Get the list of jobs stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.

Parameters:

  • tags (list of str, optional) – Desired filtering tags, any of them

  • tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them

Raises:

Deprecated since version 2.5.0: This function can be inefficient, Use all_jobs() instead.

all_pools(summary: bool = True, tags: List[str] | None = None, tags_intersect: List[str] | None = None)[source]

Get an iterator of all the pools.

if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.

Parameters:

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

  • tags (list of str, optional) – Desired filtering tags, any of them

Return type:

List of Pool.

Returns:

Pools stored on the cluster owned by the user.

Raises:
all_tasks(summary: bool = True, tags: List[str] | None = None, tags_intersect: List[str] | None = None)[source]

Get an iterator of all the tasks.

if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.

Parameters:

  • tags (list of str, optional) – Desired filtering tags, any of them

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type:

List of Task.

Returns:

Tasks stored on the cluster owned by the user.

Raises:
all_jobs(tags: List[str] | None = None, tags_intersect: List[str] | None = None) → Iterator[Iterable][source]

Get an iterator of all the jobs.

if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.

Parameters:

  • tags (list(str), optional) – Desired filtering tags, any of them

  • tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them

Raises:
all_hardware_constraints() → Iterator[Iterable][source]

Get all the hardware constraints.

Raises:
pools_page(token: str | None = None, maximum: int | None = None, summary: bool = True, tags: List | None = None, tags_intersect: List | None = None) → PaginateResponse[source]

Return a paginate pool object retriever.

Parameters:

  • summary (bool, optional) – retrieve a full pool details if False or a pool summary if True, defaults to True

  • tags_intersect (list of str , optional) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns:

A paginate object

Return type:

PaginateResponse

tasks_page(token: str | None = None, maximum: int | None = None, summary: bool = True, tags: List | None = None, tags_intersect: List | None = None) → PaginateResponse[source]

Return a paginate task object.

Parameters:

  • summary (bool, optional) – retrieve a full task details if False or a task summary if True, defaults to True

  • tags_intersect (list of str, optional) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns:

A paginate object

Return type:

PaginateResponse

jobs_page(token: str | None = None, maximum: int | None = None, tags: List | None = None, tags_intersect: List | None = None) → PaginateResponse[source]

Return a paginate job object.

Parameters:

  • tags_intersect (list of str) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns:

A paginate object

Return type:

PaginateResponse

hardware_constraints_page(limit: int | None = 50, offset: int | None = 0) → OffsetResponse[source]

Return a list of hardware constraints limited with offset.

Parameters:

  • limit (int, optional) – limit the number of displayed constraints, defaults to 50

  • offset – the number of constraints ignored in the response, defaults to 0

Returns:

An offset object

Return type:

OffsetResponse

search_cpu_model_constraints(cpu_model: str) → List[CpuModelHardware][source]

Return a list of CPU model hardware constraints matching the search term.

These constraints are ordered by relevancy.

Parameters:

cpu_model (str) – search term to match available cpu constraints.

Returns:

A list of CPU model hardware constraints ordered by relevancy.

Return type:

list(CpuModelHardware)

retrieve_pool(uuid)[source]

Retrieve a Pool from its uuid

Parameters:

uuid (str) – Desired pool uuid

Return type:

Pool

Returns:

Existing pool defined by the given uuid

Raises:
retrieve_task(uuid)[source]

Retrieve a Task from its uuid

Parameters:

uuid (str) – Desired task uuid

Return type:

Task

Returns:

Existing task defined by the given uuid

Raises:
retrieve_job(uuid)[source]

Retrieve a Job from its uuid

Parameters:

uuid (str) – Desired job uuid

Return type:

Job

Returns:

Existing job defined by the given uuid

Raises:
retrieve_or_create_bucket(uuid)[source]

Retrieve a Bucket from its description, or create a new one.

Parameters:

uuid (str) – the bucket uuid (name)

Return type:

Bucket

Returns:

Existing or newly created bucket defined by the given name

Raises:

BucketStorageUnavailableException – the bucket storage engine is not available

retrieve_bucket(uuid)[source]

Retrieve a Bucket from its uuid (name)

Parameters:

uuid (str) – Desired bucket uuid (name)

Return type:

Bucket

Returns:

Existing bucket defined by the given uuid (name)

Raises:

BucketStorageUnavailableException – the bucket storage engine is not available

Raises:

botocore.exceptions.ClientError: Bucket does not exist, or invalid credentials

create_pool(name, profile, instancecount=1, shortname=None)[source]

Create a new Pool.

Parameters:

  • name (str) – given name of the pool

  • profile (str) – which profile to use with this pool

  • instancecount (int) – number of instances to run for the pool

  • shortname (str) – optional unique friendly shortname of the pool

Return type:

Pool

Returns:

The created Pool.

Note

See available profiles with profiles().

create_elastic_pool(name, profile, minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90, shortname=None)[source]

Create a new Pool.

Parameters:

  • name (str) – given name of the pool

  • profile (str) – which profile to use with this pool

  • minimum_total_slots (int) – minimum number of instances to run for the pool

  • maximum_total_slots (int) – maximum number of instances to run for the pool

  • minimum_idle_slots (int) – the number of instances that can be idle before considering shrinking the pool

  • minimum_idle_time_seconds (int) – the number of seconds before considering shrinking the pool

  • resize_factor (float) – the speed with which we grow the pool to meet the demand

  • resize_period (int) – the time between the load checks that decide if the pool grows or shrinks

  • shortname (str) – optional unique friendly shortname of the pool

Return type:

Pool

Returns:

The created Pool.

Note

See available profiles with profiles().

create_task(name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)[source]

Create a new Task.

Parameters:

  • name (str) – given name of the task

  • profile_or_pool (str or Pool or None) –

    which profile to use with this task, or which Pool to run task, or which job to attach it to.

    Warning

    The Pool or Job need to be already submitted to the api with Pool.submit() or Job.submit() in order to be correctly linked to the task.

  • instancecount_or_range (int or str) – number of instances, or ranges on which to run task. Defaults to 1.

  • shortname (str) – optional unique friendly shortname of the task

  • job (Job) – which job to attach the task to

Return type:

Task

Returns:

The created Task.

Note

See available profiles with profiles().

submit_tasks(tasks)[source]

Submit a list of Task.

:param List of Task. :raises ~qarnot.exceptions.QarnotGenericException: API general error, see message for details

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

profiles_names()[source]

Get list of profiles names available on the cluster.

Return type:

list of str

Raises:
profile_details(profile_name)[source]

Get a profile available on the cluster.

Return type:

Profile

Raises:
profiles()[source]

Get list of profiles available on the cluster.

Return type:

list of Profile

Raises:
retrieve_profile(name)[source]

Get details of a profile from its name.

Return type:

Profile

Raises:
create_bucket(name)[source]

Create a new Bucket. If the bucket already exist, retrieve the existing bucket.

Parameters:

name (str) – bucket name

Return type:

Bucket

Returns:

The created or existing Bucket.

create_job(name, pool=None, shortname=None, useDependencies=False)[source]

Create a new Job.

Parameters:

  • name (str) – given name of the job

  • pool (Pool or None) –

    which Pool to submit the job in.

    Warning

    The Pool needs to be already submitted to the api with Pool.submit() in order to be correctly linked to the job.

  • shortname (str) – userfriendly job name

  • use_dependencies – allow dependencies between tasks in this job

Returns:

The created Job.

class qarnot.connection.UserInfo(info)[source]

Bases: object

Information about a qarnot user.

__init__(info)[source]
email
Type:

str

User email address.

max_bucket
Type:

int

Maximum number of buckets allowed (resource and result buckets).

bucket_count
Type:

int

Number of buckets owned by the user.

quota_bytes_bucket
Type:

int

Total storage space allowed for the user’s buckets (in Bytes).

used_quota_bytes_bucket
Type:

int

Total storage space used by the user’s buckets (in Bytes).

task_count
Type:

int

Total number of tasks belonging to the user.

max_task
Type:

int

Maximum number of tasks the user is allowed to create.

running_task_count
Type:

int

Number of tasks currently in ‘Submitted’ state.

max_running_task
Type:

int

Maximum number of running tasks.

max_instances
Type:

int

Maximum number of instances.

max_cores
Type:

int

Maximum number of cores.

max_pool
Type:

int

Maximum number of pool the user is allowed to create.

pool_count
Type:

int

Total number of pools belonging to the user.

max_running_pool
Type:

int

Maximum number of running pools the user is allowed to create.

running_pool_count
Type:

int

Number of pools currently submitted or running.

running_instance_count
Type:

int

Number of Instances currently submitted or running.

running_core_count
Type:

int

Number of cores currently submitted or running.

max_flex_instances
Type:

int

Maximum number of instances simultaneously used with Flex scheduling plan.

max_flex_cores
Type:

int

Maximum number of cores simultaneously used with Flex scheduling plan.

max_on_demand_instances
Type:

int

Maximum number of instances simultaneously used with OnDemand scheduling plan.

max_on_demand_cores
Type:

int

Maximum number of cores simultaneously used with OnDemand scheduling plan.

class qarnot.connection.Profile(info)[source]

Bases: object

Information about a profile.

__init__(info)[source]
name
Type:

str

Name of the profile.

constants
Type:

List of (str, str)

List of couples (name, value) representing constants for this profile and their default values.

__repr__()[source]

Return repr(self).