Connection

Module describing a connection.

class qarnot.connection.Connection(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Optional[logging.Logger] = None)[source]

Bases: object

Represents the couple cluster/user to which submit tasks.

__init__(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Optional[logging.Logger] = None)[source]

Create a connection to a cluster with given config file, options or environment variables. Available environment variable are QARNOT_CLUSTER_URL, QARNOT_CLUSTER_UNSAFE, QARNOT_CLUSTER_TIMEOUT and QARNOT_CLIENT_TOKEN.

Parameters
  • fileconf (str or dict) – path to a qarnot configuration file or a corresponding dict

  • client_token (str) – API Token

  • cluster_url (str) – (optional) Cluster url.

  • cluster_unsafe (bool) – (optional) Disable certificate check

  • cluster_timeout (int) – (optional) Timeout value for every request

  • storage_url (str) – (optional) Storage service url.

  • storage_unsafe (bool) – (optional) Disable certificate check

  • retry_count (int) – (optional) ConnectionError retry count. Default to 5.

  • retry_wait (float) – (optional) Retry on error wait time, progressive. (wait * (retry_count - retry_num). Default to 1s

  • sanitize_bucket_paths (bool) – (optional) Flag to automatically sanitize bucket paths (remove extra slashes). Default to true

  • show_bucket_warnings (bool) – (optional) Flag to show warnings of bucket paths sanitization. Default to true

  • logger (logging.Logger) – which job to attach the task to

Configuration sample:

[cluster]
# url of the REST API
url=https://localhost
# No SSL verification ?
unsafe=False
[client]
# auth string of the client
token=login
[storage]
url=https://storage
unsafe=False
property s3client

Pre-configured s3 client object.

Return type

list(S3.Client)

Returns

A list of ObjectSummary resources

property s3resource

Pre-configured s3 resource object.

Return type

list(S3.ServiceResource)

Returns

A list of ObjectSummary resources

property user_info

Get information of the current user on the cluster.

Return type

UserInfo

Returns

Requested information.

Raises
buckets()[source]

Get the list of buckets.

Return type

list(Bucket).

Returns

List of buckets

Raises

BucketStorageUnavailableException – the bucket storage engine is not available

pools(summary=True, tags_intersect=None, tags=None)[source]

Get the list of pools stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.

Parameters
  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

  • tags (list of str, optional) – Desired filtering tags, any of them

Return type

List of Pool.

Returns

Pools stored on the cluster owned by the user.

Raises

Deprecated since version 2.5.0: This function can be inefficient, Use all_pools() instead.

Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_pools function instead

tasks(tags=None, summary=True, tags_intersect=None)[source]

Get the list of tasks stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.

Parameters
  • tags (list of str, optional) – Desired filtering tags, any of them

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type

List of Task.

Returns

Tasks stored on the cluster owned by the user.

Raises

Deprecated since version 2.5.0: This function can be inefficient, Use all_tasks() instead.

Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_tasks function instead

jobs(tags=None, tags_intersect=None)[source]

Get the list of jobs stored on this cluster for this user.

if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.

Parameters
  • tags (list of str, optional) – Desired filtering tags, any of them

  • tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them

Raises

Deprecated since version 2.5.0: This function can be inefficient, Use all_jobs() instead.

Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_jobs function instead

all_pools(summary: bool = True, tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None)[source]

Get an iterator of all the pools.

if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.

Parameters
  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

  • tags (list of str, optional) – Desired filtering tags, any of them

Return type

List of Pool.

Returns

Pools stored on the cluster owned by the user.

Raises
all_tasks(summary: bool = True, tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None)[source]

Get an iterator of all the tasks.

if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.

Parameters
  • tags (list of str, optional) – Desired filtering tags, any of them

  • summary (bool) – only get the summaries.

  • tags_intersect (list of str, optional) – Desired filtering tags, all of them

Return type

List of Task.

Returns

Tasks stored on the cluster owned by the user.

Raises
all_jobs(tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None) Iterator[Iterable][source]

Get an iterator of all the jobs.

if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.

Parameters
  • tags (list(str), optional) – Desired filtering tags, any of them

  • tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them

Raises
all_hardware_constraints() Iterator[Iterable][source]

Get all the hardware constraints.

Raises
pools_page(token: Optional[str] = None, maximum: Optional[int] = None, summary: bool = True, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse[source]

Return a paginate pool object retriever.

Parameters
  • summary (bool, optional) – retrieve a full pool details if False or a pool summary if True, defaults to True

  • tags_intersect (list of str , optional) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns

A paginate object

Return type

PaginateResponse

tasks_page(token: Optional[str] = None, maximum: Optional[int] = None, summary: bool = True, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse[source]

Return a paginate task object.

Parameters
  • summary (bool, optional) – retrieve a full task details if False or a task summary if True, defaults to True

  • tags_intersect (list of str, optional) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns

A paginate object

Return type

PaginateResponse

jobs_page(token: Optional[str] = None, maximum: Optional[int] = None, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse[source]

Return a paginate job object.

Parameters
  • tags_intersect (list of str) – use a tag exclusive filter, defaults to None

  • token (str, optional) – the first paginate token to be used, defaults to None

  • maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10

Returns

A paginate object

Return type

PaginateResponse

hardware_constraints_page(limit: Optional[int] = 50, offset: Optional[int] = 0) qarnot.paginate.OffsetResponse[source]

Return a list of hardware constraints limited with offset.

Parameters
  • limit (int, optional) – limit the number of displayed constraints, defaults to 50

  • offset – the number of constraints ignored in the response, defaults to 0

Returns

An offset object

Return type

OffsetResponse

search_cpu_model_constraints(cpu_model: str) List[qarnot.hardware_constraint.CpuModelHardware][source]

Return a list of CPU model hardware constraints matching the search term.

These constraints are ordered by relevancy.

Parameters

cpu_model (str) – search term to match available cpu constraints.

Returns

A list of CPU model hardware constraints ordered by relevancy.

Return type

list(CpuModelHardware)

retrieve_pool(uuid)[source]

Retrieve a Pool from its uuid

Parameters

uuid (str) – Desired pool uuid

Return type

Pool

Returns

Existing pool defined by the given uuid

Raises
retrieve_task(uuid)[source]

Retrieve a Task from its uuid

Parameters

uuid (str) – Desired task uuid

Return type

Task

Returns

Existing task defined by the given uuid

Raises
retrieve_job(uuid)[source]

Retrieve a Job from its uuid

Parameters

uuid (str) – Desired job uuid

Return type

Job

Returns

Existing job defined by the given uuid

Raises
retrieve_or_create_bucket(uuid)[source]

Retrieve a Bucket from its description, or create a new one.

Parameters

uuid (str) – the bucket uuid (name)

Return type

Bucket

Returns

Existing or newly created bucket defined by the given name

Raises

BucketStorageUnavailableException – the bucket storage engine is not available

retrieve_bucket(uuid)[source]

Retrieve a Bucket from its uuid (name)

Parameters

uuid (str) – Desired bucket uuid (name)

Return type

Bucket

Returns

Existing bucket defined by the given uuid (name)

Raises

BucketStorageUnavailableException – the bucket storage engine is not available

Raises

botocore.exceptions.ClientError: Bucket does not exist, or invalid credentials

create_pool(name, profile, instancecount=1, shortname=None)[source]

Create a new Pool.

Parameters
  • name (str) – given name of the pool

  • profile (str) – which profile to use with this pool

  • instancecount (int) – number of instances to run for the pool

  • shortname (str) – optional unique friendly shortname of the pool

Return type

Pool

Returns

The created Pool.

Note

See available profiles with profiles().

create_elastic_pool(name, profile, minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90, shortname=None)[source]

Create a new Pool.

Parameters
  • name (str) – given name of the pool

  • profile (str) – which profile to use with this pool

  • minimum_total_slots (int) – minimum number of instances to run for the pool

  • maximum_total_slots (int) – maximum number of instances to run for the pool

  • minimum_idle_slots (int) – the number of instances that can be idle before considering shrinking the pool

  • minimum_idle_time_seconds (int) – the number of seconds before considering shrinking the pool

  • resize_factor (float) – the speed with which we grow the pool to meet the demand

  • resize_period (int) – the time between the load checks that decide if the pool grows or shrinks

  • shortname (str) – optional unique friendly shortname of the pool

Return type

Pool

Returns

The created Pool.

Note

See available profiles with profiles().

create_task(name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)[source]

Create a new Task.

Parameters
  • name (str) – given name of the task

  • profile_or_pool (str or Pool or None) –

    which profile to use with this task, or which Pool to run task, or which job to attach it to.

    Warning

    The Pool or Job need to be already submitted to the api with Pool.submit() or Job.submit() in order to be correctly linked to the task.

  • instancecount_or_range (int or str) – number of instances, or ranges on which to run task. Defaults to 1.

  • shortname (str) – optional unique friendly shortname of the task

  • job (Job) – which job to attach the task to

Return type

Task

Returns

The created Task.

Note

See available profiles with profiles().

submit_tasks(tasks)[source]

Submit a list of Task.

:param List of Task. :raises ~qarnot.exceptions.QarnotGenericException: API general error, see message for details

Note

Will ensure all added files are on the resource bucket regardless of their uploading mode.

profiles_names()[source]

Get list of profiles names available on the cluster.

Return type

list of str

Raises
profile_details(profile_name)[source]

Get a profile available on the cluster.

Return type

Profile

Raises
profiles()[source]

Get list of profiles available on the cluster.

Return type

list of Profile

Raises
retrieve_profile(name)[source]

Get details of a profile from its name.

Return type

Profile

Raises
create_bucket(name)[source]

Create a new Bucket. If the bucket already exist, retrieve the existing bucket.

Parameters

name (str) – bucket name

Return type

Bucket

Returns

The created or existing Bucket.

create_job(name, pool=None, shortname=None, useDependencies=False)[source]

Create a new Job.

Parameters
  • name (str) – given name of the job

  • pool (Pool or None) –

    which Pool to submit the job in.

    Warning

    The Pool needs to be already submitted to the api with Pool.submit() in order to be correctly linked to the job.

  • shortname (str) – userfriendly job name

  • use_dependencies – allow dependencies between tasks in this job

Returns

The created Job.

class qarnot.connection.UserInfo(info)[source]

Bases: object

Information about a qarnot user.

__init__(info)[source]
email
Type

str

User email address.

max_bucket
Type

int

Maximum number of buckets allowed (resource and result buckets).

bucket_count
Type

int

Number of buckets owned by the user.

quota_bytes_bucket
Type

int

Total storage space allowed for the user’s buckets (in Bytes).

used_quota_bytes_bucket
Type

int

Total storage space used by the user’s buckets (in Bytes).

task_count
Type

int

Total number of tasks belonging to the user.

max_task
Type

int

Maximum number of tasks the user is allowed to create.

running_task_count
Type

int

Number of tasks currently in ‘Submitted’ state.

max_running_task
Type

int

Maximum number of running tasks.

max_instances
Type

int

Maximum number of instances.

max_cores
Type

int

Maximum number of cores.

max_pool
Type

int

Maximum number of pool the user is allowed to create.

pool_count
Type

int

Total number of pools belonging to the user.

max_running_pool
Type

int

Maximum number of running pools the user is allowed to create.

running_pool_count
Type

int

Number of pools currently submitted or running.

running_instance_count
Type

int

Number of Instances currently submitted or running.

running_core_count
Type

int

Number of cores currently submitted or running.

max_flex_instances
Type

int

Maximum number of instances simultaneously used with Flex scheduling plan.

max_flex_cores
Type

int

Maximum number of cores simultaneously used with Flex scheduling plan.

max_on_demand_instances
Type

int

Maximum number of instances simultaneously used with OnDemand scheduling plan.

max_on_demand_cores
Type

int

Maximum number of cores simultaneously used with OnDemand scheduling plan.

class qarnot.connection.Profile(info)[source]

Bases: object

Information about a profile.

__init__(info)[source]
name
Type

str

Name of the profile.

constants
Type

List of (str, str)

List of couples (name, value) representing constants for this profile and their default values.

__repr__()[source]

Return repr(self).