Connection¶
Module describing a connection.
- class qarnot.connection.Connection(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Optional[logging.Logger] = None)[source]¶
Bases:
object
Represents the couple cluster/user to which submit tasks.
- __init__(fileconf=None, client_token=None, cluster_url=None, cluster_unsafe=False, cluster_timeout=None, storage_url=None, storage_unsafe=False, retry_count=5, retry_wait=1.0, cluster_custom_certificate=None, storage_custom_certificate=None, sanitize_bucket_paths=True, show_bucket_warnings=True, logger: Optional[logging.Logger] = None)[source]¶
Create a connection to a cluster with given config file, options or environment variables. Available environment variable are QARNOT_CLUSTER_URL, QARNOT_CLUSTER_UNSAFE, QARNOT_CLUSTER_TIMEOUT and QARNOT_CLIENT_TOKEN.
- Parameters
fileconf (str or dict) – path to a qarnot configuration file or a corresponding dict
client_token (str) – API Token
cluster_url (str) – (optional) Cluster url.
cluster_unsafe (bool) – (optional) Disable certificate check
cluster_timeout (int) – (optional) Timeout value for every request
storage_url (str) – (optional) Storage service url.
storage_unsafe (bool) – (optional) Disable certificate check
retry_count (int) – (optional) ConnectionError retry count. Default to 5.
retry_wait (float) – (optional) Retry on error wait time, progressive. (wait * (retry_count - retry_num). Default to 1s
sanitize_bucket_paths (bool) – (optional) Flag to automatically sanitize bucket paths (remove extra slashes). Default to true
show_bucket_warnings (bool) – (optional) Flag to show warnings of bucket paths sanitization. Default to true
logger (
logging.Logger
) – which job to attach the task to
Configuration sample:
[cluster] # url of the REST API url=https://localhost # No SSL verification ? unsafe=False [client] # auth string of the client token=login [storage] url=https://storage unsafe=False
- property s3client¶
Pre-configured s3 client object.
- Return type
list(
S3.Client
)- Returns
A list of ObjectSummary resources
- property s3resource¶
Pre-configured s3 resource object.
- Return type
list(
S3.ServiceResource
)- Returns
A list of ObjectSummary resources
- property user_info¶
Get information of the current user on the cluster.
- Return type
- Returns
Requested information.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- buckets()[source]¶
Get the list of buckets.
- Return type
list(
Bucket
).- Returns
List of buckets
- Raises
BucketStorageUnavailableException – the bucket storage engine is not available
- pools(summary=True, tags_intersect=None, tags=None)[source]¶
Get the list of pools stored on this cluster for this user.
if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.
- Parameters
summary (bool) – only get the summaries.
tags_intersect (list of str, optional) – Desired filtering tags, all of them
tags (list of str, optional) – Desired filtering tags, any of them
- Return type
List of
Pool
.- Returns
Pools stored on the cluster owned by the user.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
Deprecated since version 2.5.0: This function can be inefficient, Use
all_pools()
instead.Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_pools function instead
- tasks(tags=None, summary=True, tags_intersect=None)[source]¶
Get the list of tasks stored on this cluster for this user.
if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.
- Parameters
tags (list of str, optional) – Desired filtering tags, any of them
summary (bool) – only get the summaries.
tags_intersect (list of str, optional) – Desired filtering tags, all of them
- Return type
List of
Task
.- Returns
Tasks stored on the cluster owned by the user.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
Deprecated since version 2.5.0: This function can be inefficient, Use
all_tasks()
instead.Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_tasks function instead
- jobs(tags=None, tags_intersect=None)[source]¶
Get the list of jobs stored on this cluster for this user.
if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.
- Parameters
tags (list of str, optional) – Desired filtering tags, any of them
tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
Deprecated since version 2.5.0: This function can be inefficient, Use
all_jobs()
instead.Deprecated since version 2.5.0: This will be removed in 3.0. Use the all_jobs function instead
- all_pools(summary: bool = True, tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None)[source]¶
Get an iterator of all the pools.
if tags and tags_intersect are set, the connection will only return the pools with tag intersect values.
- Parameters
summary (bool) – only get the summaries.
tags_intersect (list of str, optional) – Desired filtering tags, all of them
tags (list of str, optional) – Desired filtering tags, any of them
- Return type
List of
Pool
.- Returns
Pools stored on the cluster owned by the user.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- all_tasks(summary: bool = True, tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None)[source]¶
Get an iterator of all the tasks.
if tags and tags_intersect are set, the connection will only return the tasks with tag_intersect values.
- Parameters
tags (list of str, optional) – Desired filtering tags, any of them
summary (bool) – only get the summaries.
tags_intersect (list of str, optional) – Desired filtering tags, all of them
- Return type
List of
Task
.- Returns
Tasks stored on the cluster owned by the user.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- all_jobs(tags: Optional[List[str]] = None, tags_intersect: Optional[List[str]] = None) Iterator[Iterable] [source]¶
Get an iterator of all the jobs.
if tags and tags_intersect are set, the connection will only return the jobs with tag intersect values.
- Parameters
tags (list(str), optional) – Desired filtering tags, any of them
tags_intersect (list of str, optional) – Desired filtering tags, the jobs must have all of them
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- all_hardware_constraints() Iterator[Iterable] [source]¶
Get all the hardware constraints.
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- pools_page(token: Optional[str] = None, maximum: Optional[int] = None, summary: bool = True, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse [source]¶
Return a paginate pool object retriever.
- Parameters
summary (bool, optional) – retrieve a full pool details if False or a pool summary if True, defaults to True
tags_intersect (list of str , optional) – use a tag exclusive filter, defaults to None
token (str, optional) – the first paginate token to be used, defaults to None
maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10
- Returns
A paginate object
- Return type
- tasks_page(token: Optional[str] = None, maximum: Optional[int] = None, summary: bool = True, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse [source]¶
Return a paginate task object.
- Parameters
summary (bool, optional) – retrieve a full task details if False or a task summary if True, defaults to True
tags_intersect (list of str, optional) – use a tag exclusive filter, defaults to None
token (str, optional) – the first paginate token to be used, defaults to None
maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10
- Returns
A paginate object
- Return type
- jobs_page(token: Optional[str] = None, maximum: Optional[int] = None, tags: Optional[List] = None, tags_intersect: Optional[List] = None) qarnot.paginate.PaginateResponse [source]¶
Return a paginate job object.
- Parameters
tags_intersect (list of str) – use a tag exclusive filter, defaults to None
token (str, optional) – the first paginate token to be used, defaults to None
maximum (int, optional) – the maximum number of pages to retrieve, defaults to 10
- Returns
A paginate object
- Return type
- hardware_constraints_page(limit: Optional[int] = 50, offset: Optional[int] = 0) qarnot.paginate.OffsetResponse [source]¶
Return a list of hardware constraints limited with offset.
- Parameters
limit (int, optional) – limit the number of displayed constraints, defaults to 50
offset – the number of constraints ignored in the response, defaults to 0
- Returns
An offset object
- Return type
- search_cpu_model_constraints(cpu_model: str) List[qarnot.hardware_constraint.CpuModelHardware] [source]¶
Return a list of CPU model hardware constraints matching the search term.
These constraints are ordered by relevancy.
- Parameters
cpu_model (str) – search term to match available cpu constraints.
- Returns
A list of CPU model hardware constraints ordered by relevancy.
- Return type
list(
CpuModelHardware
)
- retrieve_pool(uuid)[source]¶
Retrieve a
Pool
from its uuid- Parameters
uuid (str) – Desired pool uuid
- Return type
- Returns
Existing pool defined by the given uuid
- Raises
MissingPoolException – pool does not exist
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- retrieve_task(uuid)[source]¶
Retrieve a
Task
from its uuid- Parameters
uuid (str) – Desired task uuid
- Return type
- Returns
Existing task defined by the given uuid
- Raises
MissingTaskException – task does not exist
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- retrieve_job(uuid)[source]¶
Retrieve a
Job
from its uuid- Parameters
uuid (str) – Desired job uuid
- Return type
- Returns
Existing job defined by the given uuid
- Raises
MissingJobException – job does not exist
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- retrieve_or_create_bucket(uuid)[source]¶
Retrieve a
Bucket
from its description, or create a new one.- Parameters
uuid (str) – the bucket uuid (name)
- Return type
- Returns
Existing or newly created bucket defined by the given name
- Raises
BucketStorageUnavailableException – the bucket storage engine is not available
- retrieve_bucket(uuid)[source]¶
Retrieve a
Bucket
from its uuid (name)- Parameters
uuid (str) – Desired bucket uuid (name)
- Return type
- Returns
Existing bucket defined by the given uuid (name)
- Raises
BucketStorageUnavailableException – the bucket storage engine is not available
- Raises
botocore.exceptions.ClientError: Bucket does not exist, or invalid credentials
- create_pool(name, profile, instancecount=1, shortname=None)[source]¶
Create a new
Pool
.- Parameters
- Return type
- Returns
The created
Pool
.
Note
See available profiles with
profiles()
.
- create_elastic_pool(name, profile, minimum_total_slots=0, maximum_total_slots=1, minimum_idle_slots=0, minimum_idle_time_seconds=0, resize_factor=1, resize_period=90, shortname=None)[source]¶
Create a new
Pool
.- Parameters
name (str) – given name of the pool
profile (str) – which profile to use with this pool
minimum_total_slots (int) – minimum number of instances to run for the pool
maximum_total_slots (int) – maximum number of instances to run for the pool
minimum_idle_slots (int) – the number of instances that can be idle before considering shrinking the pool
minimum_idle_time_seconds (int) – the number of seconds before considering shrinking the pool
resize_factor (float) – the speed with which we grow the pool to meet the demand
resize_period (int) – the time between the load checks that decide if the pool grows or shrinks
shortname (str) – optional unique friendly shortname of the pool
- Return type
- Returns
The created
Pool
.
Note
See available profiles with
profiles()
.
- create_task(name, profile_or_pool=None, instancecount_or_range=1, shortname=None, job=None)[source]¶
Create a new
Task
.- Parameters
name (str) – given name of the task
profile_or_pool (str or
Pool
or None) –which profile to use with this task, or which Pool to run task, or which job to attach it to.
Warning
The
Pool
orJob
need to be already submitted to the api withPool.submit()
orJob.submit()
in order to be correctly linked to the task.instancecount_or_range (int or str) – number of instances, or ranges on which to run task. Defaults to 1.
shortname (str) – optional unique friendly shortname of the task
job (
Job
) – which job to attach the task to
- Return type
- Returns
The created
Task
.
Note
See available profiles with
profiles()
.
- submit_tasks(tasks)[source]¶
Submit a list of
Task
.:param List of
Task
. :raises ~qarnot.exceptions.QarnotGenericException: API general error, see message for detailsNote
Will ensure all added files are on the resource bucket regardless of their uploading mode.
- profiles_names()[source]¶
Get list of profiles names available on the cluster.
- Return type
list of str
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- profile_details(profile_name)[source]¶
Get a profile available on the cluster.
- Return type
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- profiles()[source]¶
Get list of profiles available on the cluster.
- Return type
list of
Profile
- Raises
UnauthorizedException – invalid credentials
QarnotGenericException – API general error, see message for details
- retrieve_profile(name)[source]¶
Get details of a profile from its name.
- Return type
- Raises
UnauthorizedException – invalid credentials
MissingProfileException – profile not found
QarnotGenericException – API general error, see message for details
- create_bucket(name)[source]¶
Create a new
Bucket
. If the bucket already exist, retrieve the existing bucket.
- create_job(name, pool=None, shortname=None, useDependencies=False)[source]¶
Create a new
Job
.- Parameters
name (
str
) – given name of the jobpool (
Pool
or None) –which Pool to submit the job in.
Warning
The
Pool
needs to be already submitted to the api withPool.submit()
in order to be correctly linked to the job.shortname (
str
) – userfriendly job nameuse_dependencies – allow dependencies between tasks in this job
- Returns
The created
Job
.
- class qarnot.connection.UserInfo(info)[source]¶
Bases:
object
Information about a qarnot user.
- max_flex_instances¶
- Type
Maximum number of instances simultaneously used with Flex scheduling plan.
- max_on_demand_instances¶
- Type
Maximum number of instances simultaneously used with OnDemand scheduling plan.