Advanced Buckets

class qarnot.advanced_bucket.AbstractFiltering

Bases: object

Abstract base class for resources filtering, allowing to select only a subset of a resources bucket as task resources.

abstract to_json()

Get a dict ready to be json packed.

Raises:

NotImplementedError – this is an abstract method, it should be overridden in child classes

abstract classmethod from_json(json)

Static method called to create the class obj from a json object.

Parameters:

json (Dict) – the json elements of the class

Raises:

NotImplementedError – this is an abstaract method, override it in it’s child classes.

abstract sanitize_filter_paths(show_warnings: bool)

Sanitize the filters bucket path by removing extra separators

Raises:

NotImplementedError – this is an abstract method, it should be overridden in child classes

class qarnot.advanced_bucket.BucketPrefixFiltering(prefix: str)

Bases: AbstractFiltering

Allows to filter a resources bucket by name prefix. Only bucket files starting with the given prefix will be used as task resources.

__init__(prefix: str)

classmethod from_json(json: Dict[str, str])

Static method called to create the class obj from a json object.

Parameters:

json (dict) – the json elements of the class

to_json() → object

Get a dict ready to be json packed.

Returns:

the json elements of the class.

Return type:

dict

sanitize_filter_paths(show_warnings: bool)

Sanitize bucket prefix path

class qarnot.advanced_bucket.Filtering

Bases: object

Groups the various object filters on an advanced resources bucket.

__init__()

append(filtering: AbstractFiltering)

Add a new filtering object

Parameters:

filtering (AbstractFiltering) – a filtering object of the bucket

classmethod from_json(json) → Filtering

Create the class sub objects of a Filtering from a json.

Parameters:

json (dict) – the json elements of the class

to_json() → Dict[str, Any]

Get a dict ready to be json packed.

class qarnot.advanced_bucket.AbstractResourcesTransformation

Bases: object

Abstract base class for resources transformation, allowing to transform bucket objects before they are presented to the task as resources.

abstract to_json()

Get a dict ready to be json packed.

abstract classmethod from_json(json)

Static method called to create the class obj from a json object.

Parameters:

json (dict) – the json elements of the class

abstract sanitize_transformation_paths(show_warnings: bool)

Sanitize the bucket path by removing extra separators

Raises:

NotImplementedError – this is an abstract method, it should be overridden in child classes

class qarnot.advanced_bucket.PrefixResourcesTransformation(prefix: str)

Bases: AbstractResourcesTransformation

Allows to remove a prefix from bucket files before they are presented to the task. During execution, the task will see files with paths stripped of the given prefix in its working directory.

__init__(prefix: str)

The PrefixResourcesTransformation constructor

Parameters:

prefix (str) – the prefix path of the resource bucket to be removed.

classmethod from_json(json)

Create a new instance of the class using the given json object.

Parameters:

json (dict) – The Json object representation.

Returns:

The PrefixResourcesTransformation new object

Return type:

PrefixResourcesTransformation

to_json() → object

Get a dict ready to be json packed.

sanitize_transformation_paths(show_warnings: bool)

Sanitize bucket strip prefix path

class qarnot.advanced_bucket.ResourcesTransformation

Bases: object

Groups the various object transformation on an advanced resources bucket.

__init__()

append(resource: AbstractResourcesTransformation)

Add a new resource transformation object

Parameters:

filtering (AbstractResourcesTransformation) – a filtering object of the bucket

classmethod from_json(json) → ResourcesTransformation

Create the class sub objects of a ResourcesTransformation from a json.

Parameters:

json (dict) – the json elements of the class

to_json() → object

Get a dict ready to be json packed.