Advanced Buckets

class qarnot.advanced_bucket.AbstractFiltering[source]

Bases: object

Abstract base class for resources filtering, allowing to select only a subset of a resources bucket as task resources.

abstract to_json()[source]

Get a dict ready to be json packed.

Raises

NotImplementedError – this is an abstract method, it should be overridden in child classes

abstract classmethod from_json(json)[source]

Static method called to create the class obj from a json object.

Parameters

json (Dict) – the json elements of the class

Raises

NotImplementedError – this is an abstaract method, override it in it’s child classes.

abstract sanitize_filter_paths(show_warnings: bool)[source]

Sanitize the filters bucket path by removing extra separators

Raises

NotImplementedError – this is an abstract method, it should be overridden in child classes

class qarnot.advanced_bucket.BucketPrefixFiltering(prefix: str)[source]

Bases: qarnot.advanced_bucket.AbstractFiltering

Allows to filter a resources bucket by name prefix. Only bucket files starting with the given prefix will be used as task resources.

__init__(prefix: str)[source]
classmethod from_json(json: Dict[str, str])[source]

Static method called to create the class obj from a json object.

Parameters

json (dict) – the json elements of the class

to_json() object[source]

Get a dict ready to be json packed.

Returns

the json elements of the class.

Return type

dict

sanitize_filter_paths(show_warnings: bool)[source]

Sanitize bucket prefix path

class qarnot.advanced_bucket.Filtering[source]

Bases: object

Groups the various object filters on an advanced resources bucket.

__init__()[source]
append(filtering: qarnot.advanced_bucket.AbstractFiltering)[source]

Add a new filtering object

Parameters

filtering (AbstractFiltering) – a filtering object of the bucket

classmethod from_json(json) qarnot.advanced_bucket.Filtering[source]

Create the class sub objects of a Filtering from a json.

Parameters

json (dict) – the json elements of the class

to_json() Dict[str, Any][source]

Get a dict ready to be json packed.

class qarnot.advanced_bucket.AbstractResourcesTransformation[source]

Bases: object

Abstract base class for resources transformation, allowing to transform bucket objects before they are presented to the task as resources.

abstract to_json()[source]

Get a dict ready to be json packed.

abstract classmethod from_json(json)[source]

Static method called to create the class obj from a json object.

Parameters

json (dict) – the json elements of the class

abstract sanitize_transformation_paths(show_warnings: bool)[source]

Sanitize the bucket path by removing extra separators

Raises

NotImplementedError – this is an abstract method, it should be overridden in child classes

class qarnot.advanced_bucket.PrefixResourcesTransformation(prefix: str)[source]

Bases: qarnot.advanced_bucket.AbstractResourcesTransformation

Allows to remove a prefix from bucket files before they are presented to the task. During execution, the task will see files with paths stripped of the given prefix in its working directory.

__init__(prefix: str)[source]

The PrefixResourcesTransformation constructor

Parameters

prefix (str) – the prefix path of the resource bucket to be removed.

classmethod from_json(json)[source]

Create a new instance of the class using the given json object.

Parameters

json (dict) – The Json object representation.

Returns

The PrefixResourcesTransformation new object

Return type

PrefixResourcesTransformation

to_json() object[source]

Get a dict ready to be json packed.

sanitize_transformation_paths(show_warnings: bool)[source]

Sanitize bucket strip prefix path

class qarnot.advanced_bucket.ResourcesTransformation[source]

Bases: object

Groups the various object transformation on an advanced resources bucket.

__init__()[source]
append(resource: qarnot.advanced_bucket.AbstractResourcesTransformation)[source]

Add a new resource transformation object

Parameters

filtering (AbstractResourcesTransformation) – a filtering object of the bucket

classmethod from_json(json) qarnot.advanced_bucket.ResourcesTransformation[source]

Create the class sub objects of a ResourcesTransformation from a json.

Parameters

json (dict) – the json elements of the class

to_json() object[source]

Get a dict ready to be json packed.