Bucket

Module for bucket object.

class qarnot.bucket.Bucket(connection, name, create=True, filtering: Optional[qarnot.advanced_bucket.Filtering] = None, resources_transformation: Optional[qarnot.advanced_bucket.ResourcesTransformation] = None, cacheTTLSec: Optional[int] = None)[source]

Bases: qarnot.storage.Storage

Represents a resource/result bucket.

This class is the interface to manage resources or results from a Bucket.

Raises

BucketStorageUnavailableException – the bucket storage engine is not available

Note

A Bucket must be created with Connection.create_bucket() or retrieved with Connection.buckets(), Connection.retrieve_bucket(), or Connection.retrieve_or_create_bucket().

Note

Paths given as ‘remote’ arguments, (or as path arguments for Bucket.directory()) must be valid unix-like paths.

__init__(connection, name, create=True, filtering: Optional[qarnot.advanced_bucket.Filtering] = None, resources_transformation: Optional[qarnot.advanced_bucket.ResourcesTransformation] = None, cacheTTLSec: Optional[int] = None)[source]
to_json()[source]

Get a dict ready to be json packed from this bucket.

classmethod from_json(connection, json_bucket)[source]

Create a Bucket object from a json advance bucket.

Parameters

  • connection (Connection) – the cluster connection

  • json_bucket (dict) – Dictionary representing the bucket

Returns

The created Bucket.

with_filtering(filtering)[source]

Create a new Bucket object from the given bucket with a specific filtering.

examples:

filtered_bucket = bucket.with_filtering(BucketPrefixFiltering("prefix1"))

other_filtered_bucket = Bucket(connection, "name", False).with_filtering(BucketPrefixFiltering("prefix1"))
Parameters

filtering (AbstractFiltering) – Filtering to add to the bucket.

Returns

The created Bucket.

with_resource_transformation(resource)[source]

Create a new Bucket object from the given bucket with a specific resource transformation.

examples:

trans_bucket = Bucket(connection, "name", False).with_resource_transformation(PrefixResourcesTransformation("prefix2"))

trans_filtered_bucket = bucket.with_resource_transformation(PrefixResourcesTransformation("prefix2")).with_filtering(BucketPrefixFiltering("prefix1"))
Parameters

resource (AbstractResourcesTransformation) – The resource transformation to add to the bucket.

Returns

The created Bucket.

with_cache_ttl(ttl: int)[source]

Create a new Bucket object from the given bucket with a specific cache ttl (in seconds).

examples:

new_bucket = bucket.with_cache_ttl(2592000)

new_bucket = Bucket(connection, "name", False).with_cache_ttl(2592000)
Parameters

ttl (int) – Time to live for the bucket resource cache.

Returns

The created Bucket.

delete()[source]

Delete the bucket represented by this Bucket.

list_files()[source]

List files in the bucket

Return type

list(S3.ObjectSummary)

Returns

A list of ObjectSummary resources

directory(directory='')[source]

List files in a directory of the bucket according to prefix.

Return type

list(S3.ObjectSummary)

Returns

A list of ObjectSummary resources

sync_directory(directory, verbose=False, remote=None)[source]

Synchronize a local directory with the remote buckets.

Parameters

  • directory (str) – The local directory to use for synchronization

  • verbose (bool) – Print information about synchronization operations

  • remote (str) – path of the directory on remote node (defaults to local)

Warning

Local changes are reflected on the server, a file present on the bucket but not in the local directory will be deleted from the bucket.

A file present in the directory but not in the bucket will be uploaded.

Note

The following parameters are used to determine whether synchronization is required :

  • name

  • size

  • sha1sum

sync_files(files, verbose=False, remote=None)[source]

Synchronize files with the remote buckets.

Parameters

  • files (dict) – Dictionary of synchronized files

  • verbose (bool) – Print information about synchronization operations

  • remote (str) – path of the directory on remote node (defaults to local)

Dictionary key is the remote file path while value is the local file path.

Warning

Local changes are reflected on the server, a file present on the bucket but not in the local directory will be deleted from the bucket.

A file present in the directory but not in the bucket will be uploaded.

Note

The following parameters are used to determine whether synchronization is required :

  • name

  • size

  • sha1sum

add_string(string, remote)[source]

Add a string on the storage.

Parameters

  • string (str) – the string to add

  • remote (str) – name of the remote file

add_file(local_or_file, remote=None)[source]

Add a local file or a Python File on the storage.

Note

You can also use object[remote] = local

Parameters

  • local_or_file (str or File) – path of the local file or an opened Python File

  • remote (str) – name of the remote file (defaults to local_or_file)

get_all_files(output_dir, progress=None)[source]

Get all files from the storage.

Parameters

  • output_dir (str) – local directory for the retrieved files.

  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises

Warning

Will override output_dir content.

get_file(remote, local=None, progress=None)[source]

Get a file from the storage. Create needed subfolders.

Parameters

  • remote (str) – the name of the remote file

  • local (str) – local name of the retrieved file (defaults to remote)

  • progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Return type

str

Returns

The name of the output file.

Raises

ValueError – no such file

add_directory(local, remote='')[source]

Add a directory to the storage. Does not follow symlinks. File hierarchy is preserved.

Parameters

  • local (str) – path of the local directory to add

  • remote (str) – path of the directory on remote node (defaults to local)

Raises

IOError – not a valid directory

copy_file(source, dest)[source]

Create a copy of a file

Parameters

  • source (str) – name of the existing file to duplicate

  • dest (str) – name of the created file

flush()[source]

Ensure all background uploads are complete

Deprecated since version 2.6.0: This will be removed in 3.0. Legacy function

Deprecated since version 2.6.0: This will be removed in 3.0. Legacy function

update(flush=False)[source]

Update object from remote endpoint

Parameters

flush (bool) – bypass cache

Deprecated since version 2.6.0: This will be removed in 3.0. Legacy function

Deprecated since version 2.6.0: This will be removed in 3.0. Legacy function

delete_file(remote)[source]

Delete a file from the storage.

Parameters

remote (str) – the name of the remote file

property uuid

Bucket identifier

property description

Bucket identifier