Storage

Storage prototype

class qarnot.storage.Storage

Bases: object

Common architecture for storage providers

get_all_files(output_dir, progress=None)

Get all files from the storage.

Parameters:

• output_dir (str) – local directory for the retrieved files.

• progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Raises:

• MissingBucketException – the bucket is not on the server

• QarnotGenericException – API general error, see message for details

• UnauthorizedException – invalid credentials

Warning

Will override output_dir content.

__init__()

list_files()

List files on the storage.

… note:

File object returned must have a key property.

Returns:

List of the files on the storage.

get_file(remote, local=None, progress=None)

Get a file from the storage. Create needed subfolders.

Parameters:

• remote (str) – the name of the remote file

• local (str) – local name of the retrieved file (defaults to remote)

• progress (bool or function(float, float, str)) – can be a callback (read,total,filename) or True to display a progress bar

Return type:

str

Returns:

The name of the output file.

Raises:

ValueError – no such file

copy_file(source, dest)

Create a copy of a file

Parameters:

• source (str) – name of the existing file to duplicate

• dest (str) – name of the created file

add_directory(local, remote)

Add a directory to the storage. Does not follow symlinks. File hierarchy is preserved.

Parameters:

• local (str) – path of the local directory to add

• remote (str) – path of the directory on remote node (defaults to local)

Raises:

IOError – not a valid directory

add_file(local_or_file, remote)

Add a local file or a Python File on the storage.

Note

You can also use object[remote] = local

Parameters:

• local_or_file (str or File) – path of the local file or an opened Python File

• remote (str) – name of the remote file (defaults to local_or_file)

delete_file(remote)

Delete a file from the storage.

Parameters:

remote (str) – the name of the remote file

update(flush=None)

Update object from remote endpoint

Parameters:

flush (bool) – bypass cache

flush()

Ensure all background uploads are complete