PyTorch on Qarnot Cloud

In this use case, we focus on the Pytorch library, an open source machine learning framework popular in the operational field for its ease of deployment in production. This quick tutorial will walk you through the necessary steps to train a Pytorch neural network for weather prediction on Qarnot.

Version

• Version: 1.6.0
• Release year: 2020

If you are interested in another version, please send us an email at qlab@qarnot.com.

Prerequisites

Before starting a calculation with the Python SDK, a few steps are required:

• Retrieve the authentication token
• Install Qarnot’s Python SDK

Note: in addition to the Python SDK, Qarnot provides C# and Node.js SDKs and a Command Line.

Test Case

The case presented here uses the weather dataset extracted from Rain in Australia. It contains 142,193 samples of daily weather data from several regions of Australia. This problem is treated as a case of binary classification, 0 indicating that it does not rain and 1 indicating that it rains.

Before moving forward, you should setup your working environment to contain the following files:

• input
  • rain_pytorch.py: python script training a neural network of the dataset at once. Can be found here.‍
  • weatherAUS.csv: dataset on the meteorological conditions of several days in an Australian region. Can be downloaded here.‍
• pytorch.py: script python pour lancer la tâche sur Qarnot (ci-dessous).

Launching the test case

Once everything is set up, use the following script in your favorite interpreter or compiler to launch the computation on Qarnot.

Be sure to copy your authentication token in the script (instead of <<<MY_SECRET_TOKEN>>>) to be able to launch the task on Qarnot.

To launch this script, simply copy the following code in a Python script and execute python3 pytorch.py & in your terminal.

#!/usr/bin/env python

# Import the Qarnot SDK
import qarnot

# Connect to the Qarnot platform
conn = qarnot.Connection(client_token='<<<MY_SECRET_TOKEN>>>')

# Create a task
task = conn.create_task('Hello World - Pytorch', 'docker-batch', 1)

# Create a resource bucket and add input files
input_bucket = conn.create_bucket('pytorch-in')
input_bucket.sync_directory('input')

# Attach the bucket to the task
task.resources.append(input_bucket)

# Create a result bucket and attach it to the task
task.results = conn.create_bucket('pytorch-out')

# Give parameters regarding the Docker image to be used
task.constants['DOCKER_REPO'] = "qarnotlab/pytorch-gpu"
task.constants['DOCKER_TAG'] = "v1"
task.constants['DOCKER_CMD'] = "python3 rain_pytorch.py"

# Submit the task to the Api
task.run(output_dir='outputs')

Results

At any given time, you can monitor the status of your task.

Once training is done, the task status should change to green. You can then check out the task's output bucket pytorch-out. There you will find various files like a training log, the predictions made and the actual trained model.

Wrapping up

That’s it! If you have any questions, please contact qlab@qarnot.com and we will help you with pleasure!