Uploading files to AARNet CloudStor from Linux command line

[gdt]

Uploading files to CloudStor is conveniently done through the CloudStor web page. Files can also be uploaded through GNOME's inbuilt WebDav client, see the Places | Browse networks | Other locations... dialogue.

When uploading many large files using the command line can be more straightforward then a graphical interface, as the transfer can be set running and forgot until it completes.

Install cURL. On Fedora this is:

$ sudo dnf install curl

Create a CloudStor app password. Go to cloudstor.aarnet.edu.au, log in, Select your username | Settings | Security | Create new app password. Give the app a name, like "webdavs", the name has no meaning. Copy the username and password from the displayed text boxes:

Username: fred.bloggs@example.edu.au
Password: JQRKN-EEMDI-NSPOB-ABKIE

You can use this password for all future webdav uploads.

Upload files. We'll upload example.dat to our CloudStor home folder:

$ curl -T example.dat https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/

When prompted provide your username -- complete with the @example.edu.au -- and the app password, not your real password which you logged into Shibbloeth with when connecting to the CloudStor web site.

The -o option gives upload progress:

$ curl -T example.dat -o /dev/stdout https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/

To upload to a folder: create the folder from the CloudStor webpage, then append the folder name and a trailing slash to the upload URL. Here's an example for a folder called "experiment51":

$ curl -T example.dat https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/

Uploading a lot of files. Curl allows the username and password to be supplied on the command line. This password is the app password, not your real password:

$ curl -T example.dat -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/

We can combine that with the usual UNIX shell facilities. Here we upload every .dat file in our computer's working directory to the "experiment51" folder on our CloudStor:

$ for f in *.dat
do curl -T "$f" -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/
done

It's going to take a really long time. UNIX's nohup program allows you to log out but leaves the program running. Put your commands between nohup and a &:

$ nohup for f in *.dat
do curl -T "$f" -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/
done &

If it goes wrong. Any errors are printed in their raw XML form. Just ignore the XML tags and read the text between them. Here uploading a file hasn't gone well as the uploaded size doesn't match the size on disk of the original file:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>expected filesize 127076912 got 127074304</s:message>
</d:error>

When you are done. You can return your CloudStor account's Security | Settings and delete the "webdavs" app you created from the list of apps and their passwords.