gdt | Uploading files to AARNet CloudStor from Linux command line

Uploading files to CloudStor is conveniently done through the CloudStor web page. Files can also be uploaded through GNOME's inbuilt WebDav client, see the Places | Browse networks | Other locations... dialogue.

When uploading many large files using the command line can be more straightforward then a graphical interface, as the transfer can be set running and forgot until it completes.

Install cURL. On Fedora this is:

$ sudo dnf install curl

Create a CloudStor app password. Go to cloudstor.aarnet.edu.au, log in, Select your username | Settings | Security | Create new app password. Give the app a name, like "webdavs", the name has no meaning. Copy the username and password from the displayed text boxes:

Username: fred.bloggs@example.edu.au
Password: JQRKN-EEMDI-NSPOB-ABKIE

You can use this password for all future webdav uploads.

Upload files. We'll upload example.dat to our CloudStor home folder:

$ curl -T example.dat https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/

When prompted provide your username -- complete with the @example.edu.au -- and the app password, not your real password which you logged into Shibbloeth with when connecting to the CloudStor web site.

The -o option gives upload progress:

$ curl -T example.dat -o /dev/stdout https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/

To upload to a folder: create the folder from the CloudStor webpage, then append the folder name and a trailing slash to the upload URL. Here's an example for a folder called "experiment51":

$ curl -T example.dat https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/

Uploading a lot of files. Curl allows the username and password to be supplied on the command line. This password is the app password, not your real password:

$ curl -T example.dat -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/

We can combine that with the usual UNIX shell facilities. Here we upload every .dat file in our computer's working directory to the "experiment51" folder on our CloudStor:

$ for f in *.dat
do curl -T "$f" -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/
done

It's going to take a really long time. UNIX's nohup program allows you to log out but leaves the program running. Put your commands between nohup and a &:

$ nohup for f in *.dat
do curl -T "$f" -u 'fred.bloggs@example.edu.au:JQRKN-EEMDI-NSPOB-ABKIE' https://cloudstor.aarnet.edu.au/plus/remote.php/webdav/experiment51/
done &

If it goes wrong. Any errors are printed in their raw XML form. Just ignore the XML tags and read the text between them. Here uploading a file hasn't gone well as the uploaded size doesn't match the size on disk of the original file:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\BadRequest</s:exception>
  <s:message>expected filesize 127076912 got 127074304</s:message>
</d:error>

When you are done. You can return your CloudStor account's Security | Settings and delete the "webdavs" app you created from the list of apps and their passwords.