Using Curl to access AWS S3#

In some situations, users may wish to upload content to S3 or download content from an S3 bucket. For example, when attempting manual backups of QHub’s data.

In many cases, the most straightforward way to access AWS S3 buckets is by installing and using AWS’s command-line tool. But in some situations - for example, to back up the JupyterHub SQLite database - it may be difficult to install AWS’ CLI tools due to being in a restricted container environment. In that situation, it is possible to fall back on AWS’ basic REST API and use HTTPS requests directly instead. (Ultimately, the AWS CLI is simply a wrapper around those REST APIs.)

This document describes how to use curl commands to interface with S3 directly, specifically in the case of uploading a backup of JupyterHub’s SQLite database from a restricted pod to S3 (or restoring it from a backup from S3).

Common settings#

These settings will be needed whether uploading or downloading:

s3_access_key=<AWS Key>
s3_secret_key=<AWS Secret>
region=us-west-2
bucket=<AWS bucket name>

Upload to s3#

# File locations
file_to_upload=/srv/jupyterhub/jupyterhub.sqlite
output_filename=backups/jupyterhub.sqlite

# Metadata
filepath="/${bucket}/${output_filename}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"

#prepare signature hash to be sent in Authorization header
signature_hash=`echo -en ${signature_string} | openssl sha1 -hmac ${s3_secret_key} -binary | base64`

# Send file to S3 using Curl
curl -X PUT -T "${file_to_upload}" \
  -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
  https://${bucket}.s3-${region}.amazonaws.com/${output_filename}

Download from S3#

output_file=/srv/jupyterhub/jupyterhub.sqlite
s3_file=backups/jupyterhub.sqlite

# Metadata
resource="/${bucket}/${s3_file}"
contentType="binary/octet-stream"
dateValue=`TZ=GMT date -R`
# You can leave out "TZ=GMT" if your system is already GMT (but don't have to)

stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"

signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3_secret_key} -binary | base64`

# Fetch file from S3 using curl
curl -H "Host: s3-${region}.amazonaws.com" \
     -H "Date: ${dateValue}" \
     -H "Content-Type: ${contentType}" \
     -H "Authorization: AWS ${s3_access_key}:${signature}" \
     https://s3-${region}.amazonaws.com/${bucket}/${s3_file} -o $output_file

Inspired by this article on how to use curl to upload files to was s3 and this StackOverflow answer on how to access was s3 buckets.