Quick, Cheap and Simple: Backups with s3sync

Here are some quick notes on the step-by-step process I used to set up automated backups to Amazon’s S3 on my linux server. Not much is Ubuntu, or even linux, specfic, so it should be quite applicable for other platforms.

Create an amazon s3 account

• Visit Amazon’s S3 Site
• Create an Account
• Enable S3 on your account
• Hover over the “Your Web Services button,” go to “AWS Access Identifiers”
• Grab your Access ID and Secret Key, remember them for later

Get s3sync

• Install ruby >= 1.8.4, if you don’t already have it

 sudo apt-get install ruby

• Install openssl for ruby, if needed

 sudo apt-get install libopenssl-ruby

• Download s3sync

  cd ~
  wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz

• Put it somewhere (I chose /usr/local/s3sync)

  cd /usr/local
  sudo tar xzfv ~/s3sync.tar.gz

Configure s3sync

• Copy example config to config file

  cd /usr/local/s3sync
  cp s3config.yml.example s3sync.yml

• Make sure and set permissions on this file, probably (root.root 600), so others can’t nab your key
• Edit s3sync.yml, put your own Keys in there
• I set the ssl_cert_dir to /usr/local/s3sync/certs and created that directory

Get SSL certs

• I gathered these instructions in s3sync/README.txt
• Download cert file

 wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar?rev=1.10;content-type=application%2Fx-shar

• Run this in /usr/local/s3sync/certs

Create bucket

• Pick a name for your bucket that you’ll remember
• Create the bucket

  cd /usr/local/s3sync
  ./s3cmd.rb -s createbucket bucketname

Now you can sync

• Do a dry run:

  cd /usr/local/s3sync;
  ./s3sync.rb -snrv --delete /etc/ bucketname:etc

• If it looks good, remove the -n (dry run)

 ./s3sync.rb -srv --delete /etc/ bucketname:etc

• You can use the the JeS3t applet to verify what you uploaded, it’s a java applet designed for examing and manipulating your s3 buckets.
  Go to the “Direct Login” tab and put in your access id and secret key.

• Rinse and repeat with other directories you wish to backup.

Cron it

I created a simple script and threw it in /etc/cron.weekly/s3backup

 #!/bin/bash

SARGS=“-vs” # you can add -n in here for testing, remove s for non-ssl, remove verbosity, etc SENDMAIL=/usr/sbin/sendmail EMAIL=myuser@mydomain cd /usr/local/s3sync echo -e “To: ${EMAIL}\nSubject: s3backup results\nContent-type: text/plain\n\n” > /tmp/s3backup.log pg_dumpall -Ucdmoyer | gzip > /opt/www/db/pg_dumpall.gz #/root/.pgpass is needed for this ./s3sync.rb ${SARGS} —delete -r /etc/ bucketname:etc » /tmp/s3backup.log ./s3sync.rb ${SARGS} —delete -r /opt/ bucketname:opt » /tmp/s3backup.log ./s3sync.rb ${SARGS} —delete -r /var/backups/ bucketname:var/backups » /tmp/s3backup.log cat /tmp/s3backup.log | ${SENDMAIL} “${EMAIL}”

Notes & Cost

This was originally done on Ubuntu Gutsy Gibon, so your mileage may vary.

Things I did:

• backed up /etc, deleted it, did it again
• backed up 700M in /opt
• backed up a couple Megs in /var/backups
• deleted stuff from my home directory, taking /opt back to 300M
• ran s3backup several times for testing.

Total Cost: $0.29 so far, and it will cost me $0.15 at the end of the month. It seems to cost a bit less than a penny each time I sync.