Quick, Cheap and Simple: Backups with s3sync

Here are some quick notes on the step-by-step process I used to set up automated backups to Amazon's S3 on my linux server. Not much is Ubuntu, or even linux, specfic, so it should be quite applicable for other platforms.

Create an amazon s3 account

Visit Amazon's S3 Site
Create an Account
Enable S3 on your account
Hover over the “Your Web Services button,” go to “AWS Access Identifiers”
Grab your Access ID and Secret Key, remember them for later

Get s3sync

Install ruby >= 1.8.4, if you don't already have it

 sudo apt-get install ruby

Install openssl for ruby, if needed

 sudo apt-get install libopenssl-ruby

Download s3sync

  cd ~

  wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz

Put it somewhere (I chose /usr/local/s3sync)

  cd /usr/local

  sudo tar xzfv ~/s3sync.tar.gz

Configure s3sync

Copy example config to config file

  cd /usr/local/s3sync

  cp s3config.yml.example s3sync.yml

Make sure and set permissions on this file, probably (root.root 600), so others can't nab your key
Edit s3sync.yml, put your own Keys in there
I set the ssl_cert_dir to /usr/local/s3sync/certs and created that directory

Get SSL certs

I gathered these instructions in s3sync/README.txt
Download cert file

 wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar?rev=1.10;content-type=application%2Fx-shar

Run this in /usr/local/s3sync/certs

Create bucket

Pick a name for your bucket that you'll remember
Create the bucket

  cd /usr/local/s3sync

  ./s3cmd.rb -s createbucket bucketname

Now you can sync

Do a dry run:

  cd /usr/local/s3sync;

  ./s3sync.rb -snrv --delete /etc/ bucketname:etc

If it looks good, remove the -n (dry run)

 ./s3sync.rb -srv --delete /etc/ bucketname:etc

You can use the the JeS3t applet to verify what you uploaded, it's a java applet designed for examing and manipulating your s3 buckets.
Go to the “Direct Login” tab and put in your access id and secret key.

Rinse and repeat with other directories you wish to backup.

Cron it

I created a simple script and threw it in /etc/cron.weekly/s3backup

 #!/bin/bash

SARGS=“-vs” # you can add -n in here for testing, remove s for non-ssl, remove verbosity, etc SENDMAIL=/usr/sbin/sendmail EMAIL=myuser@mydomain cd /usr/local/s3sync echo -e “To: ${EMAIL}\nSubject: s3backup results\nContent-type: text/plain\n\n” > /tmp/s3backup.log pg_dumpall -Ucdmoyer | gzip > /opt/www/db/pg_dumpall.gz #/root/.pgpass is needed for this ./s3sync.rb ${SARGS} —delete -r /etc/ bucketname:etc » /tmp/s3backup.log ./s3sync.rb ${SARGS} —delete -r /opt/ bucketname:opt » /tmp/s3backup.log ./s3sync.rb ${SARGS} —delete -r /var/backups/ bucketname:var/backups » /tmp/s3backup.log cat /tmp/s3backup.log | ${SENDMAIL} “${EMAIL}”

Notes & Cost

This was originally done on Ubuntu Gutsy Gibon, so your mileage may vary.

Things I did:

backed up /etc, deleted it, did it again
backed up 700M in /opt
backed up a couple Megs in /var/backups
deleted stuff from my home directory, taking /opt back to 300M
ran s3backup several times for testing.

Total Cost: $0.29 so far, and it will cost me $0.15 at the end of the month. It seems to cost a bit less than a penny each time I sync.

Back to Blog Home »

Hi, I'm Chris Moyer's Blog.