In A Row

Get your ducks there!

home :: linux :: backups with s3sync

:: Quick, Cheap and Simple: Backups with s3sync

2007.12.15

Here are some quick notes on the step-by-step process I used to set up automated backups to Amazon's S3 on my linux server. Not much is Ubuntu, or even linux, specfic, so it should be quite applicable for other platforms.

Create an amazon s3 account

  • Visit Amazon's S3 Site
  • Create an Account
  • Enable S3 on your account
  • Hover over the "Your Web Services button," go to "AWS Access Identifiers"
  • Grab your Access ID and Secret Key, remember them for later

Get s3sync

  • Install ruby >= 1.8.4, if you don't already have it
 sudo apt-get install ruby
  • Install openssl for ruby, if needed
 sudo apt-get install libopenssl-ruby
  • Download s3sync
  cd ~
  wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
  • Put it somewhere (I chose /usr/local/s3sync)
  cd /usr/local
  sudo tar xzfv ~/s3sync.tar.gz

Configure s3sync

  • Copy example config to config file
  cd /usr/local/s3sync
  cp s3config.yml.example s3sync.yml
  • Make sure and set permissions on this file, probably (root.root 600), so others can't nab your key
  • Edit s3sync.yml, put your own Keys in there
  • I set the ssl_cert_dir to /usr/local/s3sync/certs and created that directory

Get SSL certs

  • I gathered these instructions in s3sync/README.txt
  • Download cert file
 wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar?rev=1.10;content-type=application%2Fx-shar
  • Run this in /usr/local/s3sync/certs

Create bucket

  • Pick a name for your bucket that you'll remember
  • Create the bucket
  cd /usr/local/s3sync
  ./s3cmd.rb -s createbucket bucketname

Now you can sync

  • Do a dry run:
  cd /usr/local/s3sync;
  ./s3sync.rb -snrv --delete /etc/ bucketname:etc
  • If it looks good, remove the -n (dry run)
 ./s3sync.rb -srv --delete /etc/ bucketname:etc
  • You can use the the JeS3t applet to verify what you uploaded, it's a java applet designed for examing and manipulating your s3 buckets.
    Go to the "Direct Login" tab and put in your access id and secret key.
  • Rinse and repeat with other directories you wish to backup.

Cron it

I created a simple script and threw it in /etc/cron.weekly/s3backup

 #!/bin/bash
  
 SARGS="-vs"  # you can add -n in here for testing, remove s for non-ssl, remove verbosity, etc
 SENDMAIL=/usr/sbin/sendmail
 EMAIL=myuser@mydomain
  
 cd /usr/local/s3sync
  
 echo -e "To: ${EMAIL}\nSubject: s3backup results\nContent-type: text/plain\n\n" > /tmp/s3backup.log
  
 pg_dumpall -Ucdmoyer | gzip > /opt/www/db/pg_dumpall.gz  #/root/.pgpass is needed for this
 ./s3sync.rb  ${SARGS} --delete -r /etc/ bucketname:etc >> /tmp/s3backup.log
 ./s3sync.rb  ${SARGS} --delete -r /opt/ bucketname:opt >> /tmp/s3backup.log
 ./s3sync.rb  ${SARGS} --delete -r /var/backups/ bucketname:var/backups >> /tmp/s3backup.log
  
 cat /tmp/s3backup.log | ${SENDMAIL} "${EMAIL}"

Notes & Cost

This was originally done on Ubuntu Gutsy Gibon, so your mileage may vary.

Things I did:
* backed up /etc, deleted it, did it again
* backed up 700M in /opt
* backed up a couple Megs in /var/backups
* deleted stuff from my home directory, taking /opt back to 300M
* ran s3backup several times for testing.

Total Cost: $0.29 so far, and it will cost me $0.15 at the end of the month. It seems to cost a bit less than a penny each time I sync.

Comments

LV handbags :: Tue, 28 Dec 2010 21:14

I like it. It is great to see the great videos.

I like it interesting photos.your themes and visuals are very cool…

Harrisburg web design :: Wed, 06 Apr 2011 09:57

Good work! Your post is an excellent example of why I keep coming back to read your excellent quality content that is forever updated. Thank you!

Mumbai Escorts :: Thu, 25 Aug 2011 11:24

Unbelievable Read! Looking forward to more bookmarked the site.

Mumbai Escorts :: Tue, 13 Sep 2011 11:12

I find this information very useful and it has considerably saved my time.thanks

credit score repair :: Tue, 27 Sep 2011 10:34

One benefit over some other comparable tools is that s3sync goes out of its way to mirror the directory structure on S3.

home security miami :: Sat, 12 Nov 2011 14:33

I tend to ignore this as it is an old post. But suddenly thought of asking my IE team to look at and see if this works.

Cheap handbags :: Fri, 25 Nov 2011 01:49

Just hope that you will keep publishing such great ones in the future too.

Moncler Women Coats :: Mon, 28 Nov 2011 01:19

Moncler Women Coats

Christian Louboutin Sale :: Mon, 28 Nov 2011 01:26

nice post

Moncler Vest Sale :: Mon, 28 Nov 2011 01:32

ok i like it

Moncler Down Jackets :: Mon, 28 Nov 2011 01:43

followed as shorts do look superb with Moncler Down Jackets provided they are flawlessly tailored. Even a Cheap Moncler Sale will also be preferred provided you skip knee highs and avoid socks,long Moncler Mens Coats representing a decent look goes with the atmosphere of a boardroom or Bar Mitzvah only. For rest of your life, hemming your moncler vests men sale will be perfect for your relaxed look. The Mens Moncler Jackets Branson Navy Blue can be bare but if covered with pop colored socks will add a nice touch. http://www.monclermensales.com/

Moncler Down Jackets :: Mon, 28 Nov 2011 01:44

followed as shorts do look superb with Moncler Down Jackets provided they are flawlessly tailored. Even a Cheap Moncler Sale will also be preferred provided you skip knee highs and avoid socks,long Moncler Mens Coats representing a decent look goes with the atmosphere of a boardroom or Bar Mitzvah only. For rest of your life, hemming your moncler vests men sale will be perfect for your relaxed look. The Mens Moncler Jackets Branson Navy Blue can be bare but if covered with pop colored socks will add a nice touch. http://www.monclermensales.com/

Moncler Mens Jackets :: Mon, 28 Nov 2011 01:53

Although Cheap Moncler Uk stated that the tax system shouldn't pick winners and losers, he contradicted himself with Moncler Jackets Sale. If the government maintains the mortgage interest deduction, as Moncler Mens Jackets wants, then selected winners are those Moncler Coats Uk Online enough to have sufficient income and good credit to obtain a mortgage. Selected Moncler Womens Jackets are anyone who doesn't own a piece of winter jackets and those who've paid off their http://www.moncleruk4sale.com/.

Christian Louboutin Boots :: Mon, 28 Nov 2011 02:01

Christian Louboutin Boots are extremely mechanical inside the procedure of producing or breaking any dressing Discount Christian Louboutin Sale. And no one of us would actually want their respective type to go as Sales Louboutin. And women are especially extra delicate with these problems associated to styling than their Cheap Louboutin Pumps. These Fashion Shoes play a pivotal transfer from transforming a flop model to Christian Louboutin Sandals as they gel correctly even together with your standard Christian Louboutin Booties. So, make sure to play it intelligent on this occasion, go for http://www.saleslouboutinboots.com/ and get prepared to get all these appreciative attentions. For the nba, and for all those nba shoes this year, you can’t go wrong with nba players shoes. Basketball shoes are men’s favorite that never really seem to completely go out of style although cheap nba jerseys can wax and wane as often as the moon. They’re superb in nba shoes sales, but beyond these locations could perhaps not be the finest choice. Still, they do have their place and selected best nba shoes can utilize an outstanding pair of Kobe Bryant Shoes much more effectively than others. But moving beyond these basic carefree Dwight Howard Shoes, there can be plenty of comfy, cool sneakers that are going to http://www.nbaplayerssshoe.com/favored all through the coming year.

north face outlet :: Wed, 30 Nov 2011 03:02

This is a great news,which has a lot of readers.Such kind of post giving us various knowaledge,it worthy to read.It is so wonderful.

Mens Designer Clothing :: Wed, 30 Nov 2011 08:06

Mens Designer Clothing

new orleans home security :: Mon, 05 Dec 2011 18:41

Thanks for the detailed instructions on s3sync..hope to get this done.

Womens Designer Clothing :: Thu, 08 Dec 2011 09:08

Women enjoy <a href="http://www.womensdesignershop.com/">Womens Designer Clothing</a> and if they're designer the better as this translates to style <a href="http://www.womensdesignershop.com/">Women Shopping</a>, elegance, and quality. <a href="http://www.womensdesignershop.com/">Womens Designer</a> item are made with the finest materials. Fashion is all that everyone wants and right now generation <a href="http://www.womensdesignershop.com/">Womens Designer Shops</a> matters much more than money.

thesis :: Wed, 14 Dec 2011 10:13

Great stuff, worth reading. Thanks for sharing!

A Home Security :: Sun, 18 Dec 2011 05:43

Thanks for this coding help...our IT team benefited from this. They were looking for this around for some time and I got hold of this at right time.

Add a Comment:
 
Name:
URL/Email: [http://... or mailto:you@wherever] (optional)
Comment:
Please verify that you are human, or a good AI/OCR combo:
  Add to Technorati Favorites   Copyright 2008, Chris Moyer