In A Row

Get your ducks there!

home :: linux

:: Debian OpenSSL Security Snafu

Darn, I thought I dealt with this. So, a couple weeks back news broke about the terrible OpenSSL problem that had caused gobs of easily crackable ssl keys to generated on debian and its derivatives, and I read up on it. Being that this is an ubuntu server, I was vulnerable, so I could have swarn I did an apt-get install openssl and got updated versions and regenerated my keys.

Fast forward to yesterday evening. I got an invite to Heroku and was playing with the git interface. I uploaded by public key, and it told me that I had a blacklisted key and pointed me to the ubuntu security warning.

The removed lines:

MD_Update(&m,buf,j);
[ .. ]
MD_Update(&m,buf,j); /* purify complains */

If you didn't hear about the snafu, then you should look into it. If you did, maybe you should look into it again. Took me 10 minutes to fix up 3 bad keys and install the new openssl stuff via apt-get.

(For those who didn't hear: Many moons ago, a line in an openssl file caused a warning in a tool the debian maintainers used. The maintainer removed it, and the complaint went away and everything seemed to work fine. Unfortunately, that line initialized the random number generator. Without that line, every key on a debian system was generated with the random number generator initialized to the process id of the generating program... a tiny, easily bruteforceable set of seeds.)

:: Am I Running out of Memory?

This was originally authored as a reply to a linode forum poster asking if they needed a virtual server with a larger memory allocation.

I think this tends to be one of those things that nobody can agree on, partially because various common linux tools tend to report memory usage in fairly obtuse and different ways. It's often difficult to find out how much memory you are actually using, because of shared libraries, memory, disk buffers, etc.

I find that using several tools helps to give you a better picture of what's going on with your system.

To see the different output, here's a couple different commands on my linode 450:

:::: cdmoyer:5 ~ :::: free -m
             total       used       free     shared    buffers     cached
Mem:           439        430          8          0         80        224
-/+ buffers/cache:        125        313
Swap:          899          0        899
 
:::: cdmoyer:5 ~ :::: vmstat -a -S M
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa
2  0      0      8     65    328    0    0     0     1   10    2  0  0 100  0
 
:::: cdmoyer:5 ~ :::: vmstat -S M
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
1  0      0      8     80    224    0    0     0     1   10    2  0  0 100  0
 
:::: cdmoyer:5 ~ :::: cat /proc/meminfo
MemTotal:       449604 kB
MemFree:          8604 kB
Buffers:         82828 kB
Cached:         229456 kB
SwapCached:        548 kB
Active:         336360 kB
Inactive:        66676 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       449604 kB
LowFree:          8604 kB
SwapTotal:      921592 kB
SwapFree:       920992 kB
Dirty:              44 kB
Writeback:           0 kB
AnonPages:       90204 kB
Mapped:          37308 kB
Slab:            27536 kB
SReclaimable:    20232 kB
SUnreclaim:       7304 kB
PageTables:       1508 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   1146392 kB
Committed_AS:   195216 kB
VmallocTotal:   397304 kB
VmallocUsed:      2836 kB
VmallocChunk:   394468 kB

What you'll see, the longer your system runs, the less free memory you'll have, even if you don't add any processes. This is because of the cache and buffers. These are used by the kernel to speed things up, and not necessarily memory "used" by the system. Especially the cache. The kernel is just taking advantage of the fact that you have spare memory and keeping things read from disk in memory, for performance. Should you fire up more processes (or they start automatically under load), the kernel will give up parts of that space holding older files that haven't been used in a while.

The other thing to note, is in that both of your cases, basically no swap has been or is being used. That's indicative that even though free claims you have limited memory free, you actually have a lot more headroom. If you start to use swap regularly, or have little free memory without it being used by cache, then you need to consider upgrading your RAM.

:: apt-get install sun-java6-jdk

I was looking to play with scala, and needed to install the java jdk. (FYI, this is the post that got me interested in scala, part of a series by Alex Blewitt) I kept reading that you could smply apt-get java now, after Sun had changed their licensing a while back, but it took me quite a bit of googling to find out how. (although, that may just be deficiency on my part, as the answer is pretty obvious if you know about Ubuntu components.)

It turns out you can, you just need to add multiverse to your list of apt components. Modify /etc/apt/sources.list from:

#deb http://archive.ubuntu.com/ubuntu/ gutsy universe 
#deb-src http://archive.ubuntu.com/ubuntu/ gutsy universe

to:

deb http://archive.ubuntu.com/ubuntu/ gutsy universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ gutsy universe multiverse

and then:

sudo apt-get install sun-java6-jdk

Note: I uncommented the universe lines, and added multiverse. The universe component isn't strictly needed for this, but I find it useful. universe is software maintained by the community, but which the Ubuntu maintainers make no claims to support, including security fixes. multiverse software includes software which does not meet the Ubuntu free licensing requirements.

:: Quick, Cheap and Simple: Backups with s3sync

Here are some quick notes on the step-by-step process I used to set up automated backups to Amazon's S3 on my linux server. Not much is Ubuntu, or even linux, specfic, so it should be quite applicable for other platforms.

Create an amazon s3 account

  • Visit Amazon's S3 Site
  • Create an Account
  • Enable S3 on your account
  • Hover over the "Your Web Services button," go to "AWS Access Identifiers"
  • Grab your Access ID and Secret Key, remember them for later

Get s3sync

  • Install ruby >= 1.8.4, if you don't already have it
 sudo apt-get install ruby
  • Install openssl for ruby, if needed
 sudo apt-get install libopenssl-ruby
  • Download s3sync
  cd ~
  wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
  • Put it somewhere (I chose /usr/local/s3sync)
  cd /usr/local
  sudo tar xzfv ~/s3sync.tar.gz

Configure s3sync

  • Copy example config to config file
  cd /usr/local/s3sync
  cp s3config.yml.example s3sync.yml
  • Make sure and set permissions on this file, probably (root.root 600), so others can't nab your key
  • Edit s3sync.yml, put your own Keys in there
  • I set the ssl_cert_dir to /usr/local/s3sync/certs and created that directory

Get SSL certs

  • I gathered these instructions in s3sync/README.txt
  • Download cert file
 wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar?rev=1.10;content-type=application%2Fx-shar
  • Run this in /usr/local/s3sync/certs

Create bucket

  • Pick a name for your bucket that you'll remember
  • Create the bucket
  cd /usr/local/s3sync
  ./s3cmd.rb -s createbucket bucketname

Now you can sync

  • Do a dry run:
  cd /usr/local/s3sync;
  ./s3sync.rb -snrv --delete /etc/ bucketname:etc
  • If it looks good, remove the -n (dry run)
 ./s3sync.rb -srv --delete /etc/ bucketname:etc
  • You can use the the JeS3t applet to verify what you uploaded, it's a java applet designed for examing and manipulating your s3 buckets.
    Go to the "Direct Login" tab and put in your access id and secret key.
  • Rinse and repeat with other directories you wish to backup.

Cron it

I created a simple script and threw it in /etc/cron.weekly/s3backup

 #!/bin/bash
  
 SARGS="-vs"  # you can add -n in here for testing, remove s for non-ssl, remove verbosity, etc
 SENDMAIL=/usr/sbin/sendmail
 EMAIL=myuser@mydomain
  
 cd /usr/local/s3sync
  
 echo -e "To: ${EMAIL}\nSubject: s3backup results\nContent-type: text/plain\n\n" > /tmp/s3backup.log
  
 pg_dumpall -Ucdmoyer | gzip > /opt/www/db/pg_dumpall.gz  #/root/.pgpass is needed for this
 ./s3sync.rb  ${SARGS} --delete -r /etc/ bucketname:etc >> /tmp/s3backup.log
 ./s3sync.rb  ${SARGS} --delete -r /opt/ bucketname:opt >> /tmp/s3backup.log
 ./s3sync.rb  ${SARGS} --delete -r /var/backups/ bucketname:var/backups >> /tmp/s3backup.log
  
 cat /tmp/s3backup.log | ${SENDMAIL} "${EMAIL}"
 

Notes & Cost

This was originally done on Ubuntu Gutsy Gibon, so your mileage may vary.

Things I did:
* backed up /etc, deleted it, did it again
* backed up 700M in /opt
* backed up a couple Megs in /var/backups
* deleted stuff from my home directory, taking /opt back to 300M
* ran s3backup several times for testing.

Total Cost: $0.29 so far, and it will cost me $0.15 at the end of the month. It seems to cost a bit less than a penny each time I sync.

:: Peace of Mind, Moving /etc into Subversion

Having recently moved my various projects to a virtual private server1, I've been spending a fair amount of time mucking about in various files under /etc. Being a programmer, and not a sysadmin, at heart, I have a tendency to make my edits and test them out in a rapid-fire fashion. Most of the time, this is fine, I don't run any high availability services and am fairly careful. But, it does occasionally leave me wondering why or how I changed something... and how to put things back the way they were.

Being a programmer, I tend to live and breathe in version control, so my first thought was to version my config files. My primary concerns were twofold:

  • Security. I don't want to accidentally expose my /etc directory to the world via the anonymous subversion server I run for projects.
  • Breakage. I know many programs (well, at least sshd) are picky about the permissions and ownership of their config files, and didn't want to break things by managing to munge them all.

In my previous usage of subversion, the process of importing a new project has gone something like:

> svn import -m"Initial commit of project foo" foo file:///opt/svn/trunk/
> mv foo foo_back
> svn checkout file:///opt/svn/trunk/foo
> # look around, make sure everything is OK
> rm -rf foo_back

My concern with this was with the process of checking out a whole new copy, worried that the permissions and ownership would change. I understand that subversion is supposed to preserve permissions, but was still concerned. As far as security, it decided to simply create a whole new repository, owned and readable only by root (and outside the path served by svnserve). A short bit of googling later led me to the Subversion FAQ's "How can I do an in-place 'import'". The process turned out to be entirely painless:

> sudo svnadmin create /opt/svn_etc
> cd /opt
> sudo chmod -R go-rwx svn_etc/
> svn mkdir file:///opt/svn_etc/etc -m"make empty dir for etc"
> cd /etc
> sudo svn checkout file:///opt/svn_etc/etc .
> sudo svn add *
> sudo svn commit -m"Initial commit of all /etc"

And, done. To ensure that I never leave some changes dangling, I whipped up a quick shell script and added it to /etc/cron.daily/.2

#!/bin/bash

SENDMAIL=/usr/sbin/sendmail
EMAIL=chris@inarow.net

OUT=` svn status -q /etc `
echo -n "${OUT}" | grep "" > /dev/null && \
	echo -e "To: ${EMAIL}\nSubject: /etc locally modified\nContent-type: text/plain\n\n${OUT}\n" |\
	${SENDMAIL} "${EMAIL}"

For a final check, I verify the local security:

> whoami
cdmoyer
> cd /etc
> svn status passwd
> svn log passwd
svn: Unable to open an ra_local session to URL
svn: Unable to open repository 'file:///opt/svn_etc/etc/passwd'
svn: Can't open file '/opt/svn_etc/etc/passwd/format': Permission denied
> svn diff passwd
> svn diff -r1 passwd
svn: Unable to open an ra_local session to URL
svn: Unable to open repository 'file:///opt/svn_etc/etc/passwd'
svn: Can't open file '/opt/svn_etc/etc/passwd/format': Permission denied
> cd /opt/svn
svn/     svn_etc/
> cd /opt/svn_etc/
-bash: cd: /opt/svn_etc/: Permission denied
> ls /opt/svn
svn/     svn_etc/
> ls /opt/svn_etc/
ls: /opt/svn_etc/: Permission denied
> sudo svn log passwd
[sudo] password for cdmoyer:
------------------------------------------------------------------------
r2 | root | 2007-11-29 21:49:41 -0500 (Thu, 29 Nov 2007) | 1 line

All told, this took about 20 minutes, including the googling and monitoring script. Certainly a worthwhile investment.

1 I'll have to review my new host, Linode, at some point. (OneWordReview: Wow!)

2 And, look at that, running sudo /etc/cron.daily/etctattler and I have a new email.

View the Archives:
  Add to Technorati Favorites   Copyright 2008, Chris Moyer