How To:Set up data encryption

From The Open Source Backup Wiki (Amanda, MySQL Backup, BackupPC)
Jump to navigationJump to search

This article is a part of the How Tos collection.

This article needs attention: This is just a collection of notes; it needs testing and writing out as a 'How To' article.

What is needed to recover encrypted tapes

To properly retrieve any encrypted data, the following are needed:

  1. the key (the private key in the public-key encryption case)
  2. the passphrase
  3. the "crypt" program used. Amanda dump file header indicates what crypt program was used. For example:
AMANDA: FILE 20051215 boston.zmanda.com /usr/tmp/gpa2 lev 0 comp .gz program /bin/gtar crypt
enc client_encrypt /usr/local/sbin/amcrypt client_decrypt_option -d
To restore, position tape at start of file and run:
dd if=<tape> bs=32k skip=1 | /usr/local/sbin/amcrypt -d |   /usr/bin/gzip -dc |  /bin/gtar -f...
  • If the key or passphrase is lost or misplaced, the data cannot be recovered.
  • There is no back-door to the encryption algorithm.
  • Proper key management strategy should be in your plan before using data encryption for backup.

amcrypt 'versus' amcrypt-ossl

The amcrypt script will use a combination of aespipe and gpg (GnuPG) to perform symmetric encryption of data. This method is somewhat more difficult to implement than using openssl (through the amcrypt-ossl script), especially in non-Linux systems. First, the need to have a 65 lines key file (generated from /dev/random, piped through uuencode and then head) has a generation procedure (documented here and in the manual page) that is implementation-dependent and will not work smoothly on BSD-derived systems and possibly others. Second, because it seems that aespipe has itself issues with kernels other than Linux. Be sure to use the amaespipe script included in 3.1.1 and later if you are trying to run it amcrypt in non-linux systems. Should be safe to use that script in older amanda deployments, as a drop-in replacement, but remember to always check the resulting encrypted file, and its decryption, after any changes to the script, and use it at your own risk.

The amcrypt-ossl implementation will only require a random passphrase .am_passphrase file and will work out-of-the-box at least also in NetBSD.

Server-side and client side encryption

  • a new dumptype option, encrypt is added.
  • specify either client or server side in the dumptype (not both):
    • encrypt client or encrypt server
  • specify client side encryption program:
    • client_encrypt "your encryption program"
      • a sample encryption/decryption program amcrypt is provided. amcrypt is a wrapper of aespipe.
      • aespipe supports AES128, AES192 and AES256 and it uses SHA-256, SHA-384 and SHA-512 respectively.
      • any encryption/decryption program can be used as long as it reads from stdin and writes to stdout.
    • client_decrypt_option "decrypt parameter" #default to -d
  • specify server side encryption program:
    • server_encrypt "your encryption program"
      • can use amcrypt as in the case of client encryption.
    • server_decrypt_option "decrypt parameter" #default to -d
  • The logic assumes compression then encryption during backup(thus decrypt then uncompress during restore). Specifying client-encryption and server-compression is not supported
  • dumptype sample:
define dumptype server-encrypt-fast {
      global
      program "GNUTAR"
      comment "dump with fast client compression and server symmetric encryption"
      compress client fast
      encrypt  server
      server_encrypt "/usr/local/sbin/amcrypt"
      server_decrypt_option "-d"
}
define dumptype client-encrypt-nocomp {
     global
     program "GNUTAR"
     comment "dump with no ompression and client symmetric encryption"
     compress none
     encrypt client
     client_encrypt "/usr/local/sbin/amcrypt"
     client_decrypt_option "-d"
}
  • To restore client encrypted tape. Do either:

1. take the physical tape to the client machine and do the restore on the client machine where it has the key( am_key.gpg) and passphrase(.am_passphrase).

or

2. take the key and passphrase to the server machine where the tape is located.

amcrypt - GnuPG/aespipe method

Additional packages needed

Setup

  • Configure and compile aespipe:
tar -xjf aespipe-v2.3b.tar.bz2
cd aespipe-v2.3b
./configure
make
make install
  • Generate and store the gpg-key for the Amanda user:
# taken from the aespipe-README
head -c 2925 /dev/random | uuencode -m - | head -n 66 | tail -n 65 | \
gpg --symmetric -a > ~amanda/.gnupg/am_key.gpg

Reading from /dev/random may take indefinitely long if kernel's random entropy pool is empty. If that happens, do some other work on some other console (use keyboard, mouse and disks).

# Additional steps to force correct ownership
chown amanda:disk ~amanda/.gnupg/am_key.gpg
chmod 400 ~amanda/.gnupg/am_key.gpg

Note that the options here are very linux-specific. The idea is to read from /dev/random, uuencode it, and get 65 lines of gibberish. On FreeBSD you have to use something like head -c 4000 /dev/random ... because head -c xxxx /dev/random | uuencode -m - needs to generate 65 full lines of encoded text.

  • This will ask for a passphrase. Remember this passphrase as you will need it in the next step.

Store the passphrase inside the home-directory of the Amanda user and protect it with proper permissions:

echo my_secret_passphrase > ~amanda/.am_passphrase
chown amanda:disk ~amanda/.am_passphrase
chmod 700 ~amanda/.am_passphrase
  • We need this file because we don't want to have to enter the passphrase manually everytime we run amdump. We have to patch bz2aespipe to read the passphrase from a file. I have called that file ~amanda/.am_passphrase.
  • Store the key and the passphrase in some other place as well, without these information you can't access any tapes that have been encrypted with it (this is exactly why we are doing all this, isn't it? ;) ).
  • create amcrypt(or it will available in sourceforge and the rpms) as below:
#!/bin/sh
#
# Original wrapper by Paul Bijnens
#
# adapted by Stefan G. Weichinger
# to enable gpg-encrypted dumps via aespipe
# also adapted by Matthieu Lochegnies for server-side encryption
prefix=/usr/local
exec_prefix=${prefix}
sbindir=${exec_prefix}/sbin
AMANDA_HOME=~amanda
AM_AESPIPE=${exec_prefix}/sbin/amaespipe
AM_PASSPHRASE=$AMANDA_HOME/.am_passphrase

$AM_AESPIPE "$@" 3< $AM_PASSPHRASE
rc=$?
exit $rc


  • create amaespipe(or it will available in sourceforge and the rpms) which is based on wrapper-script bz2aespipe, which comes with the aespipe-tarball:
#! /bin/sh

# FILE FORMAT
# 10 bytes: constant string 'bz2aespipe'
# 10 bytes: itercountk digits
# 1 byte: '0' = AES128, '1' = AES192, '2' = AES256
# 1 byte: '0' = SHA256, '1' = SHA384, '2' = SHA512, '3' = RMD160
# 24 bytes: random seed string
# remaining bytes are bzip2 compressed and aespipe encrypted
# These definitions are only used when encrypting.
# Decryption will autodetect these definitions from archive.
ENCRYPTION=AES256
HASHFUNC=SHA256
ITERCOUNTK=100
AMANDA_HOME=~amanda
WAITSECONDS=1
GPGKEY=""$AMANDA_HOME/.gnupg/am_key.gpg"
FDNUMBER=3
PATH=/usr/bin:/usr/local/bin
export PATH

if test x$1 = x-d ; then
   # decrypt
   n=`head -c 10 - | tr -d -c 0-9a-zA-Z`
   if test x${n} != xbz2aespipe ; then
       echo "bz2aespipe: wrong magic - aborted" >/dev/tty
       exit 1
   fi
   itercountk=`head -c 10 - | tr -d -c 0-9`
   if test x${itercountk} = x ; then itercountk=0; fi
   n=`head -c 1 - | tr -d -c 0-9`
   encryption=AES128
   if test x${n} = x1 ; then encryption=AES192; fi
   if test x${n} = x2 ; then encryption=AES256; fi
   n=`head -c 1 - | tr -d -c 0-9`
   hashfunc=SHA256
   if test x${n} = x1 ; then hashfunc=SHA384; fi
   if test x${n} = x2 ; then hashfunc=SHA512; fi
   if test x${n} = x3 ; then hashfunc=RMD160; fi
   seedstr=`head -c 24 - | tr -d -c 0-9a-zA-Z+/`
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S ${seedstr} -C ${itercountk} -d 
else
   # encrypt
   echo -n bz2aespipe
   echo ${ITERCOUNTK} | awk '{printf "%10u", $1;}'
   n=`echo ${ENCRYPTION} | tr -d -c 0-9`
   aesstr=0
   if test x${n} = x192 ; then aesstr=1; fi
   if test x${n} = x256 ; then aesstr=2; fi
   n=`echo ${HASHFUNC} | tr -d -c 0-9`
   hashstr=0
   if test x${n} = x384 ; then hashstr=1; fi
   if test x${n} = x512 ; then hashstr=2; fi
   if test x${n} = x160 ; then hashstr=3; fi
   seedstr=`head -c 18 /dev/urandom | uuencode -m - | head -n 2 | tail -n 1`
   echo -n ${aesstr}${hashstr}${seedstr}
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
fi
exit 0


Changes from bz2aespipe:

  • Decreased WAITSECONDS: No need to wait for 10 seconds to read the passphrase.
  • Removed bzip2 from the pipes: AMANDA triggers GNU-zip-compression by itself, no need to do this twice (slows down things, blows up size).
  • Added options -K and -p: This enables aespipe to use the generated gpg-key and tells it the number of the file-descriptor to read the passphrase from.

You may set various parameters inside bz2aespipe. You may also call bz2aespipe with various command-line-parameter to choose the encryption-algorithm, hash-function etc. . For a start I have chosen to call bz2aespipe without command-line-options.

amcrypt-ossl - OpenSSL method

Additional Packages Needed

  • OpenSSL

Setup

The OpenSSL method uses only one passphrase to create a symmetric encryption of the data. Just place a (preferably random) strong key in ~amanda/.am_passphrase and add the dumptypes in amanda.conf. The security of your data depends directly on how hard to guess this passphrase is. Be sure to pick a long, random one, and that uses numbers, letters and symbols. You should use programs like pwgen or makepasswd to generate a long (more than 30 chars) random passphrase. Store it in a safe place, besides the .am_passphrase file. There is no way of recovering your data if this passphrase is lost.

Store the passphrase inside the home-directory of the Amanda user and protect it with proper permissions:

echo my_secret_passphrase > ~amanda/.am_passphrase
chown amanda:disk ~amanda/.am_passphrase
chmod 700 ~amanda/.am_passphrase


In amanda.conf:

define dumptype server-encrypt-fast {
      global
      program "GNUTAR"
      comment "dump with fast client compression and server symmetric encryption"
      compress client fast
      encrypt  server
      server_encrypt "/usr/local/sbin/amcrypt-ossl"
      server_decrypt_option "-d"
}

define dumptype client-encrypt-nocomp {
     global
     program "GNUTAR"
     comment "dump with no compression and client symmetric encryption"
     compress none
     encrypt client
     client_encrypt "/usr/local/sbin/amcrypt-ossl"
     client_decrypt_option "-d"
}


Now be sure to use one of these defined dumptypes in the disklist file.

Encryption Verification

(Based on information from http://www.eecs.iu-bremen.de/wiki/index.php/Amanda_%28encryption%29)

To verify backups are encrypted, inspect the dump file header. The restore line should include a reference to the decryption method specified in the dumptype.

fruit:/root$ less /amanda/daily/slot16/00001.kiwi.netlab.eecs.jacobs-university.de._.1 
AMANDA: FILE 20090613000352 kiwi.netlab.eecs.jacobs-university.de /  lev 1 comp N program /bin/tar crypt enc 
client_encrypt /usr/sbin/amcrypt-ossl client_decrypt_option -d
To restore, position tape at start of file and run:
       dd if=<tape> bs=32k skip=1 |      /usr/sbin/amcrypt-ossl -d  |    /bin/tar -xpGf - ...
[...]

Decryption can be verified by running the restore command as specified in the dump file header. Encryption can be verified by running the same command without piping through the decryption mechanism, and verifying the result is indecipherable.

Solaris Notes

Below are notes about setting up amcrypt under Solaris 5.10 with Amanda 2.6.0p2

  • Download gnupg-1.4.8 and build from sources. The sunfreeware version of gnupg did not pass the aespipe tests.
  • Install aespipe: It is important to run the aespipe tests. If the aespipe tests fail, then running amcrypt might fail with "Error: gpg key file decryption failed". As root, run:
wget http://loop-aes.sourceforge.net/aespipe-latest.tar.bz2
gtar -jxf downloads/aespipe-latest.tar.bz2 
cd aespipe-v2.3e/
 ./configure
gmake
gmake tests
gmake install
  • Solaris: As root, install coreutils from http://www.sunfreeware.com so that head and tail have the -c argument. If this head and tail are not updated, then the amdump.1 log file might contain:
yourhost.yourdomain.com /dev/dsk/c1t0d0s0 lev 0  FAILED [data write: Broken pipe]
  • Solaris: As root, edit ~dumper/amanda/sbin/amaespipe so that it uses /usr/local/bin/head and /usr/local/bin/tail. Also, change the tr command to use [:digit:] and [:alnum:]:
#! /bin/bash
#
# Copyright (c) 2005 Zmanda Inc.  All Rights Reserved.
# 
# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License version 2 as published
# by the Free Software Foundation.
# 
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.
# 
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA
# 
# Contact information: Zmanda Inc, 505 N Mathlida Ave, Suite 120
# Sunnyvale, CA 94085, USA, or: http://www.zmanda.com
#

prefix="/usr/local/amanda-2.6.0p2"
exec_prefix="${prefix}"
sbindir="/usr/local/amanda-2.6.0p2/sbin"
amlibexecdir="/usr/local/amanda-2.6.0p2/libexec/amanda"
. "${amlibexecdir}/amanda-sh-lib.sh"

# add sbin and ucb dirs
PATH="$PATH:/usr/sbin:/sbin:/usr/ucb"
export PATH

# wrapper script to use aespipe
# based on bz2aespipe distributed by aespipe from 
# http://loop-aes.sourceforge.net/
# FILE FORMAT
# 10 bytes: constant string 'bz2aespipe'
# 10 bytes: itercountk digits
# 1 byte: '0' = AES128, '1' = AES192, '2' = AES256
# 1 byte: '0' = SHA256, '1' = SHA384, '2' = SHA512, '3' = RMD160
# 24 bytes: random seed string
# remaining bytes are aespipe encrypted

# These definitions are only used when encrypting.
# Decryption will autodetect these definitions from archive.
ENCRYPTION=AES256
HASHFUNC=SHA256
ITERCOUNTK=100
WAITSECONDS=1
AMANDA_HOME=/users/dumper
GPGKEY="$AMANDA_HOME/.gnupg/am_key.gpg"
FDNUMBER=3

if test x$1 = x-d ; then
   # decrypt
   #n=`/usr/local/bin/head -c 10 - | tr -d -c 0-9a-zA-Z`
   n=`/usr/local/bin/head -c 10 - | /usr/bin/tr -d -c '[:alnum:]'`
   if test x${n} != xbz2aespipe ; then
       echo `_ 'bz2aespipe: wrong magic - aborted'` >/dev/tty
       exit 1
   fi
   #itercountk=`/usr/local/bin/head -c 10 - | tr -d -c 0-9`
   itercountk=`/usr/local/bin/head -c 10 - | /usr/bin/tr -d -c '[:digit:]'`
   if test "x${itercountk}" = x ; then itercountk=0; fi
   #n=`/usr/local/bin/head -c 1 - | tr -d -c 0-9`
   n=`/usr/local/bin/head -c 1 - | /usr/bin/tr -d -c '[:digit:]'`
   encryption=AES128
   if test x${n} = x1 ; then encryption=AES192; fi
   if test x${n} = x2 ; then encryption=AES256; fi
   #n=`/usr/local/bin/head -c 1 - | tr -d -c 0-9`
   n=`/usr/local/bin/head -c 1 - | /usr/bin/tr -d -c '[:digit:]'`
   hashfunc=SHA256
   if test x${n} = x1 ; then hashfunc=SHA384; fi
   if test x${n} = x2 ; then hashfunc=SHA512; fi
   if test x${n} = x3 ; then hashfunc=RMD160; fi
   #seedstr=`/usr/local/bin/head -c 24 - | tr -d -c 0-9a-zA-Z+/`
   seedstr=`/usr/local/bin/head -c 24 - | /usr/bin/tr -d -c '[:alnum:]'/`
   #echo "aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S ${seedstr} -C ${itercountk} -d" >> /tmp/amaespipe
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${encryption} -H ${hashfunc} -S ${seedstr} -C ${itercountk} -d
else
   # encrypt
   echo -n bz2aespipe
   echo ${ITERCOUNTK} | awk '{printf "%10u", $1;}'
   #n=`echo ${ENCRYPTION} | tr -d -c 0-9`
   n=`echo ${ENCRYPTION} | /usr/bin/tr -d -c '[:digit:]'`
   aesstr=0
   if test x${n} = x192 ; then aesstr=1; fi
   if test x${n} = x256 ; then aesstr=2; fi
   n=`echo ${HASHFUNC} | /usr/bin/tr -d -c '[:digit:]'`
   hashstr=0
   if test x${n} = x384 ; then hashstr=1; fi
   if test x${n} = x512 ; then hashstr=2; fi
   if test x${n} = x160 ; then hashstr=3; fi
   seedstr=`/usr/local/bin/head -c 18 /dev/urandom | uuencode -m - | /usr/local/bin/head -n 2 | /usr/local/bin/tail -n 1`
   echo -n ${aesstr}${hashstr}${seedstr}
   #echo "encrypt: aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}" >> /tmp/amaespipe
   aespipe -K ${GPGKEY} -p ${FDNUMBER} -e ${ENCRYPTION} -H ${HASHFUNC} -S ${seedstr} -C ${ITERCOUNTK} -w ${WAITSECONDS}
fi
exit 0
  • As dumper, create the keys, which are then encrypted:
mkdir ~/.gnupg
chmod 0700 ~/.gnupg
/usr/local/bin/head -c 5000 /dev/random | uuencode -m - | /usr/local/bin/head -n 66 | /usr/local/bin/tail -n 65 | gpg --symmetric -a > ~dumper/.gnupg/am_key.gpg

Note that the amcrypt man page says to get 2925 characters. This is not enough since when the text is uuencoded, the header will look like:

begin-base64 644 -

The problem is that the line is not 20 characters long. Instead, we get 5000 characters and then grab 65 lines that are at least 20 characters long. This problem ends up causing the "Error: gpg key file decryption failed" message.

  • As dumper, stash the passphrased used above
echo "mysecretpassphrase" > ~dumper/.am_passphrase
chown dumper ~dumper/.am_passphrase
chmod 0700 ~dumper/.am_passphrase
  • Check the test Amanda configuration. The disktype in amanda.conf should include:
encrypt  server
server_encrypt "/usr/local/amanda/sbin/amcrypt"
server_decrypt_option "-d"
  • Try the test dump:
amcheck test
amdump test


Encrypting backup data using hardware accelerator cards

One easy way to encrypt the backup data is to use a PCI-X or PCI-e based encryption accelerator board. A variety of such cards are available and most also store keys securely in hardware, hence there is no need to write down a key or memorize a password.

One choice is Indra Networks' StorSecure family. These products are specifically designed for backup encryption, so they accelerate both compression and encryption. Further, Indra Networks has already integrated with Amanda, so it is a readymade solution. A white paper describing this solution is available.