Disaster Recovery using Offsite backup and Data escrow

From CoCCA Registry Services (NZ) Limited

Jump to: navigation, search

Contents

Backup and Recovery

The backup/recovery procedure goal is full recovery from any failures without any loss of data, there are number of backup strategies as we describe in Postgresql_Streaming_Replication, in this section we will discuss other Backup strategies, those strategies are:

      - Normal CoCCA Database Backup/Restore
      - Offsite Full Backup.
      - Data escrow.

Normal CoCCA Database Backup/Restore


Backup using GUI


You can Backup your Registry database using Registry GUI, you can select the backup schedule you would like, then the Registry automation task will generate the Backups and store it according to your Backup Directory configuration.


Restore DataBase

if you need to restore database form backup, which you get through GUI, you should first unzip the backup, it will be like : backup-2015-04-03-08-00-02.bak.zip


Notice: In most cases if you will use the unzip tools you will get error, because the file zipped using Java, this mean if you need to unzip the Backup successfully, you should use jar tools as the following:


First export the JAVA, JAR PATH:


  #export PATH=$PATH:/opt/cocca-8/java/bin:/opt/cocca-8/jre/bin
  #jar -xf backup-2015-04-03-08-00-02.bak.zip

Then you will get the backup.sql file


The final step is import Database using command line:


   # /opt/cocca-8/postgresql/bin/psql -U postgres

        # DROP database registry
        # CREATE database registry
        # \q

   # /opt/cocca-8/postgresql/bin/psql -U postgres  -f  /PATH to/db.registry.psql

Automated Offsite Full Backup

The most important in Registry system is the Backups, you should keep your registry database backuped daily and some times hourly, and you should keep those backups locally and remotely, for this CoCCA enables the Registry administrator to take full Backups and store it offsite as an automation task.


The CoCCA enable the Registry administrator to store Backups offsite through secure transportation using SFTP and encrypt it. to configure the offsite backups you can do that via Registry GUI ----->Maintenance ---> on right side we will find "Backup and Database Maintenance", in this page you can configure the Full Backup Schedule time for take backups locally and offsite using FTP or SFTP, and if you would like to use SSH key and encrypt backups, as the following image:


File:backup_offsite.jpg

In same page you can configure the other options:

- Use SSH key in this case you will use your server private ssh key, and you should send the public key of your server to the remote offsite   
  server to store it in authorized_keys file.
 - if you need to encrypt the Backups to protect the Backup data, you should encrypt the Backup file using the Third Party PGP public key (you 
  should request it), and you will decrypt it using it's private PGP key, to restore it.
 - Notification Email to receive emails when the offsite backup done successfully.

at The same page you will see the following fields:


File:encrypt_backup.jpg


Manually Offsite Full Backup

Protect your Backup files using GPG encryption to protect your sensitive data like users info.

Encrypt The backup files Manually

To encrypt your Backup files using GPG encryption, you can follow the following steps:

1- Create your keys (private and public Keys):

 # gpg --gen-key
 Notice:
 If you will use an already exists keys, you can import those keys via the following commands:
 # gpg --import backup_publicgpg.key
 # gpg --allow-secret-key-import --import priv.key
 But when you use the script at step 3 you will always get message like this:
 "It is NOT certain that the key belongs to the person named in the user ID" and then need to enter yes every time you try to encrypt the backup file.
 The solution is to trust the imported key so you can use the gpg --yes command with file encryption.
 to trust the imported key just do the following:
 # gpg --edit-key 'Keys_Tag_or_ID'
   gpg (GnuPG) 1.4.11; Copyright (C) 2010 Free Software Foundation, Inc.This is free software: you are free to change and redistribute it.
   [Here you will See your keys Details]
   gpg>
   Then you need to enter the following command in gpg prompt:
   gpg> trust
   Please decide how far you trust this user to correctly verify other users' keys
   (by looking at passports, checking fingerprints from different sources, etc.)
   1 = I don't know or won't say
   2 = I do NOT trust
   3 = I trust marginally
   4 = I trust fully
   5 = I trust ultimately
   m = back to the main menu
   Your decision? 5
   Do you really want to set this key to ultimate trust? (y/N) y
   gpg> quit
  

Exiting the gpg program, and Continue the reset steps.


2- use the created keys as a defualt recepint via edit the gpg.conf file.

 # vi /root/.gnupg/gpg.conf
   uncomment the following lines in gpg.conf file:
   #	default-key Key ID
   #   default-recipient  Key ID

3- apply our small script to decrypt all Backup files.

  #!/bin/bash
    ls -l /YOUR/BACKUP/DIRECTORY/ > backups_list.txt
  while IFS= read -r line ||  -n "$line" ; do
           file2encrypt="$(echo $line |awk '{print $9}')"
           field1="$(echo "$file2encrypt" | awk -F- '{print $1}')"
   if [ "$field1" == "backup" ]
         then
           check_encrypted="encrypted-$file2encrypt.ryde"
           echo "$check_encrypted"
          if [ -f "/YOUR/BACKUP/DIRECTORY/$check_encrypted" ]
             then
             echo "Already Encrypted Same File Name"
          else
           echo "Now Begin Encrypt File :$file2encrypt"
           gpg --yes  --recipient "backup keys"  --encrypt --armor  --output "/YOUR/BACKUP/DIRECTORY/encrypted-$file2encrypt.ryde" "/YOUR/BACKUP/DIRECTORY/$file2encrypt"
           echo encryption_DONE
        fi
   fi
 done < backups_list.txt
 

4- Copy the backup Offsite to Secure Backup server using next section "Secure offsite backup".


5- try to decrypt the encrypted backups to ensure every thing work well:

  for example as our script one of the out put files: encrypted-backup-2016-04-24-16-00-03.bak.zip.ryde
  then rune the following command:
  # gpg --output backup-2016-05-05-00-12-14.bak.zip --decrypt encrypted-backup-2016-05-05-00-12-14.bak.zip.ryde
  then try to unzip the decrypt file to ensure your Data is OK:
  # /opt/cocca-8/java/bin/jar -xf  backup-2016-05-05-00-12-14.bak.zip
  after you execute this command you should get this file: backup.sql then you can import it to be sure evry thing is OK.

Off Site Encrypted Backups

The main important task in Registry system is to copy the backups of your database periodically and store it offsite using secure way. There are a lot of backups technology we can use it to take our back and safe it offsite, like streaming replication, static backup, Barman, ... . But the more important is to keep your production server and backup servers secure, by avoid any of the following cases:

- if The production server compromised the backups in backup server shouldn't affect.
- if the backup server compromised should not affect the database in production server.

Most of backup technology uses the automated backups using auto-login server keys without password, this can affect in servers security if the attacker hacked the Backup server then can search about the backup script to find the way to login to the production servers and vice versa.


Off Site Encrypted Backups using Rsync

There are a lot of articles in the internet to secure you backups by use chrooted environment and restrict the backup server to access your production server and use only the required backup command. we will use the simple way by using the rsync command over ssh using keys and restrict the access for specific user and from specific IP. the configuration is very simple: first if we have two servers: - production server which has the SRS database: production.nic.ccTLD , then we can create a backup user into this server : backup@production.nic.ccTLD.

  then we can configure the SRS software to save the SRS database backup files in backup user home directory : /home/backup. 

- Backup server is offsite or inside backup sever backup.nic.ccTLD, which can get (may production can deposits) backup files. The recommended backup direction is the backup servers get the backups from the production server, this mean the backup server will try to use rsync over ssh to production server: - Create the required keys for the backup server user wich will try to ssh the production server, login to backup server using root (or any user will excute the backup cron) then will execute:

         # ssh-keygen
         then it will ask you for directory to save the generated keys (by default it is the home directory), and the passphrase for this keys and we need      
         to leave it empty.

- Now you can look at the home directory for this user we will find the .ssh folder generated and contain the private and public keys:

   $ ls  ~/.ssh
     id_rsa  id_rsa.pub

- Copy the generated public key in backup server and add it to /home/backup/.ssh/authorized_keys in the production server.

  if the directory .ssh does not exists just create it and give it permissions 700 and the owner should be backup user, then create the file authorized_keys 
  and give it permissions 600 and the owner should be backup user.
  The file authorized_keys in production server for the backup user should be similar to this line:
  command="/usr/bin/rsync --server --sender -vlogDtpre.is . /home/backup/backup*",from="backup.nic.ccTLD",no-pty,no-agent-forwarding,no-port-forwarding,
  no-X11-forwarding ssh-rsa AAAA_YOUR_Backup_SERVER_GENERATED_public_KEY== root@backup.nic.ccTLD

- after adding this line to file authorized_keys in production server you can now return to backup server and try to backup the files, you will notice

  when you run the rsync command in backup server you will not use any source path and if you used the production server will ignore it and use one in authorized_keys
  file : "/usr/bin/rsync --server --sender -vlogDtpre.is . /home/backup/backup* "

- Test The backup using new configuration, in backup server run this command:

  $ rsync  -av  backup@production.nic.ccTLD:/  /backu/path/at backup server/
  you can see we donot specify any source path only just the destination backup server.
  then you can add this command to cronjob.

Decrypt encrypted Backup files

  If you encrypt the Backup files using gpg using CoCCA GUI or manually, you need to decrypt those files to be able to import it into postgrSQL Database 
  if you need to restore the database, to decrypt the encrypted backups, for example as our script one of the out put files:                                                
  encrypted-backup-2016-04-24-16-00-03.bak.zip.ryde then rune the following command:
  # gpg --output backup-2016-05-05-00-12-14.bak.zip --decrypt encrypted-backup-2016-05-05-00-12-14.bak.zip.ryde
  then try to unzip the decrypt file to ensure your Data is OK:
  # /opt/cocca-8/java/bin/jar -xf  backup-2016-05-05-00-12-14.bak.zip
  after you execute this command you should get this file: backup.sql then you can import it to be sure evry thing is OK.

Data escrow

Data Escrow is the act of storing data with a neutral third party in case of registry or registrar failure, accreditation termination, or accreditation relapse without renewal. ICANN requires all registrars and gTLD registries to contract with a data escrow provider in order to safeguard registrants.


Configure Data Escrow in CoCCA

"Escrow" which us a daily snapshot of the registry - does not have all the logs and history and could be as much as 24 hours out of date. The escrow deposits are envisioned as a "last resort" and would allow some other entity to restore the ccTLD to a working state in the event of insolvency or a catastrophic failure of ccTLD systems ( no working backups etc ).


CoCCA provide the Data Escrow facility for the ccTLD, you can configure the Escrow files via Registry GUI for each TLD, CoCCA can help the Registry to:


   - Store Data Escrow Offsite to third party, normally NCC Group.
   - Deposit the escrow data files using SFTP or FTP.
   - Use SSH key to deposit the Escrow files.
   - Sign the escrow using a PGP private key.
   - Encrypt the escrow file using a PGP public key to protect the registry data.
   - Upload the escrow report files to ICANN.
   - Run Escrow file in Scheduled time.    

you can configure the Escrow data deposit from GUI ------> Zones ---->choice TLD ---->in right side you will find ICANN Escrow, and you will configure The server directory where automatically generated escrow files will be stored as the following:


File:dir_escrow.jpg


you should also get the Server hostname from Third Party which will store your escrow files offsite, with destination folder and user/password and choice Scheduled time, you will see the following image:


File:escrow_data.jpg


In same page you can configure the other options:

- Use SSH key in this case you will use your server private ssh key, and you should send the public key of your server to the Third party 
  server to store it in authorized_keys file.
- if you need also to Sign the escrow you should use a PGP private key for escrows and you should send the public PGP key to Third party to 
  verify Escrow file signature.
- if you need to encrypt the escrow to protect the escrow data, you should encrypt the escrow files using the Third Party PGP public key (you 
  should request it), and the Third party will decrypt it using it's private PGP key.
- Notification Email to receive emails when the offsite backup done successfully.

at The same page you will see the following fields:


File:sign_escrow.jpg


Create PGP Keys Command Line


To use escrow files you should first create the required key and copy then paste it in ICANN escrow config tab, to do this you can use the GnuPg utility to do that, first you should install it:


     yum install gnupg (or in ubuntu apt-get install gnupg). 

then generate the Key pairs (private and public keys, you will provide the gnupg with the key length, expire time, server name, ... .


    gpg --gen-key 

after you create the keys you can list the created keys and see the key tag as the following:


    gpg --list-keys 

you will see something like this:


     pub   2048R/E83F4295 2015-04-14
     uid                  CoCCA Registry System (escrow files) <noc AT cocca DOT org DOT nz>
     sub   2048R/5DCC3991 2015-04-14

Then you can export thr PGP public key to file:


    $ gpg --output escrowpgp_pub.gpg --armor --export ABCDFE01 

Then you can export the private key to file :


    $ gpg --output escrowpgp_sec.gpg --armor --export-secret-key ABCDFE01  

Create RSA SSH Keys Command Line

To use SSH keys to deposit escrow files offsite, you should first create the required key and copy then paste it in ICANN escrow config tab, to do this you can use the ssh-keygen utility (as root user) to do that, but you should not provide it with the pass phrase:


    # ssh-keygen 
     Enter file in which to save the key (/root/.ssh/id_rsa):
     Enter passphrase (empty for no passphrase):
     Enter same passphrase again:
     Your identification has been saved in /root/.ssh/id_rsa.
     Your public key has been saved in /root/.ssh/id_rsa.pub.

you will find the RSA private key in this path : /root/.ssh/id_rsa


you will find the RSA public key in this path : /root/.ssh/id_rsa.pub


Personal tools