Centralized Backup Script

Posted by: Admin  :  Category: Web Hosting

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

Hello There!

I thought I’d share a backup script that was written to consolidate backups onto one server instead of spreading the backup process across several servers. The advantages are somewhat obvious to consolidating the script onto one server, namely being that editing or making changes is much easier as you only have one script to edit.

The environment where this may be ideal would be for environments with 15-20 servers or less. I’d recommend a complete end-to-end backup solution for servers that exceed that number such as Bacula perhaps.

The bash shell script that I pasted below is very straightforward and takes two arguments. The first is the hostname or ip address of the destination server you are backing up. The next (and potentially unlimited) arguments will be single quote encased folders which you would want backed up.

This script is dependent on the server the script is running on having ssh key based authentication enabled and implemented for the root user. Security considerations can be made with IP based restrictions either in the ssh configuration, firewall configuration or other considerations.

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

#!/bin/sh
# Offsite Backup script
# Written by www.stardothosting.com
# Dynamic backup script

currentmonth=`date "+%Y-%m-%d %H:%M:%S"`
currentdate=`date "+%Y-%m-%d%H_%M_%S"`
backup_email="backups@youremail.com"
backupserver="origin-backup-server.hostname.com"

# Check User Input
if [ "$  #" -lt 2 ]
then
        echo -e "nnUsage Syntax :"
        echo -e "./backup.sh [hostname] [folder1] [folder2] [folder3]"
        echo -e "Example : ./backup.sh your-server.com '/etc' '/usr/local/www' '/var/lib/mysql'nn"
        exit 1
fi

# get the server's hostname
host_name=`ssh -l root $  1 "hostname"`
echo "Host name : $  host_name"
if [ "$  host_name" == "" ]
then
        host_name="unknown_$  currentdate"
fi

echo "$  host_name Offsite Backup Report: " $  currentmonth > /var/log/backup.log
echo -e "----------------------------------------------------------" >> /var/log/backup.log
echo -e "" >> /var/log/backup.log

# Ensure permissions are correct
chown -R backups:backups /home/fileserver/backups/
ls -d /home/fileserver/backups/* | grep -v ".ssh|.bash" | xargs -d "n" chmod -R 775

# iterate over user arguments & set error level to 0
errors=0
for arg in "$  {@:2}"
do
        # check if receiving directory exists
        if [ ! -d "/home/fileserver/backups/$  host_name" ]
        then
                mkdir /home/fileserver/backups/$  host_name
        fi
        sanitize=`echo $  arg | sed 's/[^/]/+$   //'`
        sanitize_dir=`echo $  arg | awk -F '/' '{printf "%s", $  2}'`
        /usr/bin/ssh -o ServerAliveInterval=1 -o TCPKeepAlive=yes -l root $  1 "/usr/bin/rsync -ravzp --progress --exclude 'clam_quarantinedir' $  sanitize/ backups@$  backupserver:/home/fileserver/backups/$  host_name/$  sanitize_dir; echo $  ? > /tmp/bu_rlevel.txt" >> /var/log/backup.log 2>&1
        echo "/usr/bin/ssh -o ServerAliveInterval=1 -o TCPKeepAlive=yes -l root $  1 "/usr/bin/rsync -ravzp --progress --exclude 'clam_quarantinedir' $  sanitize/ backups@$  backupserver:/home/fileserver/backups/$  host_name/$  sanitize_dir""

        runlevel=`ssh -l root $  1 "cat /tmp/bu_rlevel.txt"`
        echo "Runlevel : $  runlevel"

        if [ "$  runlevel" -ge 1 ]
        then
                errors=$  ((counter+1))
        else
                echo -e "Script Backup for $  arg Completed Successfully!" >> /var/log/backup.log 2>&1
        fi

done


# Check error level
if [ $  errors -ge 1 ]
then
        echo -e "There were some errors in the backup job for $  host_name, please investigate" >> /var/log/backup.log 2>&1
        cat /var/log/backup.log | mail -s "$  host_name Backup Job failed" $  backup_email
else
        cat /var/log/backup.log | mail -s "$  host_name Backup Job Completed" $  backup_email
fi

It should be explained further that this script actually connects to the destination server as the root user, using ssh key authentication. It then initiates a remote rsync command on the destination server back to the backup server as a user called “backupuser”. So that means that not only does the ssh key need to be installed for root on the destination servers, but a user called “backupuser” needs to be added on the backup server itself, with the ssh keys of all the destination servers installed for the remote rsync.

Hopefully I did not over complicate this, because it really is quite simple :

Backup Server –> root –> destination server to backup — > backupuser rsync –> Backup Server

Once you implement the script and do a few dry run tests then it might be ready to implement a scheduled task for each destination server. Here is an example of one cron entry for a server to be backed up :

01 1 * * * /bin/sh /usr/local/bin/backups.sh destination-server-hostname '/etc' '/usr/local/www' '/home/automysql-backups'

The post Centralized Backup Script appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.

Managed WordPress Hosting | Managed VPS Hosting | Stack Star

Linux Cpanel shared hosting: 600 GB disk space, 6 TB bandwidth, free domain, unlimited databases and ftp accounts, web hosting cheap and pro at Hostony

Centralized Backup Script

Posted by: Admin  :  Category: Web Hosting

Hello There!

I thought I’d share a backup script that was written to consolidate backups onto one server instead of spreading the backup process across several servers. The advantages are somewhat obvious to consolidating the script onto one server, namely being that editing or making changes is much easier as you only have one script to edit.

The environment where this may be ideal would be for environments with 15-20 servers or less. I’d recommend a complete end-to-end backup solution for servers that exceed that number such as Bacula perhaps.

The bash shell script that I pasted below is very straightforward and takes two arguments. The first is the hostname or ip address of the destination server you are backing up. The next (and potentially unlimited) arguments will be single quote encased folders which you would want backed up.

This script is dependent on the server the script is running on having ssh key based authentication enabled and implemented for the root user. Security considerations can be made with IP based restrictions either in the ssh configuration, firewall configuration or other considerations.

#!/bin/sh
# Offsite Backup script
# Written by www.stardothosting.com
# Dynamic backup script

currentmonth=`date "+%Y-%m-%d %H:%M:%S"`
currentdate=`date "+%Y-%m-%d%H_%M_%S"`
backup_email="backups@youremail.com"
backupserver="origin-backup-server.hostname.com"

# Check User Input
if [ "$  #" -lt 2 ]
then
        echo -e "nnUsage Syntax :"
        echo -e "./backup.sh [hostname] [folder1] [folder2] [folder3]"
        echo -e "Example : ./backup.sh your-server.com '/etc' '/usr/local/www' '/var/lib/mysql'nn"
        exit 1
fi

# get the server's hostname
host_name=`ssh -l root $  1 "hostname"`
echo "Host name : $  host_name"
if [ "$  host_name" == "" ]
then
        host_name="unknown_$  currentdate"
fi

echo "$  host_name Offsite Backup Report: " $  currentmonth > /var/log/backup.log
echo -e "----------------------------------------------------------" >> /var/log/backup.log
echo -e "" >> /var/log/backup.log

# Ensure permissions are correct
chown -R backups:backups /home/fileserver/backups/
ls -d /home/fileserver/backups/* | grep -v ".ssh|.bash" | xargs -d "n" chmod -R 775

# iterate over user arguments & set error level to 0
errors=0
for arg in "$  {@:2}"
do
        # check if receiving directory exists
        if [ ! -d "/home/fileserver/backups/$  host_name" ]
        then
                mkdir /home/fileserver/backups/$  host_name
        fi
        sanitize=`echo $  arg | sed 's/[^/]/+$   //'`
        sanitize_dir=`echo $  arg | awk -F '/' '{printf "%s", $  2}'`
        /usr/bin/ssh -o ServerAliveInterval=1 -o TCPKeepAlive=yes -l root $  1 "/usr/bin/rsync -ravzp --progress --exclude 'clam_quarantinedir' $  sanitize/ backups@$  backupserver:/home/fileserver/backups/$  host_name/$  sanitize_dir; echo $  ? > /tmp/bu_rlevel.txt" >> /var/log/backup.log 2>&1
        echo "/usr/bin/ssh -o ServerAliveInterval=1 -o TCPKeepAlive=yes -l root $  1 "/usr/bin/rsync -ravzp --progress --exclude 'clam_quarantinedir' $  sanitize/ backups@$  backupserver:/home/fileserver/backups/$  host_name/$  sanitize_dir""

        runlevel=`ssh -l root $  1 "cat /tmp/bu_rlevel.txt"`
        echo "Runlevel : $  runlevel"

        if [ "$  runlevel" -ge 1 ]
        then
                errors=$  ((counter+1))
        else
                echo -e "Script Backup for $  arg Completed Successfully!" >> /var/log/backup.log 2>&1
        fi

done


# Check error level
if [ $  errors -ge 1 ]
then
        echo -e "There were some errors in the backup job for $  host_name, please investigate" >> /var/log/backup.log 2>&1
        cat /var/log/backup.log | mail -s "$  host_name Backup Job failed" $  backup_email
else
        cat /var/log/backup.log | mail -s "$  host_name Backup Job Completed" $  backup_email
fi

It should be explained further that this script actually connects to the destination server as the root user, using ssh key authentication. It then initiates a remote rsync command on the destination server back to the backup server as a user called “backupuser”. So that means that not only does the ssh key need to be installed for root on the destination servers, but a user called “backupuser” needs to be added on the backup server itself, with the ssh keys of all the destination servers installed for the remote rsync.

Hopefully I did not over complicate this, because it really is quite simple :

Backup Server –> root –> destination server to backup — > backupuser rsync –> Backup Server

Once you implement the script and do a few dry run tests then it might be ready to implement a scheduled task for each destination server. Here is an example of one cron entry for a server to be backed up :

01 1 * * * /bin/sh /usr/local/bin/backups.sh destination-server-hostname '/etc' '/usr/local/www' '/home/automysql-backups'

The post Centralized Backup Script appeared first on Managed WordPress Hosting | Managed VPS Hosting | Stack Star.

Managed WordPress Hosting | Managed VPS Hosting | Stack Star

Centralized remote backup script with SSH key authentication

Posted by: Admin  :  Category: Web Hosting

Greetings,

It has been a while since we posted any useful tidbits for you , so we have decided to share one of our quick & dirty centralized backup scripts.

The script relies on ssh key based authentication, described here on this blog. It essentially parses a configuration file where each variable is separated by a comma and colon, as in the example config here :

hostname1,192.168.1.1,etc:var:root
hostname2,192.168.1.2,etc:var:root:usr

Note the intended backup directories in the 3rd variable, separated by colon’s. Simply populate the backup-hosts.txt config file (located in the same folder as the script) with all the hosts you want to be backed up.

The script then ssh’s to the intended host, and sends a tar -czf stream (securely) over ssh, to be output into the destination of your choice. Ideally you should centralize this script on a box that has direct access to alot of disk space.

Find the script here :

#!/bin/sh
# Centralized Linux Backup Script
# By Star Dot Hosting , www.stardothosting.com
# Uses SSH Key based authentication and remote ssh commands to tar.gz folders to iSCSI storage

todaysdate=`date "+%Y-%m-%d %H:%M:%S"`
backupdest="/backups/linux-backups"

echo "Centralized Linux Backup: " $ todaysdate > /var/log/linux-backup.log
echo -e "———————————————-" >> /var/log/linux-backup.log
echo -e >> /var/log/linux-backup.log

for obj0 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | awk -F "," ‘{printf "%s\n", $ 2}’);
do
        backupname=`cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 1}’`

        for obj1 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 3’} | awk ‘{gsub(":","\n");printf"%s", $
0}’);
        do
                echo -e "backing up $ obj0 with $ obj1 directory" >> /var/log/linux-backup.log
                ssh -l root $ obj0 "(cd /$ obj1/ && tar -czf – . -C /$ obj1)" >> $ backupdest/$ backupname.$ obj1.tar.gz 2>&1
                if [ "$ ?" -eq 1 ]
                then
                        echo -e "There were some errors while backing up $ obj0 / $ backupname within the $ obj1 directory" >> /var/log/linux-backup.log
                        #exit 1
                else
                        echo -e "Backup completed on $ obj0 / $ backupname while backing up $ obj1 directory" >> /var/log/linux-backup.log
                fi
        done
done

echo "Backup Script Completed." >> /var/log/linux-backup.log
cat /var/log/linux-backup.log | mail -s "Centralized Backup Complete" topsoperations@topscms.com

You could modify this script to keep different daily backups , pruned to keep only X number of days of backups (i.e. only 7 days worth). There is alot you can do here.

If you have a handful of linux or bsd servers that you would like to backup in a centralized location, without having an individual script to maintain on each server, then perhaps you could use or modify this script to suit your needs.

I hope this helps.

DiggTwitterRedditDeliciousShare

*.hosting

Centralized remote backup script with SSH key authentication

Posted by: Admin  :  Category: Web Hosting

Greetings,

It has been a while since we posted any useful tidbits for you , so we have decided to share one of our quick & dirty centralized backup scripts.

The script relies on ssh key based authentication, described here on this blog. It essentially parses a configuration file where each variable is separated by a comma and colon, as in the example config here :

hostname1,192.168.1.1,etc:var:root
hostname2,192.168.1.2,etc:var:root:usr

Note the intended backup directories in the 3rd variable, separated by colon’s. Simply populate the backup-hosts.txt config file (located in the same folder as the script) with all the hosts you want to be backed up.

The script then ssh’s to the intended host, and sends a tar -czf stream (securely) over ssh, to be output into the destination of your choice. Ideally you should centralize this script on a box that has direct access to alot of disk space.

Find the script here :

#!/bin/sh
# Centralized Linux Backup Script
# By Star Dot Hosting , www.stardothosting.com
# Uses SSH Key based authentication and remote ssh commands to tar.gz folders to iSCSI storage

todaysdate=`date "+%Y-%m-%d %H:%M:%S"`
backupdest="/backups/linux-backups"

echo "Centralized Linux Backup: " $ todaysdate > /var/log/linux-backup.log
echo -e "———————————————-" >> /var/log/linux-backup.log
echo -e >> /var/log/linux-backup.log

for obj0 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | awk -F "," ‘{printf "%s\n", $ 2}’);
do
        backupname=`cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 1}’`

        for obj1 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 3’} | awk ‘{gsub(":","\n");printf"%s", $
0}’);
        do
                echo -e "backing up $ obj0 with $ obj1 directory" >> /var/log/linux-backup.log
                ssh -l root $ obj0 "(cd /$ obj1/ && tar -czf – . -C /$ obj1)" >> $ backupdest/$ backupname.$ obj1.tar.gz 2>&1
                if [ "$ ?" -eq 1 ]
                then
                        echo -e "There were some errors while backing up $ obj0 / $ backupname within the $ obj1 directory" >> /var/log/linux-backup.log
                        #exit 1
                else
                        echo -e "Backup completed on $ obj0 / $ backupname while backing up $ obj1 directory" >> /var/log/linux-backup.log
                fi
        done
done

echo "Backup Script Completed." >> /var/log/linux-backup.log
cat /var/log/linux-backup.log | mail -s "Centralized Backup Complete" topsoperations@topscms.com

You could modify this script to keep different daily backups , pruned to keep only X number of days of backups (i.e. only 7 days worth). There is alot you can do here.

If you have a handful of linux or bsd servers that you would like to backup in a centralized location, without having an individual script to maintain on each server, then perhaps you could use or modify this script to suit your needs.

I hope this helps.

DiggTwitterRedditDeliciousShare

*.hosting

Centralized remote backup script with SSH key authentication

Posted by: Admin  :  Category: Web Hosting

Greetings,

It has been a while since we posted any useful tidbits for you , so we have decided to share one of our quick & dirty centralized backup scripts.

The script relies on ssh key based authentication, described here on this blog. It essentially parses a configuration file where each variable is separated by a comma and colon, as in the example config here :

hostname1,192.168.1.1,etc:var:root
hostname2,192.168.1.2,etc:var:root:usr

Note the intended backup directories in the 3rd variable, separated by colon’s. Simply populate the backup-hosts.txt config file (located in the same folder as the script) with all the hosts you want to be backed up.

The script then ssh’s to the intended host, and sends a tar -czf stream (securely) over ssh, to be output into the destination of your choice. Ideally you should centralize this script on a box that has direct access to alot of disk space.

Find the script here :

#!/bin/sh
# Centralized Linux Backup Script
# By Star Dot Hosting , www.stardothosting.com
# Uses SSH Key based authentication and remote ssh commands to tar.gz folders to iSCSI storage

todaysdate=`date "+%Y-%m-%d %H:%M:%S"`
backupdest="/backups/linux-backups"

echo "Centralized Linux Backup: " $ todaysdate > /var/log/linux-backup.log
echo -e "———————————————-" >> /var/log/linux-backup.log
echo -e >> /var/log/linux-backup.log

for obj0 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | awk -F "," ‘{printf "%s\n", $ 2}’);
do
        backupname=`cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 1}’`

        for obj1 in $ (cat /usr/local/bin/backup-hosts.txt | grep -v "\#" | grep $ obj0 | awk -F "," ‘{printf "%s\n", $ 3’} | awk ‘{gsub(":","\n");printf"%s", $
0}’);
        do
                echo -e "backing up $ obj0 with $ obj1 directory" >> /var/log/linux-backup.log
                ssh -l root $ obj0 "(cd /$ obj1/ && tar -czf – . -C /$ obj1)" >> $ backupdest/$ backupname.$ obj1.tar.gz 2>&1
                if [ "$ ?" -eq 1 ]
                then
                        echo -e "There were some errors while backing up $ obj0 / $ backupname within the $ obj1 directory" >> /var/log/linux-backup.log
                        #exit 1
                else
                        echo -e "Backup completed on $ obj0 / $ backupname while backing up $ obj1 directory" >> /var/log/linux-backup.log
                fi
        done
done

echo "Backup Script Completed." >> /var/log/linux-backup.log
cat /var/log/linux-backup.log | mail -s "Centralized Backup Complete" topsoperations@topscms.com

You could modify this script to keep different daily backups , pruned to keep only X number of days of backups (i.e. only 7 days worth). There is alot you can do here.

If you have a handful of linux or bsd servers that you would like to backup in a centralized location, without having an individual script to maintain on each server, then perhaps you could use or modify this script to suit your needs.

I hope this helps.

Digg Twitter Reddit Delicious Share

*.hosting