My Collections: June 2015

A bash script to backup web application files and database to Dropbox | ineed.coffee

The Problem

How much does it take you to write a blog post? What about 100 posts? What about writing the blog engine itself?
What if you suddenly loose everything? What if it was not your fault and your hosting provider does not backup your data? This is an extreme (but not unlikely) case.

Yesterday, I was greeted by an error page from all of my websites.

The problem was due to MySQL related issues. Fortunately, Webfaction is run by clever developers who solved the issue in a matter of minutes. They backup data, as well. However, hosting providers are not required to do backups of your data on a hourly basis, not even on a daily basis.

This made me think. Since the infamous black day where I (stupidly) lost 6 years of data, I occasionally do a manual backup of my web applications and store them on my computer and external storage. What about a shell script that automatically backups all of our web application’s files and databases, and optionally uploads everything to DropBox?

The Solution

Here is my solution:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
#!/bin/bash
#
# This program is free software. It comes without any warranty, to
# the extent permitted by applicable law. You can redistribute it
# and/or modify it under the terms of the Do What The Fuck You Want
# To Public License, Version 2, as published by Sam Hocevar. See
# http://sam.zoy.org/wtfpl/COPYING for more details.
#
 
# AUTHOR:       Daniel Graziotin <dgraziotin AT task3 DOT cc> - http://ineed.coffee
# URL:          http://ineed.coffee/1003
# PURPOSE:      Automatically create compressed archives of websites related
#               data and MySQL (PostgreSQL) databases. Specifically designed to
#               work with Webfaction's App layout. Should work out of the box on 
#               any other system if the webapps are all contained in a given
#               directory.
# INSTRUCTIONS: Create a text file (i.e., ~/.backup_passwd) containing one line 
#               for each entry, with the following syntax:
#
#               app_name dbms_type database_name database_user database_password
#
#               That is, a series of
#               value<SPACE>value<SPACE>value<SPACE>value<SPACE>value<ENTER>
#               Where dbms_type is M for MySQL and P for PostgreSQL.
#               Only app_name is mandatory. All other variables are optional but
#               must be entered alltogether if any.
 
#               Edit the following three variables and (optionally) the end 
#               of the script to enable dropbox upload. Enjoy.
 
WEBAPPS_DIR=~/webapps
BACKUP_DIR=~/backup
BACKUP_PSW_FILE=~/.backup_passwd
 
OLD_IFS="$IFS"
IFS=$'\n'
 
for LINE in `cat $BACKUP_PSW_FILE`; do
    IFS=' '
    ARRAY=($LINE)
    APP_NAME=${ARRAY[0]}
    APP_DIR=$WEBAPPS_DIR/$APP_NAME
    DB_TYPE=`echo ${ARRAY[1]} | tr '[:lower:]' '[:upper:]'`
    DB_NAME=${ARRAY[2]}
    DB_USER=${ARRAY[3]}
    DB_PASS=${ARRAY[4]}
    DB_SQL="${DB_NAME}.${DB_TYPE}.sql"
 
    /bin/tar -cf $BACKUP_DIR/$APP_NAME.tar $APP_DIR
    /bin/rm -rf $BACKUP_DIR/$APP_NAME.tar.bz2
    /usr/bin/bzip2 $BACKUP_DIR/$APP_NAME.tar
 
    if [[ ! -z "$DB_TYPE" ]] && [[ ! -z "$DB_USER" ]] && [[ ! -z "$DB_PASS" ]]; then
        if [[ $DB_TYPE == "M" ]];then
            /usr/bin/mysqldump -u$DB_USER -p$DB_PASS $DB_NAME > $BACKUP_DIR/$DB_SQL
        elif [[ $DB_TYPE == "P" ]];then
            PGPASSWORD=$DB_PASS pg_dump -U $DB_USER -f $BACKUP_DIR/$DB_SQL $DB_NAME
        else
            continue
        fi
        /bin/rm -rf $BACKUP_DIR/$DB_SQL.bz2
        /usr/bin/bzip2 $BACKUP_DIR/$DB_SQL
    fi
done
 
IFS="$OLD_IFS"
 
# OPTIONAL - do something with the backups!
# I like to upload them to my Dropbox, but it can be any other method
# (ftp, scp, rsync, ..)
# The following loop uses dropbox_uploader.sh by Andrea Fabrizi
# See https://github.com/andreafabrizi/Dropbox-Uploader
 
#for BACKUP_FILE in $BACKUP_DIR/*; do 
#   FILE=`basename $BACKUP_FILE`
#   ~/bin/dropbox_uploader.sh upload ${BACKUP_FILE} backup/${FILE}
#done

Download

You can download the script here.

Configuration

The script assumes that all the websites and web applications live under the same directory, e.g., /home/dgraziotin/webapps.
It also assumes that each web application can have up to one MySQL (or PostgreSQL) database and a database user with full access on it. On Webfaction, each new database is accompanied by the creation of a user with the same name of the database. Only this user can access to the database. This is an elegant (and secure) solution, which should be employed on any other hosting platform and server.
We store all the information in the file /home/yourusername/.backup_passwd.
Below is an example of the password file:

1
2
3
myblog M dgraziotin_myblog dgraziotin_myblog saD23fWffFEfwg
landing_page
myecommercesite P dgraziotin_esite dgraziotin_esite DFWkf3kf24KGKf

The provided password file contains three entries.
The first one refers to the website myblog, using a MySQL database calleddgraziotin_myblog. The user dgraziotin_myblog accesses it with the passwordsaD23fWffFEfwg.
The second entry refers to a website called landing_page, not using any database at all.
The third entry is for a website called myecommercesite. It uses a PostgreSQL databas,dgraziotin_esite. The user dgraziotin_esite owns it with the password DFWkf3kf24KGKf.

Regarding Dropbox integration, install first Dropbox-Uploader and uncomment the last four lines of the script. Adjust the path of dropbox_uploader.sh to where you put the script.

How does it work?

The script exploits the Internal Field Separator of GNU/Linux to define the separator to identify patterns of text.

At line 36, we set the IPS to \n, or newline. Because of this, the for loop at line 38 will iterate every line of the password file.
Within the for loop, we set IPS to " ", or space character.
Each line is split up to three times, according to the number of space character encountered. These three substrings are stored in the variables APP_NAME, DB_NAME, andDB_USER.
These values are used in lines 49-51 to create a bzipped archive of the website (or the web application).
In lines 53-63 we create a bzipped archive of the database (MySQL or PostgreSQL), if the database has been defined in the password file.

I am using it through a cronjob to backup all of my websites. I hope that someone else finds it useful, too. Feel free to comment on this post to report possible bugs.