A bash script to backup web application files and database to Dropbox | ineed.coffee

A bash script to backup web application files and database to Dropbox | ineed.coffee

The Problem

How much does it take you to write a blog post? What about 100 posts? What about writing the blog engine itself?
What if you suddenly loose everything? What if it was not your fault and your hosting provider does not backup your data? This is an extreme (but not unlikely) case.

Yesterday, I was greeted by an error page from all of my websites.

The problem was due to MySQL related issues. Fortunately, Webfaction is run by clever developers who solved the issue in a matter of minutes. They backup data, as well. However, hosting providers are not required to do backups of your data on a hourly basis, not even on a daily basis.

This made me think. Since the infamous black day where I (stupidly) lost 6 years of data, I occasionally do a manual backup of my web applications and store them on my computer and external storage. What about a shell script that automatically backups all of our web application’s files and databases, and optionally uploads everything to DropBox?

The Solution

Here is my solution:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

#!/bin/bash # # This program is free software. It comes without any warranty, to # the extent permitted by applicable law. You can redistribute it # and/or modify it under the terms of the Do What The Fuck You Want # To Public License, Version 2, as published by Sam Hocevar. See # http://sam.zoy.org/wtfpl/COPYING for more details. #   # AUTHOR:       Daniel Graziotin <dgraziotin AT task3 DOT cc> - http://ineed.coffee # URL:          http://ineed.coffee/1003 # PURPOSE:      Automatically create compressed archives of websites related #               data and MySQL (PostgreSQL) databases. Specifically designed to #               work with Webfaction's App layout. Should work out of the box on #               any other system if the webapps are all contained in a given #               directory. # INSTRUCTIONS: Create a text file (i.e., ~/.backup_passwd) containing one line #               for each entry, with the following syntax: # #               app_name dbms_type database_name database_user database_password # #               That is, a series of #               value<SPACE>value<SPACE>value<SPACE>value<SPACE>value<ENTER> #               Where dbms_type is M for MySQL and P for PostgreSQL. #               Only app_name is mandatory. All other variables are optional but #               must be entered alltogether if any.   #               Edit the following three variables and (optionally) the end #               of the script to enable dropbox upload. Enjoy.   WEBAPPS_DIR=~/webapps BACKUP_DIR=~/backup BACKUP_PSW_FILE=~/.backup_passwd   OLD_IFS="$IFS" IFS=$'\n'   for LINE in `cat $BACKUP_PSW_FILE`; do     IFS=' '     ARRAY=($LINE)     APP_NAME=${ARRAY[0]}     APP_DIR=$WEBAPPS_DIR/$APP_NAME     DB_TYPE=`echo ${ARRAY[1]} | tr '[:lower:]' '[:upper:]'`     DB_NAME=${ARRAY[2]}     DB_USER=${ARRAY[3]}     DB_PASS=${ARRAY[4]}     DB_SQL="${DB_NAME}.${DB_TYPE}.sql"       /bin/tar -cf $BACKUP_DIR/$APP_NAME.tar $APP_DIR     /bin/rm -rf $BACKUP_DIR/$APP_NAME.tar.bz2     /usr/bin/bzip2 $BACKUP_DIR/$APP_NAME.tar       if [[ ! -z "$DB_TYPE" ]] && [[ ! -z "$DB_USER" ]] && [[ ! -z "$DB_PASS" ]]; then         if [[ $DB_TYPE == "M" ]];then             /usr/bin/mysqldump -u$DB_USER -p$DB_PASS $DB_NAME > $BACKUP_DIR/$DB_SQL         elif [[ $DB_TYPE == "P" ]];then             PGPASSWORD=$DB_PASS pg_dump -U $DB_USER -f $BACKUP_DIR/$DB_SQL $DB_NAME         else             continue         fi         /bin/rm -rf $BACKUP_DIR/$DB_SQL.bz2         /usr/bin/bzip2 $BACKUP_DIR/$DB_SQL     fi done   IFS="$OLD_IFS"   # OPTIONAL - do something with the backups! # I like to upload them to my Dropbox, but it can be any other method # (ftp, scp, rsync, ..) # The following loop uses dropbox_uploader.sh by Andrea Fabrizi # See https://github.com/andreafabrizi/Dropbox-Uploader   #for BACKUP_FILE in $BACKUP_DIR/*; do #   FILE=`basename $BACKUP_FILE` #   ~/bin/dropbox_uploader.sh upload ${BACKUP_FILE} backup/${FILE} #done

Download

You can download the script here.

Configuration

The script assumes that all the websites and web applications live under the same directory, e.g., /home/dgraziotin/webapps.
It also assumes that each web application can have up to one MySQL (or PostgreSQL) database and a database user with full access on it. On Webfaction, each new database is accompanied by the creation of a user with the same name of the database. Only this user can access to the database. This is an elegant (and secure) solution, which should be employed on any other hosting platform and server.
We store all the information in the file /home/yourusername/.backup_passwd.
Below is an example of the password file:

1 2 3

myblog M dgraziotin_myblog dgraziotin_myblog saD23fWffFEfwg landing_page myecommercesite P dgraziotin_esite dgraziotin_esite DFWkf3kf24KGKf

The provided password file contains three entries.
The first one refers to the website myblog, using a MySQL database calleddgraziotin_myblog. The user dgraziotin_myblog accesses it with the passwordsaD23fWffFEfwg.
The second entry refers to a website called landing_page, not using any database at all.
The third entry is for a website called myecommercesite. It uses a PostgreSQL databas,dgraziotin_esite. The user dgraziotin_esite owns it with the password DFWkf3kf24KGKf.

Regarding Dropbox integration, install first Dropbox-Uploader and uncomment the last four lines of the script. Adjust the path of dropbox_uploader.sh to where you put the script.

How does it work?

The script exploits the Internal Field Separator of GNU/Linux to define the separator to identify patterns of text.

At line 36, we set the IPS to \n, or newline. Because of this, the for loop at line 38 will iterate every line of the password file.
Within the for loop, we set IPS to " ", or space character.
Each line is split up to three times, according to the number of space character encountered. These three substrings are stored in the variables APP_NAME, DB_NAME, andDB_USER.
These values are used in lines 49-51 to create a bzipped archive of the website (or the web application).
In lines 53-63 we create a bzipped archive of the database (MySQL or PostgreSQL), if the database has been defined in the password file.

I am using it through a cronjob to backup all of my websites. I hope that someone else finds it useful, too. Feel free to comment on this post to report possible bugs.

How to inspect iOS's HTTP traffic without spending a dime

How to inspect iOS's HTTP traffic without spending a dime

I recently had a problem. I was seeing intermittent issues with an iPhone app, Tapatalk, not working properly with a web forum hosted by a friend of mine. I knew there was a much better chance of getting the bug fixed if I could a) prove it was a bug and b) show the devs exactly where the problem was, but I was hampered by the usual problem: iOS apps are a bit of a black box, and I couldn't see what it was doing internally.

However, like almost all network-aware iOS apps, this one was clearly using a web service to get data from the backend. So, all I needed to do was figure out a way to see the traffic on the web service. This is the sort of thing I used to do all the time when my day job was writing load testing scripts for big ecommerce sites, but the first time I'd had to do it on the Mac or from an iOS client. I managed to get it working after doing a little research. If you find yourself in need of a similar solution -- perhaps for iOS app development, reporting a bug or some other reason (or just plain hacker interest!) -- then click through for step-by-step instructions on how to intercept and view your iOS web traffic from any Mac running on the same network.

The first thing you need is an HTTP sniffer program. The grandaddy of all network traffic sniffers is Wireshark, but it's rather low-level and overpowered for quickly looking through HTTP traces. It's a bit like using an electron microscope when what you wanted was a magnifying glass. I came across several glowing references to Tuffcode's MacScoop HTTP Scoop during my research, but didn't really want to spend $15 on an app I was only going to use once. I settled on PortSwigger's Burp Suite, a comprehensive HTTP security analysis tool. The free version has the HTTP Proxy feature, which is the only bit we need; grab that. (Windows users: you can use the excellent and free Fiddler Web Debugger, but I won't be walking you through that today, sorry. The steps are very similar though.)

Burp Suite is a Java program, so when it downloads, you'll see a directory with a JAR file in it. If you double click that, it should start up after a warning about an untested JVM version. Be thankful we still get JVMs with our OS X for now or this would be more complex.

Burp is written by and for security experts, so the UI is a bit ... Spartan, ... but it's easy to configure it for the simple feature we want to do. First, click the Proxy tab at the top, then click the "intercept is on" button to make it say "intercept is off," like so:

Next, select the Options tab. It'll show you a single "proxy listener" running in a list. Click on it, then click the Edit button. Untick the "Listen on loopback interfaces only" checkbox, then click "Update." You'll get a warning you can ignore, and when you're done, the app window should look like this:

When you're done, click on the History tab. If you're one of the many Mac users who find Java UIs give you hives, you can rejoice, because you're done with that for now!

Next, open Apple > System Preferences > Network. Select your current active network connection and make a note of the IP address your Mac is using. For me, this is 192.168.2.32.

Now, turn to your iOS device -- I'll be using my iPhone, but this works the same way on an iPad or an iPod touch. Go into Settings > Wi-Fi > your Wi-Fi network, and then click the blue "more details" arrow. Scroll down and at the bottom there's a set of three buttons under "HTTP Proxy." Select "Manual" and fill in your Mac's IP address (found in the last step) and the Burp Suite port number (8080 unless you changed it earlier), like so:

And that's it. If you flick back to the Burp Suite window on the Mac now and start browsing the Web or using apps on your iPhone, you'll see all the traffic show up in a neat little list. You can click on each individual request to see the full text sent and received as part of the request. Note that you might see some security warnings on the iPhone for any app that uses an encrypted link; this is because the app thinks that Burp Suite might be a man-in-the-middle attack. It's not, though, and it's safe to ignore that warning.

And there you have it -- with one free download and a few minutes of configuration work, you can snoop all of iOS's web traffic for fun and/or profit.

Update: corrected name "MacScoop" to "HTTP Scoop". Thanks for the correction, Jonathan.

Update 2: please check the comments for lots of suggestions for alternate tools to do this job. Thank you all for your insightful additions!

How to find outdated joomla versions on your server to reduce the risk of being hacked

How to find outdated joomla versions on your server to reduce the risk of being hacked

Today I want to focus on a topic that can lead to huge problems of hacked accounts, spam mailings etc.: Outdated Joomla installations on your server.

Of course, this is valid for other software, too. The mentioned method should work in a similar way for other software. Since Joomla is widely spread throughout the internet, especially on shared hosting systems CMS (Content Management System), this howto will only cover Joomla so far.

Prerequisites

You will need at least the tools bc and sed that are not always installed by default.

Let us begin

At first there is one question: how can I recognize the joomla version from the installed files?

This depends on the installed version of Joomla. Until now I have found three different files that contain the version information:
/libraries/joomla/version.php
/libraries/cms/version/version.php
/includes/version.php

The most important lines of these files are the version number and the minor version that are contained in the following variables:
var $RELEASE = '1.0';
var $DEV_LEVEL = '12';

In a next step we search for the most recent version of Joomla on the official website. At the time of writing this howto there are three version groups:1.5 (1.5.26), 2.5 (2.5.17) and 3.2 (3.2.1).

Find joomla installations on your server

One thing that all joomla installations have in common is the folder name "components", so we search for all folders with this name. At the same time we ignore all those components folders that are contained in a subfolder "administrator". The base path /var/www has to be adjusted if the websites on your server do not lie in there.

find /var/www/ -type d -name 'components' ! -wholename '**/administrator/components'

This command will give you a list of all those folders. The command dirname is suitable for getting the path that contains the components folder. This is the base path of the Joomla installation.

We do this in a loop for all found components folders:

for L in `find /var/www/ -type d -name 'components' ! -wholename '**/administrator/components'` ; do
    D=`dirname $L` ;
done

Get the major and minor version

To get the version of your installation we will use the combined commands "grep" and "sed".

First we check which of the three files I mentioned earlier exists in the installation path.

F=$D/libraries/joomla/version.php ;
F2=$D/libraries/cms/version/version.php ;
F3=$D/includes/version.php ;
if [[ -e "$F" || -e "$F2" || -e "$F3" ]] ; then
    if [[ -e "$F" ]] ; then
        F=$F ;
    elif [[ -e "$F2" ]] ; then
        F=$F2 ;
    elif [[ -e "$F3" ]] ; then
        F=$F3 ;
    fi
else
    echo "No joomla version file found." ;
fi

Now we read the major and minor version from this file:

VERSION=`grep '$RELEASE' $F | sed -r "s/^.*=\s*'(.*)'.*$/\1/g"` ;
SUBVERSION=`grep '$DEV_LEVEL' $F | sed -r "s/^.*=\s*'(.*)'.*$/\1/g"` ;

Compare versions

Since version numbers are no integers, we cannot compare them inside the bash script using -lt etc. Instead we need to use an external programm called bc:

ISOK=1 ;
if [[ $(echo "if (${VERSION} < 1.5) 1 else 0" | bc) -eq 1 ]] ; then
    # version is lower than 1.5
    ISOK=0 ;
elif [[ $(echo "if (${VERSION} == 1.5) 1 else 0" | bc) -eq 1 && $(echo "if (${SUBVERSION} < 26) 1 else 0" | bc) -eq 1 ]] ; then
    # version is 1.5.x but lower than 1.5.26
    ISOK=0 ;
### and so on - further version checks
else
    ISOK=1 ;
fi

The complete script

Now we are ready to assemble all the parts into a ready-to-use script and add some minor improvements. The script will search for all joomla versions in the given base path and prints information about the status. Of course you cannot speak of a version 1.5.26 as "current" but as it is the most current version of the 1.5 branch and an update to the 2.5 or 3.x branch is very complicated in some cases, this script will mark this version as "OK". You can change this if you like.

Here is a sample output of the script:

[INFO] version 1.5.26 in /var/www/xxx is ok.
[WARN] outdated Joomla version 1.0.12 in /var/www/yyy
[WARN] outdated Joomla version 1.5.14 in /var/www/zzz
[WARN] outdated Joomla version 2.5.8 in /var/www/aaa
[WARN] outdated Joomla version 1.5.10 in /var/www/bbb

And now: the complete script. Just save it as "joomlascan.sh" and call it via

bash joomlascan.sh

If you give a file name as an additional argument (like bash joomlascan.sh list.csv), you get a file called list.csv that contains a list of all outdated installations:

/var/www/yyy;1.0.12;1.5.26
/var/www/zzz;1.5.14;1.5.26
/var/www/aaa;2.5.8;2.5.17
/var/www/bbb;1.5.10;1.5.26

Tip:
If you use ISPConfig 3, you should alter the BASEPATH to BASEPATH="/var/www/clients/client*/web*".

#!/bin/bash

# current version 1.5.x
CUR15=26
# aktuelle version 2.5.x
CUR25=17
# aktuelle version 3.2.x
CUR3=1

#base path of the websites
BASEPATH="/var/www/"

# write to csv file (optional argument)
OUTFILE=$1

if [[ "$OUTFILE" != "" ]] ; then
    # empty CSV file
    echo -n "" > $OUTFILE ;
fi

for L in `find ${BASEPATH} -type d -name 'components' ! -wholename '**/administrator/components' | grep -v '/tmp/'` ; do
    D=`dirname $L` ;
    F=$D/libraries/joomla/version.php ;
    F2=$D/libraries/cms/version/version.php ;
    F3=$D/includes/version.php ;
    ISOK=0 ;
    SHOWNEWEST="" ;
    IMPORTANCE=0 ;
    if [[ -e "$F" || -e "$F2" || -e "$F3" ]] ; then
        if [[ -e "$F" ]] ; then
            F=$F ;
        elif [[ -e "$F2" ]] ; then
            F=$F2 ;
        elif [[ -e "$F3" ]] ; then
            F=$F3 ;
        fi
        VERSION=`grep '$RELEASE' $F | sed -r "s/^.*=\s*'(.*)'.*$/\1/g"` ;
        SUBVERSION=`grep '$DEV_LEVEL' $F | sed -r "s/^.*=\s*'(.*)'.*$/\1/g"` ;
        if [[ $(echo "if (${VERSION} < 1.5) 1 else 0" | bc) -eq 1 ]] ; then
            # version is lower than 1.5
            SHOWNEWEST="1.5.${CUR15}" ;
            IMPORTANCE=3 ;
        elif [[ $(echo "if (${VERSION} == 1.5) 1 else 0" | bc) -eq 1 && $(echo "if (${SUBVERSION} < ${CUR15}) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 1.5.x but not most current version
            SHOWNEWEST="1.5.${CUR15}" ;
            IMPORTANCE=2 ;
        elif [[ $(echo "if (${VERSION} == 1.6) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 1.6
            SHOWNEWEST="2.5.${CUR25}" ;
            IMPORTANCE=2 ;
        elif [[ $(echo "if (${VERSION} == 1.7) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 1.7
            SHOWNEWEST="2.5.${CUR25}" ;
            IMPORTANCE=2 ;
        elif [[ $(echo "if (${VERSION} > 1.7) 1 else 0" | bc) -eq 1 && $(echo "if (${VERSION} < 2.5) 1 else 0" | bc) -eq 1 ]] ; then
            # version is somewhere between 1.7 and 2.5
            SHOWNEWEST="2.5.${CUR25}" ;
            IMPORTANCE=2 ;
        elif [[ $(echo "if (${VERSION} == 2.5) 1 else 0" | bc) -eq 1 && $(echo "if (${SUBVERSION} < ${CUR25}) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 2.5 but lower than current
            SHOWNEWEST="2.5.${CUR25}" ;
            IMPORTANCE=1 ;
        elif [[ $(echo "if (${VERSION} >= 3) 1 else 0" | bc) -eq 1 && $(echo "if (${VERSION} < 3.2) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 3.0 or 3.1
            SHOWNEWEST="3.2.${CUR3}" ;
            IMPORTANCE=2 ;
        elif [[ $(echo "if (${VERSION} == 3.2) 1 else 0" | bc) -eq 1 && $(echo "if (${SUBVERSION} < ${CUR3}) 1 else 0" | bc) -eq 1 ]] ; then
            # version is 3.2 but lower than current
            SHOWNEWEST="3.2.${CUR3}" ;
            IMPORTANCE=1 ;
        else
            ISOK=1 ;
            echo "[INFO] version $VERSION.$SUBVERSION in $D is ok." ;
        fi
    else
        # seems not to bee a joomla version ...
        ISOK=1 ;
    fi ;
    
    if [[ $ISOK -eq 0 ]] ; then
        echo "[WARN] outdated Joomla version $VERSION.$SUBVERSION in $D" ;
        if [[ "$OUTFILE" != "" ]] ; then
            # write CSV file
            echo "\"$D\";$VERSION.$SUBVERSION;$SHOWNEWEST;$IMPORTANCE" >> $OUTFILE ;
        fi
    fi
done

exit 0 ;

There is one known issue whith this script: it cannot recognize Mambo. So it will mark all mambo installations as "OK" regardless which version they have.