GitLab update script

I've recently installed GitLab and they provide easy to install deb and rpm packages but not a repository to help us keep our installation up to date. So I developed the following script that will check https://about.gitlab.com/downloads/archives/ for newer versions and install them when available:

#!/bin/bash
OS="ubuntu"
OS_Version="14.04"
OS_ARCHITECTURE="amd64"
# Ubuntu/Debian:
INSTALLED_VERSION=$(dpkg -s gitlab | grep -i version | cut -d" " -f2)
# CentOS:
#INSTALLED_VERSION=$(rpm -qa | grep omnibus)
# Uses sort -V to compare versions
LATEST=$(wget -q -O- https://about.gitlab.com/downloads/archives/ | grep -i "$OS" | grep -i "$OS_VERSION" | grep -i $OS_ARCHITECTURE | grep -Eo 'href=".*"' | cut -d'"' -f2 | sort -V | tail -n 1)
PACKAGE=${LATEST##*/}
LATEST_VERSION=$(echo $PACKAGE | cut -d_ -f2)
echo ""
echo " Current version: $INSTALLED_VERSION"
echo " Latest version: $LATEST_VERSION"
if [[ "$INSTALLED_VERSION" != "$LATEST_VERSION" && "$LATEST_VERSION" != "" ]]; then
    echo "    Update to $LATEST_VERSION available!"
    echo -n "     Do you wich to upgrade? [y/N]? "
    read answer
    case $answer in
        y*)
            # Backup branding:
            cp /opt/gitlab/embedded/service/gitlab-rails/public/assets/*logo*.png /tmp/
            wget $LATEST
            # Stop unicorn and sidekiq so we can do database migrations
            sudo gitlab-ctl stop unicorn
            sudo gitlab-ctl stop sidekiq
            # Create a database backup in case the upgrade fails
            sudo gitlab-rake gitlab:backup:create
            # Install the latest package
            # Ubuntu/Debian:
            sudo dpkg -i $PACKAGE
            # CentOS:
            #sudo rpm -Uvh $PACKAGE
            # Restore branding:
            sudo cp /tmp/*logo*.png /opt/gitlab/embedded/service/gitlab-rails/public/assets/
            # Reconfigure GitLab (includes database migrations)
            sudo gitlab-ctl reconfigure
            # Restart all gitlab services
            sudo gitlab-ctl restart
            rm $PACKAGE
        ;;
        *)
            echo "No change"
        ;;
    esac
else
    echo "    Nothing to do!"
fi
echo ""

I haven't tested this script on a CentOS machine so it might need some adjustments to work there.

Removing hosts from backuppc

Simply remove the host from the web interface and rm -rf the pc/<host> directory, then wait for the next BackupPC_nightly run - it will remove all superfluous files from the pools.

The path for this directory usually is:

/var/lib/Backuppc/pc/<hostname>

If you want to force the clean-up process, you can remove your host like this:

1. Login to the Backuppc server
2. Remove the host in the Backuppc web-interface (under hosts)
3. remove it's directory /var/lib/Backuppc/pc/<hostname>:

rm -rf /var/lib/Backuppc/pc/<hostname>

4. Shutdown backuppc:

service backuppc stop

5. Change into backuppc:

su - backuppc

6. Run the nightly script:

/usr/share/BackupPC/bin/BackupPC_nightly 0 255

7. Go back to root:

exit

8. Start backuppc again:

service backuppc start

Reclaim free space from Time Machine sparsebundle

You msut run these commands as root:

sudo su - 

Make sure the mount point exists:

mkdir -p /Volumes/TM

Then mount the afp share:

mount_afp 'afp://user:password@afp_server_address/share_name' /Volumes/TM

Now use hdiutil to reclaim the available free space:

hdiutil compact /Volumes/TM/ComputerName.sparsebundle/

unmont the share:

umount /Volumes/TM/

If you get an error message saying:

hdiutil: compact failed - Resource temporarily unavailable

You must make sure you don't have the afp share mounted elsewhere, you can check your mounts with:

df -h

If the output contains a line with your afp server's address or with the string "Time Machine" you have to unmount them.

The following script will do all that for you:

SRV="afp_server_address"
SAVEIFS=$IFS
IFS=$'\n';
for v in $(df -h | grep -E "$SRV|Time\sMachine"  | cut -d"%" -f3 |  sed -e "s/ *\//\//"); do
    umount "$v"
done
IFS=$SAVEIFS
mkdir -p /Volumes/TM
mount_afp "afp://user:password@$SRV/share" /Volumes/TM
hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
umount /Volumes/TM/

Changing Time Machine Backup Interval

You can use the following command to change the Time Machine backup interval:

sudo defaults write /System/Library/LaunchDaemons/com.apple.backupd-auto StartInterval -int 14400

The time interval is in seconds, so 43200 will start a backup every 12hrs.

Checkout my previous post to learn how to manually delete Time Machine backups.

Manage time machine backups

Some times you get some errors saying that your time machine's disk is full and Time Machine could not complete the backup so you need to  manually delete old backups.

tmutil provides methods of controlling and interacting with Time Machine, as well as examining and manipulating Time Machine backups. Common abilities include restoring data from backups, editing exclusions, and comparing backups.

tmutil latestbackup

Will output the path to the most recent backup and

tmutil listbackups

will list all existing backups, if you use the same backup disk for multiple machines, you can get just the backups from your machine with:

tmutil listbackups | grep "$(scutil --get ComputerName)"

The following command will delete the backups from a mac named old_mac_name:

sudo tmutil delete /Volumes/drive_name/Backups.backupdb/old_mac_name

If you want to be safe, you can pick one snapshot to delete first to be sure the command works as intended. This is nice since it could take hours to clean up some larger backup sets and you want to leave the Mac confident it's deleting the correct information store.

You can use the tmutil tool to delete backups one by one.

sudo tmutil delete /Volumes/drive_name/Backups.backupdb/mac_name/YYYY-MM-DD-hhmmss

Since tmutil was introduced with Lion, this will not work on earlier OS versions.

The tmutil delete command only removes the backup from the sparse bundle. It doesn’t actually free the disk space. To do that, you have to go a little deeper.

On your Mac is a mount point called /Volumes. You can examine the contents of this mount point with ls:

cd /Volumes
ls -1

Should output something like:

Macintosh HD
Recovery HD
Time Machine Backups
TimeMachine

These are the names of all the mounted disks (or things that look like disks) on your Mac. Notice two likely candidates for your actual TimeMachine volume. Yours may be named slightly differently, but the one you want is the one that actually shows files of type .sparsebundle . In my case, it is the volume TimeMachine:

sudo ls -l TimeMachine/

and you should see something similar to:

...
drwxr-x---@ 1 root wheel 264 Jul 25 08:21 sysadmin’s MacbookPro.sparsebundle
...

Notice that you don’t actually own the file. (Had I not used the sudo command with ls I could not have listed the contents of /Volumes/TimeMachine)

That .sparsebundle file for your Mac is where all your backup sets live. TimeMachine manages the contents of this file, but doesn’t do anything automatically to reduce its size. Luckily there is another tool for that, but you’ll have to be root to run it:

sudo su -
hdiutil compact /Volumes/TimeMachine/YourBackup.sparsebundle

Sample output:

Starting to compact…
Reclaiming free space…
...................................................
Finishing compaction…
Reclaimed 3.1 GB out of 304.1 GB possible.

That’s it! In this example I reclaimed 3.1GB of actual disk space on my TimeMachine volume. 

The following bash script will remove the oldest backup and reclaim the free space:

COMPUTER_NAME=$(/usr/sbin/scutil --get ComputerName)
NBACKUPS=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/wc -l)
OLDEST_BACKUP=$(/usr/bin/tmutil listbackups | /usr/bin/grep "$COMPUTER_NAME" | /usr/bin/head -n1)
LATEST_BACKUP=$(/usr/bin/tmutil latestbackup)
echo Latest backup: $LATEST_BACKUP
if [[ -n "$LATEST_BACKUP" && "$LATEST_BACKUP" != "$OLDEST_BACKUP" ]]
then
    echo "$NBACKUPS backups. Delete oldest: ${OLDEST_BACKUP##*/} [y/N]? \c"
    read answer
    case $answer in
   y*)
  echo Running: /usr/bin/sudo /usr/bin/tmutil delete "$OLDEST_BACKUP"
  /usr/bin/sudo time /usr/bin/tmutil delete "$OLDEST_BACKUP"
    echo "Do you wish to reclaim the free space now? [y/N]? \c"
    read answer
    case $answer in
    y*)
     mkdir -p /Volumes/TM
     mount_afp 'afp://user:pass@afp_server_address/share_name' /Volumes/TM
     hdiutil compact "/Volumes/TM/$(scutil --get ComputerName).sparsebundle"
     umount /Volumes/TM/
    ;;
    *)
 echo No change
    ;;
        esac
   ;;
   *)
  echo No change
   ;;
    esac
else
    echo "No backup available for deletion"
fi

In the script above, don't forget to change the afp URL (afp://user:pass@afp_server_address/share_name) to your own.

Sorry, Command-not-found Has Crashed

When you try to execute a command that is not installed Ubuntu tries to hint you on the package that you should install but some times, especially after an upgrade, you get an error message saying:

Sorry, command-not-found has crashed! Please file a bug report at:
(...)

This solves the problem:

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

Ubuntu as Time Machine server

This guide will help you to install and configure the netatalk servise on an Ubuntu server so it can function as a Time Machine backup server for your Mac OS machines.

First install the necessary packages:

sudo apt-get install netatalk avahi-daemon libnss-mdns

Open the netatalk default configuration file:

sudo vi /etc/default/netatalk

Modify the lines:

ATALKD_RUN=yes
PAPD_RUN=no
CNID_METAD_RUN=yes
AFPD_RUN=yes
TIMELORD_RUN=no
A2BOOT_RUN=no

Edit the atalkd.conf file:

sudo vi /etc/netatalk/atalkd.conf

add to the bottom

eth0

Edit the AppleVolumes.default file:

sudo vi /etc/netatalk/AppleVolumes.default

add to the bottom:

/backups/timemachine "Time Machine" allow:@admin cnidscheme:cdb volsizelimit:200000 options:usedots,upriv,tm

The example above also limits the size shown to OS X as 200 GB (the number is given in MiB, so it's 200,000 times 1024 in the real world)

Edit the afpd configuration file:

sudo vi  /etc/netatalk/afpd.conf

add to the bottom:

- -transall -uamlist uams_dhx.so,uams_dhx2.so -nosavepassword -advertise_ssh -mimicmodel TimeCapsule6,106 -setuplog "default log_warn  /var/log/afpd.log"

Create a configuration file for the avahi afpd discovery:

sudo vi  /etc/avahi/services/afpd.service

and enter the following into it:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
 <name replace-wildcards="yes">%h</name>
 <service>
 <type>_afpovertcp._tcp</type>
 <port>548</port>
 </service>
 <service>
 <type>_device-info._tcp</type>
 <port>0</port>
 <txt-record>model=MacPro</txt-record>
 </service>
</service-group> 

Restart the services:

sudo service netatalk restart
sudo service avahi-daemon restart