Calculating total disk usage by files with specific extension

For example if you want to check how much space is being used by log files on your entire system, you can use the following:

find / -type f -name "*.log*" -exec du -b {} \; | awk '{ sum += $1 } END { kb = sum / 1024; mb = kb / 1024; gb = mb / 1024; printf "%.0f MB (%.2fGB) disk space used\n", mb, gb}'

Just replace "*.log*" with the file extension you want to search for and the above will give you the disk used by the sum of all the files with that extension.

PostgreSQL cluster using DRBD and hot standby

Cluster Configuration:

First install all the necessary packages:

yum install gfs2-utils cman fence-virtd-checkpoint lvm2-cluster perl-Net-Telnet rgmanager device-mapper-multipath ipvsadm piranha luci modcluster cluster-snmp ricci

yum groupinstall "High Availability"
yum install postgresql-server

chkconfig --level 123456 ricci on
chkconfig --level 123456 luci on
chkconfig --level 123456 cman on
chkconfig --level 123456 iptables off
chkconfig --level 123456 ip6tables off
chkconfig postgresql on
chkconfig cman on
chkconfig rgmanager on

Now edit the cluster configuration file:

vi vi /etc/cluster/cluster.conf

Make it look like this:

<?xml version="1.0"?>
<cluster config_version="7" name="pgcluster">
<clusternodes>
<clusternode name="10.39.30.7" votes="1" nodeid="1">
<fence/>
</clusternode>
<clusternode name="10.39.30.8" votes="1" nodeid="2">
<fence/>
</clusternode>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="PGSQL" nofailback="0" ordered="0" restricted="0">
<failoverdomainnode name="10.39.30.7"/>
<failoverdomainnode name="10.39.30.8"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.39.30.6" monitor_link="on" sleeptime="10"/>
<postgres-8 config_file="/var/lib/pgsql/data/postgresql.conf" name="pgsql" shutdown_wait="5" />
</resources>
<service autostart="1" exclusive="0" domain="PGSQL" name="pgsql" recovery="relocate">
<drbd name="drdb-postgres" resource="r0">
<fs device="/dev/drbd0" fsid="6202" fstype="ext3" mountpoint="/var/lib/pgsql" name="pgsql" options="noatime"/>
</drbd>
<ip ref="10.39.30.6"/>
<postgres-8 ref="pgsql"/>
</service>
</rm>
<cman expected_votes="1" two_node="1"/>
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>
</cluster>

DRDB Configuration:

Install the necessary files:

yum install gcc flex make libxslt rpm-build redhat-rpm-config kernel-devel

You need to download and install DRBD manually

wget http://oss.linbit.com/drbd/8.4/drbd-8.4.1.tar.gz 

 the following commands will generate DRBD RPM packages:

tar -xvf *.tar.gz
mkdir -p /root/rpmbuild/SOURCES/
cp drbd*.tar.gz /root/rpmbuild/SOURCES/
cd drbd-8.4.1
./configure --with-rgmanager --enable-spec --with-km
make tgz
rpmbuild --bb drbd.spec --without xen --without heartbeat --without udev --without pacemaker --with rgmanager
rpmbuild --bb drbd-kernel.spec
rpmbuild --bb drbd-km.spec

Now install the newly created packages:

cd /root/rpmbuild/RPMS/x86_64
rpm -i drbd-utils-8.4.1-1.el6.x86_64.rpm drbd-bash-completion-8.4.1-1.el6.x86_64.rpm drbd-8.4.1-1.el6.x86_64.rpm drbd-rgmanager-8.4.1-1.el6.x86_64.rpm drbd-km-2.6.32_279.14.1.el6.x86_64-8.4.1-1.el6.x86_64.rpm

Add your nodes IP addresses to the hosts file on both machines:

vi /etc/hosts

10.39.30.7 RHPG1
10.39.30.8 RHPG2

Create a DRBD configuration file:

vi /etc/drbd.d/r0.res

resource r0 {
   device /dev/drbd0;
   meta-disk internal;
   on RHPG1 {
      address 10.39.30.7:7789;
      disk /dev/sdb1;
   }
   on RHPG2 {
      address 10.39.30.8:7789;
      disk /dev/sdb1;
   }
}

Create the partion /dev/sdb1 but do not format it:

fdisk /dev/sdb

Run on both machines:

drbdadm create-md r0
modprobe drbd
drbdadm up r0

Run on one of the machines to create the file system:

drbdadm -- --overwrite-data-of-peer primary r0

Check the sync status on any of the hosts with:

service drbd status

Create the file system on /dev/drbd0

mkfs.ext3 /dev/drbd0

and move over the PostgreSQL data

mkdir /tmp/pgdata
mount /dev/drbd0 /tmp/pgdata
cp -r /var/lib/pgsql /tmp/pgdata

Wait until the data is synced over the two hosts check the status with:

service drbd status

Unmount the drbd device:

umount /dev/drbd0

and then, on both hosts do:

rm -rf /var/lib/pgsql/*

Restart the drbd service

service drbd restart

the status from:

service drbd status

should show that both hosts are in secondary:

0:r0 Connected Secondary/Secondary UpToDate/UpToDate C

And ready to be managed by rgmanager.

Encrypt and decrypt files using openssl

Here's a safe way to pass sensitive files over email...

To encrypt use:

openssl enc -e -bf-cbc -in <FILE.zip> -out <FILE.ENC>

This will ask you for a password that you'll need to decrypt the file.

And to decrypt:

openssl enc -d -bf-cbc -in <FILE.ENC> -out <FILE.zip>

File rotator script

This is a script that I use to rotate some logs, the commented lines will tell you what it does exactly:

#!/bin/sh
#-----------------------------------------------------------------------
# FILE ROTATOR SCRIPT
#
# The purpose of this script is to rotate, compress and delete files
# - Files older than ARC_AGE are gzipped and rotated
# - Files bigger than SIZE_LIM are gzipped and rotated
# - Gzipped files older than DEL_AGE are deleted
#
#-----------------------------------------------------------------------
# Vars
DATE=`date +%F"-"%H:%M`
FILEDIR="/storage/logs/"
DEL_AGE="30"
ARC_AGE="1"
SIZE_LIM="20M"
# Diagnostics
echo "-= Rotation starting =-"
echo "    Directory to search: $FILEDIR"
echo "    File age to check for delition: $DEL_AGE"
echo "    File age to check for archive: $ARC_AGE"
echo "    File size to check for archive: $SIZE_LIM"
echo " "
# Compress all unconpressed files which last modification occured more than ARC_AGE days ago
echo "-= Looking for old files =-"
FILES=`find $FILEDIR -type f -mtime +$ARC_AGE -not \( -name '*.gz' \) -print`
echo "Files to be archived:"
echo $FILES
echo " "
for FILE in $FILES; do
    # Compress but keep the original file
    gzip -9 -c "$FILE" > "$FILE".$DATE.gz;
    # Check if file is beeing used:
    lsof $FILE
    ACTIVE=$?
    # Delete inactive files, truncate if active
    if [ $ACTIVE != 0 ]; then
        # Delete the file
        rm "$FILE";
    else
        # Truncate file to 0
        :>"$FILE";
    fi
done
# Compress all unconpressed files that are bigger than SIZE_LIM
echo "-= Looking for big files =-"
FILES=`find $FILEDIR -type f -size +$SIZE_LIM -not \( -name '*.gz' \) -print`
echo "Files to be archived:"
echo $FILES
echo " "
for FILE in $FILES; do
    # Compress but keep the original file
    gzip -9 -c "$FILE" > "$FILE".$DATE.gz;
    # Truncate original file to 0
    :>"$FILE";
done
echo "-= Deleting old archived files =-"
FILES_OLD=`find $FILEDIR -type f -mtime +$DEL_AGE -name '*.gz' -print`
echo "Archived files older than $DEL_AGE days to be deleted:"
echo $FILES_OLD
echo " "
# Deletes old archived files.
find $FILEDIR -type f -mtime +$DEL_AGE -name '*.gz' -exec rm -f {} \;
echo "-= Rotation completed =-"
echo " "

Rename files from upper case filename to lower case

The following one line script will rename every file (in the current folder) to lowercase:

for i in *; do mv $i `echo $i | tr [:upper:] [:lower:]`; done

Ubuntu on a MacBook Pro

This are the steps I followed to get Ubuntu running on a MacBook Pro 9,2.

Install

Note: this procedure requires an .img file that you will be required to create from the .iso file you download.

TIP: Drag and Drop a file from Finder to Terminal to 'paste' the full path without typing and risking type errors.

• Download the desired file
• Open the Terminal (in /Applications/Utilities/ or query Terminal in Spotlight)
• Convert the .iso file to .img using the convert option of hdiutil

hdiutil convert -format UDRW -o ~/path/to/target.img ~/path/to/ubuntu.iso

Note: OS X tends to put the .dmg ending on the output file automatically.

Create a bootable Ubuntu install flash drive:
Run:

diskutil list

to get the current list of devices

Insert your flash media and run:

diskutil list

again and determine the device node assigned to your flash media (e.g. /dev/disk2)
Run:

diskutil unmountDisk /dev/diskN

(replace N with the disk number from the last command; in the previous example, N would be 2)

Execute

sudo dd if=/path/to/downloaded.img of=/dev/diskN bs=1m

(replace /path/to/downloaded.img with the path where the image file is located; for example, ./ubuntu.img or ./ubuntu.dmg).
Using /dev/rdisk instead of /dev/disk may be faster.

If you see the error dd: Invalid number '1m', you are using GNU dd. Use the same command but replace bs=1m with bs=1M.

If you see the error dd: /dev/diskN: Resource busy, make sure the disk is not in use. Start the Disk Utility.app and unmount (don't eject) the drive.

Finally run:

diskutil eject /dev/diskN

and remove your flash media when the command completes

Restart your Mac and press alt while the Mac is restarting to choose the USB-Stick

Follow the on screen instructions.

WiFi

After booting into Ubuntu, the wifi card was not working, to get it to work I connected it to my router with a network cable and followed this steps:

Download the driver:

wget http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2

And install it:

tar xf broadcom-wl-5.100.138.tar.bz2
sudo apt-get install b43-fwcutter
sudo b43-fwcutter -w "/lib/firmware" broadcom-wl-5.100.138/linux/wl_apsta.o

then add:

b43

to /etc/modules and reboot

Keyboard and mouse:

This is a little extra to get natural scrolling and OS X like key bindings.

Create a Xmodmap conf file

vi ~/.Xmodmap

and paste the following inside:

!!Enable Natural scrolling (vertical and horizontal)
pointer = 1 2 3 5 4 7 6 8 9 10 11 12
!!Swap CMD and CTRL keys
remove control = Control_L
remove mod4 = Super_L Super_R
keysym Control_L = Super_L
keysym Super_L = Control_L
keysym Super_R = Control_L
add control = Control_L Control_R
add mod4 = Super_L Super_R

Under Mac OS X, the combination cmd+space opens Spotlight, to emulate this, install the package compizconfig-settings-manager.

sudo aptitude install compizconfig-settings-manager

Open it using the ccsm command, or search for it in Dash.
Find Ubuntu Unity Plugin->Behavior->Key to show the launcher and change it to <Primary>space, using the Grab key combination button. It may be also shown as <Control><Primary>space.

You can now have a behavior similar to Mac OS X in Ubuntu 12.04. You can change the virtual desktop using cmd+alt+arrow. You can cut, copy, and paste using cmd+x, cmd+c, and cmd+v and summon the dash with cmd+space.

Installing Oracle 11g R2 Express Edition on Ubuntu 64-bit

This are the steps I took to install Oracle 11g R2 Express Edition on an Ubuntu 12.04 LTS (Precise Pangolin) Server and are based on the tutorial found here:

Download the Oracle 11gR2 express edition installer from the link given below:

http://www.oracle.com/technetwork/products/express-edition/downloads/index.html

( You will need to create a free oracle web account if you don't already have it )

Unzip it :

unzip oracle-xe-11.2.0-1.0.x86_64.rpm.zip

Install the following packages :

sudo apt-get install alien libaio1 unixodbc vim

The Red Hat based installer of Oracle XE 11gR2 relies on /sbin/chkconfig, which is not used in Ubuntu. The chkconfig package available for the current version of Ubuntu produces errors and my not be safe to use. So you'll need to create a special chkconfig script, below is a simple trick to get around the problem and install Oracle XE successfully:

sudo vi /sbin/chkconfig

(copy and paste the following into the file )

#!/bin/bash
# Oracle 11gR2 XE installer chkconfig hack for Ubuntu
file=/etc/init.d/oracle-xe
if [[ ! `tail -n1 $file | grep INIT` ]]; then
echo >> $file
echo '### BEGIN INIT INFO' >> $file
echo '# Provides: OracleXE' >> $file
echo '# Required-Start: $remote_fs $syslog' >> $file
echo '# Required-Stop: $remote_fs $syslog' >> $file
echo '# Default-Start: 2 3 4 5' >> $file
echo '# Default-Stop: 0 1 6' >> $file
echo '# Short-Description: Oracle 11g Express Edition' >> $file
echo '### END INIT INFO' >> $file
fi
update-rc.d oracle-xe defaults 80 01
#EOF

Save the above file and provide appropriate execute privilege :

chmod 755 /sbin/chkconfig

Oracle 11gR2 XE requires to set the following additional kernel parameters:

sudo vi /etc/sysctl.d/60-oracle.conf 

(Enter the following)

# Oracle 11g XE kernel parameters
fs.file-max=6815744
net.ipv4.ip_local_port_range=9000 65000
kernel.sem=250 32000 100 128
kernel.shmmax=536870912

(Save the file)

Note: kernel.shmmax = max possible value , e.g. size of physical RAM ( in bytes e.g. 512MB RAM == 512*1024*1024 == 536870912 bytes )

Verify the change :

sudo cat /etc/sysctl.d/60-oracle.conf

Load new kernel parameters:

sudo service procps start

Verify:

sudo sysctl -q fs.file-max
-> fs.file-max = 6815744

Increase the system swap space : Analyze your current swap space by following command :

free -m

Minimum swap space requirement of Oracle 11gR2 XE is 2 GB . In case, your is lesser , you can increase it by following steps in one of my previous posts.

make some more required changes :

sudo ln -s /usr/bin/awk /bin/awk
sudo mkdir -p /var/lock/subsys
sudo touch /var/lock/subsys/listener

Convert the red-hat ( rpm ) package to Ubuntu-package :

sudo alien --scripts -d oracle-xe-11.2.0-1.0.x86_64.rpm

(this may take a long time)

Go to the directory where you created the ubuntu package file in the previous step and enter following commands in terminal :

sudo dpkg --install oracle-xe_11.2.0-2_amd64.deb 

Do the following to avoid getting MEMORY TARGET error ( ORA-00845: MEMORY_TARGET not supported on this system ) :

sudo rm -rf /dev/shm
sudo mkdir /dev/shm
sudo mount -t tmpfs shmfs -o size=2048m /dev/shm

(here size will be the size of your RAM in MBs ).

The reason of doing all this is that on a Ubuntu system /dev/shm is just a link to /run/shm but Oracle requires to have a seperate /dev/shm mount point.

To make the change permanent do the following :

create a file named S01shm_load in /etc/rc2.d :

sudo vi /etc/rc2.d/S01shm_load

Then copy and paste following lines into the file :

#!/bin/sh
case "$1" in
start) mkdir /var/lock/subsys 2>/dev/null
touch /var/lock/subsys/listener
rm /dev/shm 2>/dev/null
mkdir /dev/shm 2>/dev/null
mount -t tmpfs shmfs -o size=2048m /dev/shm ;;
*) echo error
exit 1 ;;
esac

Save the file and provide execute permissions :

chmod 755 /etc/rc2.d/S01shm_load

This will ensure that every-time you start your system, you get a working Oracle environment.

You can now proceed to the Oracle initialization script

sudo /etc/init.d/oracle-xe configure

Enter the following configuration information:

• A valid HTTP port for the Oracle Application Express (the default is 8080)
• A valid port for the Oracle database listener (the default is 1521)
• A password for the SYS and SYSTEM administrative user accounts
• Confirm password for SYS and SYSTEM administrative user accounts
• Whether you want the database to start automatically when the computer starts (next reboot).

Before you start using Oracle 11gR2 XE you have to set-up a few more things :

a) Set-up the environmental variables, add following lines to the bottom of /etc/bash.bashrc :

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export ORACLE_SID=XE
export NLS_LANG=`$ORACLE_HOME/bin/nls_lang.sh`
export ORACLE_BASE=/u01/app/oracle
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
export PATH=$ORACLE_HOME/bin:$PATH

b) execute your .profile to load the changes:

source /etc/bash.bashrc

Start the Oracle 11gR2 XE :

sudo service oracle-xe start

The output should be similar to following :

user@machine:~$ sudo service oracle-xe start
Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.
user@machine:~$

And you're done :)