Running a PowerCLI Script

I'm a Linux User and fond some interesting scripts for VMWare on the Internet, the problem was I didn't knew how to run them.

Now I finally learned how to run a Power Shell script, so here's how I've done it:

Save the script to a file with a filetype of .ps1

Open the PowerCLI prompt on your PC

Connect to the vCenter with the Connect-VICenter cmdlet

dot-source the .ps1 file. That way the function is know in your PowerCLI session

. ./yourfile.ps1

Call the function implemented by the scipt.

Export everpad notes to HTML files

If you use Everpad you can use the following script to export your notes to html files. The script will write the notes into ~/exported_notes and the notes will be sorted under folders with their notebook names:

import os
import sys
import datetime
import sqlite3
export_path = os.getenv("HOME") + '/exported_notes'
#Create directory if it does not exist
if not os.path.exists(export_path):
    os.makedirs(export_path)
# Create a connection to the database.
filename = os.getenv("HOME") + '/.everpad/everpad.5.db'
conn = sqlite3.connect(filename, detect_types=sqlite3.PARSE_DECLTYPES|sqlite3.PARSE_COLNAMES)
# Create a cursor object to do the interacting.
c = conn.cursor()
# Grab the columns we need.
sql = 'SELECT notebooks.name, notes.title, notes.content FROM notes'
sql += ' inner join notebooks on notes.notebook_id = notebooks.id'
rows = c.execute(sql)
# Iterate over the result.
for row in rows:
    note_path = export_path + '/' + row[0].replace("/", "_")
    note_path = note_path.replace(" ", "_");
    if not os.path.exists(note_path):
        os.makedirs(note_path)
    note_path += '/' + row[1].replace("/", "_") + '.html'
    note_path = note_path.replace(" ", "_");
    note = open(note_path, 'a+')
    note.write('<h1>'+row[1].encode('utf-8')+'</h1></br>')
    note.write(row[2].encode('utf-8'))
    note.close()
# Commit the changes and close everything.
conn.commit()
c.close()
conn.close()

302 redirects behind SSL reverse proxy

Problem

You have a web server running plain http behind an Apache reverse proxy running https. Your application uses 302 redirects to announce new URLs or whatever the reason is for doing so.

The client will be redirected a plain http url.

Solution

You can create a Vhost listenig for plain http and all that it does is redirect the clients to https, like this:

NameVirtualHost *:80
<VirtualHost *:80>
   ServerName www.example.com
   Redirect permanent / https://secure.example.com/
</VirtualHost>

This way, the client is redirected to the plain http by the proxied server and then redirected back to https by the proxy server.

Another solution is to use mod_rewrite:

RewriteEngine On
# This will enable the Rewrite capabilities
RewriteCond %{HTTPS} !=on
# This checks to make sure the connection is not already HTTPS
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
# This rule will redirect users from their original location, to the same location but using HTTPS.
# i.e.  http://www.example.com/foo/ to https://www.example.com/foo/
# The leading slash is made optional so that this will work either in httpd.conf
# or .htaccess context

This does the same as the previous solution.

But what if there is some firewall or load balancer that prevents plain http packets from reaching your server?

Since Apache version 2.2.4 mod_headers is able to rewrite response headers. Just add the following to your https vhost that is serving as reverse proxy to your application:

Header edit Location ^http://(.*)$ https://$1

This configuration statement will solve your problem. Redirects triggered by your back end web servers will be re-rewritten to comply with your SSL terminating reverse proxy/load balancer.

Example:

Listen a.b.c.d:443
<VirtualHost a.b.c.d:443>
    ServerName example.org
    # …SSL configuration…
    ProxyRequests off
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/
    RequestHeader set X-Forwarded-Protocol "https"
    Header edit Location ^http://(.*)$ https://$1
</VirtualHost>
<VirtualHost a.b.c.d:80>
    ServerName example.org
    ProxyRequests off
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/
    RequestHeader set X-Forwarded-Protocol "http"
</VirtualHost>

Iptables blacklists

Many of you already use online blacklists to fight spam. Recently I've dicovered http://www.openbl.org/ and started using their lists on my firewall to prevent attacks from hosts that are known to be preforming attacks. It works in a very similar way to all the spam blacklists out there, and this is how I've implemented them on my Firewall.

First of all you'll need to  have some packages installed:

sudo apt-get install iptables ipset wget

now create an ipset to store all the abusing IP addresses and use iptables to block them:

#!/bin/bash
BLOCKDB="block.txt"
WORKDIR="/tmp"
pwd=$(pwd)
cd $WORKDIR
#List of ips to block
ipset --create blackips iphash
## Obtain List of badguys from openbl.org
wget -q -c --output-document=$BLOCKDB http://www.openbl.org/lists/base.txt
if [ -f $BLOCKDB ]; then
    IPList=$(grep -Ev "^#" $BLOCKDB | sort -u)
    for i in $IPList
    do
        ipset --add blackips $i
    done
fi
rm $BLOCKDB
## Obtain List of badguys from ciarmy.com
wget -q -c --output-document=$BLOCKDB http://www.ciarmy.com/list/ci-badguys.txt
if [ -f $BLOCKDB ]; then
    IPList=$(grep -Ev "^#" $BLOCKDB | sort -u)
    for i in $IPList
    do
        ipset --add blackips $i
    done
fi
rm $BLOCKDB
## Obtain List of badguys from dshield.org
wget -q -c --output-document=$BLOCKDB http://feeds.dshield.org/top10-2.txt
if [ -f $BLOCKDB ]; then
    IPList=$(grep -E "^[1-9]" $BLOCKDB | cut -f1 | sort -u)
    for i in $IPList
    do
        ipset --add blackips $i
    done
fi
rm $BLOCKDB
#List of networks to block
ipset --create blacknets nethash
## Obtain List of badguys from dshield.org
wget -q -c --output-document=$BLOCKDB http://feeds.dshield.org/block.txt
if [ -f $BLOCKDB ]; then
    IPList=$(grep -E "^[1-9]" $BLOCKDB | cut -f1,3 | sed "s/\t/\//g" | sort -u)
    for i in $IPList
    do
        ipset --add blacknets $i
    done
fi
rm $BLOCKDB
## Obtain List of badguys from spamhaus.org
    wget -q -c --output-document=$BLOCKDB http://www.spamhaus.org/drop/drop.lasso
    if [ -f $BLOCKDB ]; then
      IPList=$(grep -E "^[1-9]" $BLOCKDB | cut -d" " -f1 | sort -u)
      for i in $IPList
      do
        ipset --add blacknets $i
      done
    fi
    rm $BLOCKDB
#Drop blacklisted ips
iptables -A FORWARD -m set --match-set blackips src -j DROP
iptables -A FORWARD -m set --match-set blacknets src -j DROP
cd $pwd

In the above script I've used two ipsets, one for storing IP addresses and another to store network addresses, you can add this scritp to your existing firewall and start taking advantage of the blacklists.
OpenBL is accepting donations http://www.openbl.org/donations.html if you can you should help.

NetFLOW over IPSEC VPN

I had the following NetFlow configuration on my router:

 ip flow-cache timeout active 1
 ip flow-export source FastEthernet0/1.1
 ip flow-export version 5
 ip flow-export destination 10.39.30.5 9996
!
interface FastEthernet0/0.2
 ip flow ingress
interface FastEthernet0/1.12
 ip flow ingress
interface FastEthernet0/1.40
 ip flow ingress

Witch worked fine on my other routers. But in this particular case, the NetFLOW server was only accecible through an IPSEC VPN and the flows weren't getting there.

The solution to this was to use "Flexible Netflow" configuration. This allows for the NetFlow export to be sent down the standard IPSEC VPN tunnel.

An example of the NetFlow config is as follows:

flow exporter FLOW_EXPORTER
 destination 10.39.30.5
 source FastEthernet0/1.1
 output-features
 transport udp 9996
 export-protocol netflow-v5
!
!
flow monitor FLOW_MONITOR
 record netflow-original
 exporter FLOW_EXPORTER
 cache timeout active 1
!
interface FastEthernet0/0.2
ip flow monitor FLOW_MONITOR input
!
interface FastEthernet0/1.12
ip flow monitor FLOW_MONITOR input
!
interface FastEthernet0/1.40
ip flow monitor FLOW_MONITOR input
!

Disable IPv6 on Ubuntu

Edit your /etc/sysctl.conf file and add the following to the bottom:

#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Or you can use the following script:

echo "#disable ipv6" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf

For this changes to take effect you must reboot your system.
After rebooting you can check if IPv6 has been disabled with the following command:

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

0 means it's enabled and 1 - disabled.

Linux Stress tests

Consume CPU:

Fork bomb:

:(){ :|:& };:

The next one will load four CPU cores at 100%:

for i in `seq 1 4` ; do while : ; do : ; done & ; done

Or:

for i in `seq 1 4` ; do cat /dev/zero > /dev/null & ; done

Or:

#!/bin/bash
duration=120 # seconds
instances=4 # cpus
endtime=$(($(date +%s) + $duration))
for ((i=0; i<instances; i++))
do
while (($(date +%s) < $endtime)); do : ; done &
done

Using the stress tool:

stress --cpu 3

Consume RAM:

Create a 30gb ramdisk and fills it with file full of zeroes:

sudo mount -t tmpfs -o size=30G tmpfs /mnt
dd if=/dev/zero of=/mnt/tmp bs=10240 count=30720MB

Create a giant virable:

x="x" ; while : ; do x=$x$x ; echo -n "." ; done

Consume Disk:

dd if=/dev/zero of=bigfile bs=10240 count=30720MB

Simulate packet loss:

For randomly dropping 10% of incoming packets:

iptables -A INPUT -m statistic --mode random --probability 0.1 -j DROP

and for dropping 10% of outgoing packets:

iptables -A OUTPUT -m statistic --mode random --probability 0.1 -j DROP

How to reset the root password of VMware ESXi 4.1 and 5.0

You can't recover your old password but by following this steps you can set it to blank and then set a new one.

First you'll need a Linux live CD, any one will do.

After booting to a live session of Linux you must look for a file named state.tgz on your vmware host's hard drive. To do so I used:

parted -l

To list all available partitions and mounted every VFAT partition to look inside, I found it was on /dev/sda5, in your case it might be on a different one, you can mount the partitions with:

mount /dev/sda1 /mnt

(replace sda with your device's name and 1 with your partition numver)

then check inside if the file state.tgz exists:

ls /mnt/

After finding the state.tzg file you must uncompress it using:

cd /tmp
tar xzf /mnt/Hypervisor3/state.tgz

this will get you a local.tgz file witch you have to extract using:

tar xzf local.tgz

now edit the file /tmp/etc/shadow

vi etc/shadow

Inside locate the root account and just remove it's hash (everything between the first and the second colon) and login to the service console as root with no password at all.

Finally re-pack the files and move the modified state.tgz back to the VFAT partition. Probably it is a good idea to make a backup copy of the original state.tgz in case something goes wrong:

mv /mnt/state.tgz /mnt/state.tgz.bak
rm local.tgz
tar czf local.tgz etc
tar czf state.tgz local.tgz
mv state.tgz /mnt/

Reboot back into ESXi and you're done.

Calculating total disk usage by files with specific extension

For example if you want to check how much space is being used by log files on your entire system, you can use the following:

find / -type f -name "*.log*" -exec du -b {} \; | awk '{ sum += $1 } END { kb = sum / 1024; mb = kb / 1024; gb = mb / 1024; printf "%.0f MB (%.2fGB) disk space used\n", mb, gb}'

Just replace "*.log*" with the file extension you want to search for and the above will give you the disk used by the sum of all the files with that extension.

PostgreSQL cluster using DRBD and hot standby

Cluster Configuration:

First install all the necessary packages:

yum install gfs2-utils cman fence-virtd-checkpoint lvm2-cluster perl-Net-Telnet rgmanager device-mapper-multipath ipvsadm piranha luci modcluster cluster-snmp ricci

yum groupinstall "High Availability"
yum install postgresql-server

chkconfig --level 123456 ricci on
chkconfig --level 123456 luci on
chkconfig --level 123456 cman on
chkconfig --level 123456 iptables off
chkconfig --level 123456 ip6tables off
chkconfig postgresql on
chkconfig cman on
chkconfig rgmanager on

Now edit the cluster configuration file:

vi vi /etc/cluster/cluster.conf

Make it look like this:

<?xml version="1.0"?>
<cluster config_version="7" name="pgcluster">
<clusternodes>
<clusternode name="10.39.30.7" votes="1" nodeid="1">
<fence/>
</clusternode>
<clusternode name="10.39.30.8" votes="1" nodeid="2">
<fence/>
</clusternode>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="PGSQL" nofailback="0" ordered="0" restricted="0">
<failoverdomainnode name="10.39.30.7"/>
<failoverdomainnode name="10.39.30.8"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.39.30.6" monitor_link="on" sleeptime="10"/>
<postgres-8 config_file="/var/lib/pgsql/data/postgresql.conf" name="pgsql" shutdown_wait="5" />
</resources>
<service autostart="1" exclusive="0" domain="PGSQL" name="pgsql" recovery="relocate">
<drbd name="drdb-postgres" resource="r0">
<fs device="/dev/drbd0" fsid="6202" fstype="ext3" mountpoint="/var/lib/pgsql" name="pgsql" options="noatime"/>
</drbd>
<ip ref="10.39.30.6"/>
<postgres-8 ref="pgsql"/>
</service>
</rm>
<cman expected_votes="1" two_node="1"/>
<fence_daemon clean_start="1" post_fail_delay="0" post_join_delay="3"/>
</cluster>

DRDB Configuration:

Install the necessary files:

yum install gcc flex make libxslt rpm-build redhat-rpm-config kernel-devel

You need to download and install DRBD manually

wget http://oss.linbit.com/drbd/8.4/drbd-8.4.1.tar.gz 

 the following commands will generate DRBD RPM packages:

tar -xvf *.tar.gz
mkdir -p /root/rpmbuild/SOURCES/
cp drbd*.tar.gz /root/rpmbuild/SOURCES/
cd drbd-8.4.1
./configure --with-rgmanager --enable-spec --with-km
make tgz
rpmbuild --bb drbd.spec --without xen --without heartbeat --without udev --without pacemaker --with rgmanager
rpmbuild --bb drbd-kernel.spec
rpmbuild --bb drbd-km.spec

Now install the newly created packages:

cd /root/rpmbuild/RPMS/x86_64
rpm -i drbd-utils-8.4.1-1.el6.x86_64.rpm drbd-bash-completion-8.4.1-1.el6.x86_64.rpm drbd-8.4.1-1.el6.x86_64.rpm drbd-rgmanager-8.4.1-1.el6.x86_64.rpm drbd-km-2.6.32_279.14.1.el6.x86_64-8.4.1-1.el6.x86_64.rpm

Add your nodes IP addresses to the hosts file on both machines:

vi /etc/hosts

10.39.30.7 RHPG1
10.39.30.8 RHPG2

Create a DRBD configuration file:

vi /etc/drbd.d/r0.res

resource r0 {
   device /dev/drbd0;
   meta-disk internal;
   on RHPG1 {
      address 10.39.30.7:7789;
      disk /dev/sdb1;
   }
   on RHPG2 {
      address 10.39.30.8:7789;
      disk /dev/sdb1;
   }
}

Create the partion /dev/sdb1 but do not format it:

fdisk /dev/sdb

Run on both machines:

drbdadm create-md r0
modprobe drbd
drbdadm up r0

Run on one of the machines to create the file system:

drbdadm -- --overwrite-data-of-peer primary r0

Check the sync status on any of the hosts with:

service drbd status

Create the file system on /dev/drbd0

mkfs.ext3 /dev/drbd0

and move over the PostgreSQL data

mkdir /tmp/pgdata
mount /dev/drbd0 /tmp/pgdata
cp -r /var/lib/pgsql /tmp/pgdata

Wait until the data is synced over the two hosts check the status with:

service drbd status

Unmount the drbd device:

umount /dev/drbd0

and then, on both hosts do:

rm -rf /var/lib/pgsql/*

Restart the drbd service

service drbd restart

the status from:

service drbd status

should show that both hosts are in secondary:

0:r0 Connected Secondary/Secondary UpToDate/UpToDate C

And ready to be managed by rgmanager.