Check file system usage using command line

If you want to check you disk space left you can use:

df -h

And if you whant to find the files bigger than a given size, you can use this command:

find </path/to/directory/> -type f -size +<size-in-kb>k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’

all you need is to specify the path and the size in kb (50000 for 50Mb for example).

You can also check the top 10 biggest files in a given directory with:

du -sk </path/to/directory/>* | sort -r -n | head -10

Or with a more readable output:

du -sh $(du -sk ./* | sort -r -n | head -10 | cut -d / -f 2)

Baruwa - Quarantine Release and Preview not Working

If you can't preview or release messages from the baruwa GUI, and allways get an error message saying "Failed: Message not found in the quarantine" in the logs, you might have one of the following problems:

Bad hostname:

Check the hostname listed in "Received by: " which is the hostname field in the messages
table in the DB and see if it corresponds to the host name where the message should be stored and also make sure you can resolve that host name.

The "Received by: " hostname comes from mailscanner which gets the server's hostname at startup so if you change your machine's hostmane you'll have to restart malscanner for it to take effect.

MailScanner Spool folders permissions:

Check the users you are using for the quarantine folder in your MalScanner.conf.
Quarantine user should not be root.

Quarantine User = postfix
Quarantine Group = celeryd
Quarantine Permissions = 0660
Quarantine Whole Message = yes
Quarantine Whole Messages As Queue Files = no

And check the permissions of your spool folders:

/var/spool/
drwxr-xr-x 5 postfix postfix 4096 Aug 21 03:04 MailScanner

/var/spool/MailScanner
drwxr-xr-x 10 postfix postfix 4096 Sep 24 19:26 incoming
drwxr-xr-x 10 postfix celeryd 4096 Sep 24 00:00 quarantine

Also, your webserver's user should have access to the quarantine folder, you can add it to the celeryd group in this case.

MailScanner Quick.Peek:

check the path for the Quick.Peek executable:

which Quick.Peek
/usr/sbin/Quick.Peek

and make sure it is correct in the Baruwa's settings.py file.
Also, in settings.py check the provided path for the MalSacnner.conf file.

Vodafone K5005 (Huawei E389) 4G modem on Ubuntu

This modem works with Ubuntu Precise Pangolin (12.04) but it is not detected automatically by network manager.

UPDATE: In the comments, a reader named "Big Brother" has a nicer solution, instead of using the scripts below, just follow this steps:

1- Add these lines to /lib/udev/rules.d/40-usb_modeswitch.rules:

# Vodafone K5005 (Huawei E398)
ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="14c3", RUN+="usb_modeswitch '%b/%k'"

2- Create file /etc/usb_modeswitch.d/12d1:14c3:

# Vodafone K5005 (Huawei E398)
TargetVendor= 0x12d1
TargetProduct= 0x14c8
MessageContent="55534243123456780000000000000011062000000100000000000000000000"

3- Unplug device, plug it back and it should work automagically ;)

Deprecated method:
In order to get it working with network manager I have to use the following script (it must be ran as root):

#!/bin/bash
rmmod option
modprobe option
echo "12d1 14c8" > /sys/bus/usb-serial/drivers/option1/new_id
usb_modeswitch -v 12d1 -p 14c3 -V 12d1 -P 14c8 -M "55534243123456780000000000000011062000000100000000000000000000" -n 1

Note that the commands above are for the Vodafone branded (K5005) Huawei E389 dongle, for the unbranded device the product ID is different and you should use:

#!/bin/bash
rmmod option
modprobe option
echo "12d1 1506" > /sys/bus/usb-serial/drivers/option1/new_id
usb_modeswitch -v 12d1 -p 1505 -V 12d1 -P 1506 -M "55534243123456780000000000000011062000000100000000000000000000" -n 1

You can check the product id with:

lsusb

In my case I get:

Bus 002 Device 007: ID 12d1:14c3 Huawei Technologies Co., Ltd.

Dovecot sDbox vs mDbox

Recently I've been planing to update our dovecot installation and migrate from maildir do dbox witch raised the question, should we go with sdbox or with mdbox?

I've been reading some threads on the dovecot mailing list and compiled this list of questions and answers to help in the decision, all answers are from Timo Sirainen:

1. What is the advantage to using multiple files?

A: mdbox in theory uses less disk I/O for "normal users".

2. What is the advantage to using a single sdbox file for each user?

A: It's simpler. More difficult to get corrupted. Also if in future there exists a filesystem that supports smaller files better, it's then faster than mdbox. Probably unlikely that it will happen anytime soon.

3. Is this a binary format, or txt (UTF?)?

A: dbox headers/metadata is ASCII. The message bodies can of course be anything.

4. Are there real-world benchmarks showing measurable differences between maildir, sdbox, mdbox?

A: Not that I'm aware of. So far everyone I've tried to ask have replaced their whole mail system and their storage, so the before/after numbers can't be compared. I'm very interested in knowing myself too.

5. Are sdbox & mdbox equally stable to Maildir? Are they recommended for production systems?

A: sdbox is so simple that I doubt anyone will find any kind of corruption bugs. mdbox is more complex, but people are using it in production and I haven't heard of any problems recently. Although there have been bugs in how mdbox handles already corrupted files, v2.0.10 had several fixes related to that.

6. In mdbox we should not use a ramdisk for indexes. But what about sdbox? sdbox indexes work as maildir indexes? Are sdbox indexes bigger than maildir indexes?

A: If this is a heavy use box, having everyone's indexes being rebuilt at the same time could bring it to its knees...

Since this is a server I'm sure you have adequate power protection (UPS), so only extended power outages might be an issue - but then you should also have it configured to safely shut down in this event, no?

But anyway, yes, the indexes will be rebuilt and everything should continue working...

7. One of the main advantages (speed wise) of dbox over maildir is that index files are the only storage for message flags and keywords. What happens when we want to recover some messages from backup? With maildir we can rebuild message indexes, but I am not sure about dbox. Should we also restore "old indexes" and merge with the "new indexes" in order to restore the deleted messages?

A: The intended way to restore stuff is to either restore the entire dbox to a temp directory, or at least all the important parts of it (indexes + the files that contain the wanted mails) and then use something like:

doveadm import sdbox:/tmp/restoredbox "" saved since 2011-01-01

8. The previous question applies to sdbox and mdbox. In the case of mdbox, we can configure rotation of files using /mdbox_rotate_size/ . We would like to rotate daily, not based in size (our users ask us for yesterday's backup). How can we accomplish this?

A: mdbox_rotate_interval = 1d

But note that that doesn't guarantee that there will be only one file. Even if you set mdbox_rotate_size to 10 GB or something (or I think 0 makes it unlimited, not sure), it's possible that two files will be created if mails are being saved at the same time. mdbox never waits for locks when writing to a file, instead it'll just use another file or create a new one.

Anyway, if it's not a big deal restoring the user's entire mailbox temporarily you can restore only yesterday's mails by giving proper search query parameter to doveadm import.

9. We have now 17.000.000 messages in our maildir, almost 1.5 TB (zlib compresssion enabled). Our backup time with bacula is rather bad: 24 hours for a full backup, most of the time the backup is busy fstat'ing all those little messages.

A: In case of Maildir there's no point in fstating any mail files. I'd guess it should be possible to patch bacula to not do that.

10. We think that mdbox can help us in this. Does anybody has good experiences migrating from maildir->mdox in "large" enviroments? What about mdox performance & reliability?

A: I haven't recently heard of corruption complaints about mdbox.. Previously when there were those, I didn't hear of complains about losing mails or anything, so that's good :)

Someone's experience from around 2011-03, I believe that dsync has improved by now:

• Sdbox is using far too much I/O on a busy server, I had to switch to mdbox. sdbox is not sustainable when having very large mailbox, IO becomes too high (even with high-end storage devices)
  
  • Timo said that sdbox is not expected to have more I/O than maildir.
• Mdbox is running well so far, and resources (IO or CPU) are not an issue anymore.
• Converting from Maildir to s/mdbox is easy
• Converting from sdbox to mdbox has been a complete nightmare. I have never managed to make it completely, finally made it through imap protocol between 2 instance of dovecot. You better choose before sd or md, but not try to convert between the 2. Dsync is too buggy to convert sdbox to mdbox. The only solution I found was to use IMAP protocol to read from sdbox and write as mdbox.

View running processes in Oracle DB

This will show you a list of all running processes:

SET LINESIZE 200
SET PAGESIZE 200
SELECT PROCESS pid, sess.process, sess.status, sess.username, sess.schemaname, sql.sql_text FROM v$session sess, v$sql sql WHERE sql.sql_id(+) = sess.sql_id AND sess.type = 'USER';

Identify database SID based on OS Process ID

use the following SQL query, when prompted enter the OS process PID:

SET LINESIZE 100
col sid format 999999
col username format a20
col osuser format a15
SELECT b.spid,a.sid, a.serial#,a.username, a.osuser
FROM v$session a, v$process b
WHERE a.paddr= b.addr
AND b.spid='&spid'
ORDER BY b.spid;

For making sure you are targeting the correct session, you might want to review the SQL associated with the offensive task, to view the SQL being executed by the session you can use the following SQL statement:

SELECT
b.username, a.sql_text
FROM
v$sqltext_with_newlines a, v$session b, v$process c
WHERE
c.spid = '&spid'
AND
c.addr = b.paddr
AND
b.sql_address = a.address;

Killing the session

The basic syntax for killing a session is shown below.

ALTER SYSTEM KILL SESSION 'sid,serial#';

In a RAC environment, you optionally specify the INST_ID, shown when querying the GV$SESSION view. This allows you to kill a session on different RAC node.

ALTER SYSTEM KILL SESSION 'sid,serial#,@inst_id';

The KILL SESSION command doesn't actually kill the session. It merely asks the session to kill itself. In some situations, like waiting for a reply from a remote database or rolling back transactions, the session will not kill itself immediately and will wait for the current operation to complete. In these cases the session will have a status of "marked for kill". It will then be killed as soon as possible.

In addition to the syntax described above, you can add the IMMEDIATE clause.

ALTER SYSTEM KILL SESSION 'sid,serial#' IMMEDIATE;

This does not affect the work performed by the command, but it returns control back to the current session immediately, rather than waiting for confirmation of the kill.

If the marked session persists for some time you may consider killing the process at the operating system level. Before doing this it's worth checking to see if it is performing a rollback. If the USED_UREC value is decreasing for the session in question you should leave it to complete the rollback rather than killing the session at the operating system level.

Unlock Oracle user account

Here's how to lock or unlock Oracle database user accounts.

ALTER USER username ACCOUNT LOCK;

ALTER USER username ACCOUNT UNLOCK;

you may also have to use:

GRANT connect, resource TO username;

to solve the "ORACLE ERROR:ORA-28000: the account is locked" error.

Limit bandwidth of rsync over ssh

This will limit the connection to 80Kb/s:

rsync -auvPe "trickle -d 80 ssh" user@host:/src/ /dst/

Add extra disk as /home

Install the disk and then use:

fdisk -l

to check the new device name, if the new disk is not detected try installing scsitools:

apt-get install scsitools

then run:

rescan-scsi-bus

and issue fdisk -l again, supposing your new disk is /dev/sdb use

fdisk /dev/sdb

to create a new partition, press n, then p for a primary partition then enter 1 sinse this will be the only partition on the drive, when it asks you about the first and last cylinders, just use the defaults.
now, to format the newly created partition use:

mkfs.ext4 /dev/sdb1

when done use:

blkid

to check the new partition's uuid and using that edit the /etc/fstab file andding:

UUID=d70d801e-5246-46e2-a7ed-1a95819fd326 /home ext4 errors=remount-ro 0 1

Now mount the new partition on a temporary location with:

mount /dev/sdb1/mnt/

and copy the contents of the current home to the new partition:

cp -r /home/*/mnt/

when done delete all contents from the current home

rm -rf /home/*

unmount the new partiotion

umount /mnt

and remount it under /home

mount /dev/sdb1/home/

Duplicate Oracle Schema

Using imp/exp:
Export the database using:

exp 'system/password' owner=schema_to_be_duplicated file=filename.dmp log=logfile.log

Then import the dump file into the target schema:

imp 'system/password' fromuser=schem_to_be_duplicated touser=target_username file=
filename.dmp

But keep in mind that the target schema must be available on the database. If it does not, then you must have to create it first using CREATE USER command). The import does not create user for you it only migrates objects & data within schemas.

Using impdp/expdp:

We must first create a directory object you can access. The directory object is only a pointer to a physical directory, creating it does not actually create the physical directory on the file system.

sqlplus / AS SYSDBACREATE OR REPLACE DIRECTORY DUMP_DIR AS '/home/oracle/dumpdir/';

Export with:

expdp schema_to_be_duplicated/password DUMPFILE=filename.dmp DIRECTORY=dmpdir

Import with:

impdp REMAP_SCHEMA=schema_to_be_duplicated:new_schema DUMPFILE=filename.dmp DIRECTORY=dmpdir EXCLUDE=JOB

you will be asked for a username and password, the default is system/system

Remember that you must first create the destination user:

CREATE USER new_schema IDENTIFIED BY new_password DEFAULT TABLESPACE users TEMPORARY TABLESPACE temp QUOTA UNLIMITED ON users;
GRANT CREATE session, CREATE table, CREATE view, CREATE procedure, CREATE synonym, CREATE SEQUENCE, CREATE TRIGGER, CREATE TYPE, CREATE MATERIALIZED VIEW, CREATE DATABASE LINK,Debug Any Procedure,Debug Connect Session TO new_schema;
ALTER USER new_schema QUOTA UNLIMITED ON USERS;

CloudStack reset password script

The process to install the password reset script described in the Cloudstack's admin guide was not working for me on an Ubuntu template so I tried to figure what was wrong with it.

In the admin guide they say that we should place the script in /etc/init.d/ and enable it using update-rc.d but that didn't work so I tried to place this in /etc/init/cloudstack.conf

##########################################################################
description "CloudStack password reset"
author "Luis Davim"
# Be sure to block the display managers until our job has completed. This
# is to make sure our kernel services are running before user
# may launch.
start on runlevel [235] or starting gdm or starting kdm or starting prefdm
stop on runlevel [06]
pre-start exec /etc/init.d/cloud-set-guest-password
post-stop exec /etc/init.d/cloud-set-guest-password

That didn't not work either, so I took a look at the script and figured it needed to have the network configured to run.

So I configured my network interface like this:

# The primary network interface
auto eth0
iface eth0 inet dhcp
post-up /etc/init.d/cloud-set-guest-password
pre-down /etc/init.d/cloud-set-guest-password

you can also link the script into the /etc/network/if-up(down) folders:

ln -s /etc/init.d/cloud-set-guest-password/etc/network/if-up/cloud-set-guest-password
ln -s /etc/init.d/cloud-set-guest-password/etc/network/if-down/cloud-set-guest-password

And that was it, now I have an Ubuntu template with a working password reset script.

Note: I've also modified the password script to use chpasswd insted of passwd --stdin since ubuntu does not have the --stdin option in passwd and both ubuntu and centos have chpasswd but that was/is not the problem because usermod with mkpasswd was working...

just replaced:

echo $password | passwd --stdin $user

with

echo "$user:$password" | chpasswd

Linux Transparent bridge

First you need to install the bridge-utils:

apt-get install bridge-utils

Configuring the bridge:

ifconfig eth0 0.0.0.0 promisc up
ifconfig eth1 0.0.0.0 promisc up
brctl addbr br0
brctl addif br0 eth0
brctl addif br0 eth1
ifconfig br0 1.2.3.4 netmask 255.255.255.0 up
route add default gw 1.2.3.4 dev br0

In this example, I suppose you are using eth0 and eth1. In the ifconfig line, I assigned IP address 1.2.3.4 to the bridge so I can access it remotely. Use an IP address in your network.
You may check that the bridge is working by using tcpdump:

# tcpdump -n -i eth0
...
(lots of funny stuff)
...
# tcpdump -n -i eth1
...
(lots of funny stuff)
...

Plug your machine into the network, and everything should work. Your Linux box is now a big, expensive two-port switch.

Making the Bridge Permanent

Edit the file /etc/network/interfaces and add:

auto br0
iface br0 inet dhcp
bridge_ports eth1 eth2
bridge_stp on

CloudStack LDAP

References:

• http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/LDAP_Authentication
• http://docs.cloudstack.org/CloudStack_Documentation/Developer's_Guide%3A_CloudStack

First you need to configure LDAP by making an API call with an URL like this:

http://127.0.0.1:8096/client/api?command=ldapConfig&hostname=127.0.0.1&searchbase=ou%3Dpeople%2Co%3DsevenSeas&queryfilter=%28%26%28uid%3D%25u%29%29&binddn=%20cn%3DJohn+Fryer%2Cou%3Dpeople%2Co%3DsevenSeas&bindpass=secret&port=10389&response=json

Or in a more readable format:

http://127.0.0.1:8096/client/api?command=ldapConfig
&hostname=127.0.0.1
&searchbase=ou%3Dpeople%2Co%3DsevenSeas
&queryfilter=%28%26%28uid%3D%25u%29%29
&binddn=%20cn%3DJohn+Fryer%2Cou%3Dpeople%2Co%3DsevenSeas
&bindpass=secret
&port=10389
&response=json

Note the URL encoded values, here you have the decoded version:

http://127.0.0.1:8096/client/api?command=ldapConfig
&hostname=127.0.0.1
&searchbase=ou=people,o=sevenSeas
&queryfilter=(&(uid=%u))
&binddn= cn=John Fryer,ou=people,o=sevenSeas
&bindpass=secret
&port=10389
&response=json

You can use this link to encode/decode your url -> http://meyerweb.com/eric/tools/dencoder/

After you've created your URL (with encoded values) open your browser, login into cloudstack and then fire up your ldap config URL.
Now if you go back to cloudstack and under "Global Settings" search for LDAP and you should see that LDAP is configured.

Now you have to manually create the user accounts with the same logins as in your LDAP server or you can use the CloudStack API to make a script and "sync" your LDAP users into CloudStack, I've written a PHP script that does this.
You'll have to modify it to match your LDAP schema and you can get it after the break.

How to reset lost user password on a Linux Machine

First you’ll want to make sure to choose the regular boot kernel that you use (typically just the default one), and then use the “e” key to choose to edit that boot option.

Now just hit the down arrow key over to the “kernel” option, and then use the “e” key to switch to edit mode for the kernel option.

Note: if the grub menu does not show up try hitting the Shift or the Space keys right after the bios screen.

You’ll want to remove the “ro quiet splash” part with the backspace key, and then add this onto the end:

rw init=/bin/bash

Press F10 or ctrl+x to boot with that option.

At this point the system should boot up very quickly to a command prompt.

Now you can use the passwd command to change the password.

passwd username

where username is the username you want to reset.
After changing your password, use the following commands to reboot your system. (The sync command makes sure to write out data to the disk before rebooting)

sync
reboot –f

I found that the –f parameter was necessary to get the reboot command to work for some reason. You could always hardware reset instead, but make sure to use the sync command first.

NOTE for VMWare users:

Probably you wont ever see the boot menu on a VM because it will boot to fast, to work around this you can set up the vm to delay the bios for a few seconds:

bios.bootDelay = "15000"

This causes the bios to delay for 15 seconds so you can press keys, you can set it in the VM .vmx file or using vSphere under the vm settings, on the options tab, boot options.

Then just start the VM and press esc on the grub loading message and follow the steps described above.

Increase LVM Volume

This is how I did it on a VMWare Machine but the procedure should be the same for a phisical machine:

First off, VMWare, like many other hypervisors, allows to create a second HDD on the fly while the vm is running.

Once that was done, login into the server and:

# echo “- – -” > /sys/class/scsi_host/host#/scan

partprobe should also do the trick

Just to see that the new disk is available use:

# fdisk -l

In this case it was /dev/sdb

create a new partition with

# fdisk /dev/sdb

press n and then w

Format the new partition:

# mkfs.ext3 /dev/sdb1

list the volume groups:

# vgs

add new physical volume

# pvcreate /dev/sdb1

extend the default volume group from the vgs command

# vgextend VolGroup /dev/sdb1

check to see pv and vg has another volume with:

# vgs

And list logical volumes

# lvdisplay

extend my / volume by the entire size of /dev/sdb1

# lvextend /dev/VolGroup/lv_root/dev/sdb1

resize filesystem to match vol size increase

# resize2fs /dev/VolGroup/lv_root

(requires a 2.6 kernel to resize while fs running)