Tuesday, 4 September 2012

Anti-Spam Email server

In this post I'll show you how to install an anti-spam smart host relay server, based on Ubuntu 12.04 LTS, that will include:

Postfix w/Bayesian Filtering and Anti-Backscatter (Relay Recipients via look-ahead), Apache2, Mysql, Dnsmasq, MailScanner (Spamassassin, ClamAV, Pyzor, Razor, DCC-Client), Baruwa, SPF Checks, FuzzyOcr, Sanesecurity Signatures, PostGrey, KAM, Scamnailer, FireHOL (Iptables Firewall) and Relay Recipients Script.

Continue reading for the instructions.

Install main packages:

First we need to add Baruwa's repo to /etc/apt/sources.list
deb http://apt.baruwa.org/ubuntu precise main
To use this repo add the gpg key:
wget -O - http://apt.baruwa.org/baruwa-apt-keys.gpg | apt-key add -
Now we'll install some dependencies:
apt-get install binutils cpp fetchmail flex gcc libarchive-zip-perl libc6-dev libcompress-raw-zlib-perl libcompress-bzip2-perl libpcre3 libpopt-dev lynx m4 make ncftp nmap openssl perl perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++ build-essential telnet wget gawk
Use bash as default shell:
dpkg-reconfigure dash
Caching Dnsmasq:
apt-get install dnsmasqvi /etc/dnsmasq.conf
and make Dnsmasq listen on localhost:
Install Mysql:
apt-get install mysql-client mysql-server libdbd-mysql-perl

Install and configure Postfix:

apt-get install postfix postfix-mysql postfix-doc procmail
You will be asked two questions. Answer as follows:

General type of mail configuration: --> Internet Site
System mail name: --> server1.example.com

Stop Postfix
postfix stop
We’ll want to edit Postfix with the below:
vi master.cf
We need to add two items below the pickup service type. The pickup service "picks up" local mail (local meaning "on this machine") and delivers it. This is a way to bypass content filtering for mail generated by this machine.

It should look like this when you are done:
pickup fifo n - - 60 1 pickup -o content_filter= -o receive_override_options=no_header_body_checks
Edit main.cf with a script:
vi /usr/src/postfix.sh
postconf -e "alias_maps = hash:/etc/aliases"
postconf -e "myorigin = domain.tld"postconf -e "myhostname = server1.domain.tld"postconf -e "mynetworks =,"postconf -e "message_size_limit = 10485760"
postconf -e "local_transport = error:No local mail delivery"
postconf -e "mydestination = "
postconf -e "local_recipient_maps = "
postconf -e "relay_domains = mysql:/etc/postfix/mysql-relay_domains.cf"
postconf -e "relay_recipient_maps = mysql:/etc/postfix/mysql-relay_recipients.cf"
postconf -e "transport_maps = mysql:/etc/postfix/mysql-transports.cf"
postconf -e "virtual_alias_maps = hash:/etc/postfix/virtual"
postconf -e "disable_vrfy_command = yes"
postconf -e "strict_rfc821_envelopes = no"
postconf -e "smtpd_banner = $myhostname ESMTP SpamSnake"
postconf -e "smtpd_delay_reject = yes"
postconf -e "smtpd_recipient_limit = 100"
postconf -e "smtpd_helo_required = yes"
postconf -e "smtpd_client_restrictions = permit_sasl_authenticated, permit_mynetworks, permit"
postconf -e "smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, permit"
postconf -e "smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_non_fqdn_sender, reject_unknown_sender_domain, permit"
postconf -e "smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unknown_recipient_domain, reject_unauth_destination, whitelist_policy, grey_policy, rbl_policy, spf_policy, permit"
postconf -e "smtpd_data_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_pipelining"
postconf -e "smtpd_restriction_classes = spf_policy, rbl_policy, grey_policy, whitelist_policy"
postconf -e "spf_policy = check_policy_service unix:private/policy"
postconf -e "rbl_policy = reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net"
postconf -e "grey_policy = check_policy_service unix:private/greyfix"
postconf -e "whitelist_policy = check_client_access mysql:/etc/postfix/mysql-global_whitelist.cf, check_sender_access mysql:/etc/postfix/mysql-global_whitelist.cf"
postconf -e "header_checks = regexp:/etc/postfix/header_checks"
touch /etc/postfix/virtual

echo "root administrator@example.com"
>> /etc/postfix/virtual && echo "abuse administrator@example.com" >> /etc/postfix/virtual && echo "postmaster administrator@example.com" >> /etc/postfix/virtual
postmap /etc/postfix/virtual

touch /etc/postfix/header_checks

echo "/^Received:/ HOLD" >> /etc/postfix/header_checks

postmap /etc/postfix/header_checks

cat > /etc/postfix/mysql-global_whitelist.cf
user = baruwa
password = password
dbname = baruwa
query = select concat('PERMIT') 'action' from lists where from_address='%s' AND list_type='1';
hosts =
cat > /etc/postfix/mysql-relay_domains.cf
user = baruwa
password = password
dbname = baruwa
query = select concat(address, ' ', 'OK') 'domain' from user_addresses where user_addresses.address='%s' and user_addresses.enabled='1';
hosts =
cat > /etc/postfix/mysql-relay_recipients.cf
user = baruwa
password = password
dbname = baruwa
query = select concat('@', address, 'OK') 'email' from user_addresses where user_addresses.address='%d';
hosts =
cat > /etc/postfix/mysql-transports.cf
user = baruwa
password = password
dbname = baruwa
query = select concat('smtp:[', mail_hosts.address, ']', ':', port) 'transport' from mail_hosts, user_addresses where user_addresses.address = '%s' AND user_addresses.id = mail_hosts.useraddress_id;
hosts =
Now make it executable:
chmod +x /usr/src/postfix.sh
and run it using
Postfix Recipient Callout:

This feature queries the recipient server to see if the recipient exists. If not, it replies with a 550 error to the sending server and drops the connection. If the user does exist, the SpamSnake will continue processing the email. This is just another method to prevent backscatter, but comes at a price. Read up on it at http://www.postfix.org/ADDRESS_VERIFICATION_README.html. You can skip this method and use the script method (later on in this guide) if you decide it will bog down your server.
vi /etc/postfix/main.cf
and add the following:
verify_recipient = reject_unknown_recipient_domain, reject_unverified_recipientlook_ahead = check_recipient_access hash:/etc/postfix/accessunverified_recipient_reject_code = 550address_verify_map = btree:/var/lib/postfix/verify
Add this to your smtpd_restriction_classes:
verify_recipient, look_ahead
Add this to smptd_recipient_restrictions:
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, look_ahead, whitelist_policy, grey_policy, rbl_policy, spf_policy, permit
Create the access file:
touch /etc/postfix/access
and add your domains:
domainA.com verify_recipientdomainB.com verify_recipient
Note: Make sure to add valid domains you're filtering for.

Postmap it:
postmap /etc/postfix/access
Final look at the Postfix install:
less /etc/postfix/main.cf
Check the contents of the file for errors and repair if needed. Fire up Postfix:

Start Postfix
postfix start
Check that Postfix responds:
telnet 25
You should see:
220 [yourFQDNhere] ESMTP Postfix (Ubuntu)

Install MailScanner (Apparmor, Clamav, DCC, Pyzor, Razor and Spamassassin):

aptitude install razor pyzor clamav-daemon libclamav6 apparmor mailscanner libltdl7
Apparmor configuration for Clamav

Add clamav to the www-data group so that it can access the directory:
usermod -a -G www-data clamav
Now edit the profile for clamd:
vi /etc/apparmor.d/usr.sbin.clamd
and add the Incoming folder to the list of folders:
/usr/sbin/clamd {
/var/spool/MailScanner/** rw,
/var/spool/MailScanner/incoming/** rw,
Reload apparmor:
/etc/init.d/apparmor reload

DCC 32bit/64bit Configuration:

Install DCC from .deb source, you can just download the necessary packages:
cd /tmp
wget http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-common_1.3.130-0ubuntu1~ppa2~precise1_$(uname
-m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb && dpkg -i dcc-common_1.3.130-0ubuntu1~ppa2~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb
wget http://ppa.launchpad.net/jonasped/ppa/ubuntu/pool/main/d/dcc/dcc-client_1.3.130-0ubuntu1~ppa2~precise1_$(uname
-m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb && dpkg -i dcc-client_1.3.130-0ubuntu1~ppa2~precise1_$(uname -m | sed -e 's/x86_64/amd64/' -e 's/i686/i386/').deb
Or you can add the PPA and install:
add-apt-repository ppa:jonasped/ppaaptitude updateaptitude install  dcc-client dcc-common
Test our installation with:
cdcc info
You should get 'requests ok' from the servers.

Pyzor Configuration
Because pyzor doesn’t work with python2.6 very well, the workaround is to append the following to the first line of /usr/bin/pyzor to make it look like:
#!/usr/bin/python -Wignore::DeprecationWarning
Here we supply the IP address of the Pyzor server to Pyzor. This will create the server's IP address in a servers file therein. Then it will test the connection. If you are behind a firewall, open port 24441/udp in and out to your server. While you're at it also open up 6277/udp for DCC, 2703/tcp for Razor and 783/tcp for SpamAssassin:
mkdir /var/lib/MailScannerpyzor --homedir=/var/lib/MailScanner discover
pyzor ping
Razor Configuration

Create the .razor configuration:
cd && rm /etc/razor/razor-agent.confmkdir /var/lib/MailScanner/.razorrazor-admin -home=/var/lib/MailScanner/.razor -createrazor-admin -home=/var/lib/MailScanner/.razor -discoverrazor-admin -home=/var/lib/MailScanner/.razor -register
Edit the razor agent conf.
vi /var/lib/MailScanner/.razor/razor-agent.conf
And set:
debuglevel = 0razorhome = /var/lib/MailScanner/.razor/
Install dependencies:
apt-get install libconvert-tnef-perl libdbd-sqlite3-perl libfilesys-df-perl libmailtools-perl libmime-tools-perl libmime-perl libnet-cidr-perl libsys-syslog-perl libio-stringy-perl libfile-temp-perl libole-storage-lite-perl libarchive-zip-perl libsys-hostname-long-perl libnet-cidr-lite-perl libhtml-parser-perl libdb-file-lock-perl libnet-dns-perl libncurses5-dev libdigest-hmac-perl libdigest-sha-perl libnet-ip-perl liburi-perl libfile-spec-perl spamassassin libnet-ident-perl libmail-spf-perl libmail-dkim-perl dnsutils libio-socket-ssl-perl gdebi-core
Edit Crontab:
vi /etc/crontab
And add:
37 5 * * * /usr/sbin/update_phishing_sites
&> /dev/null
07 * * * * /usr/sbin/update_bad_phishing_sites
&> /dev/null
#58 23 * * * /usr/sbin/clean.quarantine
&> /dev/null
42 * * * * /usr/sbin/update_virus_scanners
&> /dev/null
3,23,43 * * * * /usr/sbin/check_mailscanner
&> /dev/null
Install SpamAssassin

First we need to disable the default SpamAssassin configuration file:
mv /etc/spamassassin/local.cf/etc/spamassassin/local.cf.disabled
Now let's backup the SpamAssassin configuration file in MailScanner then edit:
cp /etc/MailScanner/spam.assassin.prefs.conf /etc/MailScanner/spam.assassin.prefs.conf.back
SpamAssassin SQL Bayes
Pre-requisities: You'll need the perl-DBI and perl-DBD-MySQL modules installed.

Assumptions and Variables:
SpamAssassin Bayes Database Name: sa_bayesSpamAssassin Bayes Database UserName: sa_userSpamAssassin Bayes Database Password: sa_password
Create the MySQL database on the server where you intend on storing the bayesian information.

mysql -u root -pmysql> create database sa_bayes;mysql> GRANT ALL ON sa_bayes.* TO sa_user@localhost IDENTIFIED BY 'sa_password';mysql> flush privileges;
Import database structure:
mysql -u sa_user -p sa_bayes < /usr/share/doc/spamassassin/sql/bayes_mysql.sql
To enable DCC:
vi /etc/spamassassin/v310.pre
And add:
loadplugin Mail::SpamAssassin::Plugin::DCC
Edit Spam Assassin's conf file:
vi /opt/MailScanner/etc/spam.assassin.prefs.conf
and add the following to the top:

#pyzoruse_pyzor 1pyzor_options --homedir /var/lib/MailScanner/
#razoruse_razor2 1razor_config /var/lib/MailScanner/.razor/razor-agent.confFix DCC path:
dcc_path /usr/bin/dccproc
Update header string:

bayes_ignore_header X-YOURDOMAIN-COM-MailScannerbayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamCheckbayes_ignore_header X-YOURDOMAIN-COM-MailScanner-SpamScorebayes_ignore_header X-YOURDOMAIN-COM-MailScanner-Information#use_auto_whitelist 0

"YOURDOMAIN-COM" should be replaced with whatever you used for "%org-name%" in the MailScanner.conf file. Leave the "X-" in place. This is the same orgname used in the MailScanner.conf above.

Add sql connection string to the bottom:
bayes_store_module Mail::SpamAssassin::BayesStore::SQLbayes_sql_dsn DBI:mysql:sa_bayes:localhostbayes_sql_username sa_userbayes_sql_password sa_passwordbayes_sql_override_username rootvi v310.pre
and comment out domainkeys since DKIM has superseeded it:
#loadplugin Mail::SpamAssassin::Plugin::DomainKeys
Add it to cron:
30 01 * * * /usr/bin/sa-learn --force-expire --sync -p /etc/MailScanner/spam.assassin.prefs.conf
Install Apache and mod-passenger:
aptitude install apache2 libapache2-mod-passenger
Create the following to prevent an error in a lint test:
mkdir /var/www/.spamassassin
Install missing perl packages:
aptitude install libencode-detect-perl libcrypt-openssl-rsa-perlperl -MCPAN -e shellinstall IP::Country::Fastexit
Set permissions to bring it all together:
chown -R postfix:www-data /var/spool/postfix/holdchmod -R ug+rwx /var/spool/postfix/hold
Test out the setup:
spamassassin -x -D -p /opt/MailScanner/etc/spam.assassin.prefs.conf --lint
Check for lines like:
debug: bayes: Database connection establisheddebug: bayes: found bayes db version 3debug: bayes: Using userid: 2
You should see lines come up with DCC, Pyzor and Razor that say loading plugin and hopefully no errors.

MailScanner Configuration:

We need to make a directory for SpamAssassin in the spool and give postfix permissions to it, if you run sa-learn --force as root, bayes databese that is stored in these directories will change to root:root and spamassassin will error looking at the db. Just keep an eye on the mail.log and you'll remember to change the permissions back. Also disable the MailScanner default configs:
mkdir /var/spool/MailScanner/spamassassin
Backup your MailScanner.conf file:
cp /etc/MailScanner/MailScanner.conf /etc/MailScanner/MailScanner.conf.dist
Change the following parameters in MailScanner.conf with the following script:
vi /usr/src/mailscanner.sh
sed -i "/^%org-name% =/ c\%org-name% =orgname" /etc/MailScanner/MailScanner.conf
sed -i "/^%org-long-name% =/ c\%org-long-name% = longorgname" /etc/MailScanner/MailScanner.conf
sed -i "/^%web-site% =/ c\%web-site% = www.domain.tld" /etc/MailScanner/MailScanner.conf
sed -i "/^Run As User =/ c\Run As User = postfix" /etc/MailScanner/MailScanner.conf
sed -i "/^Run As Group =/ c\Run As Group = www-data" /etc/MailScanner/MailScanner.conf
sed -i "/^Incoming Work Group =/ c\Incoming Work Group = clamav" /etc/MailScanner/MailScanner.conf
sed -i "/^Incoming Work Permissions =/ c\Incoming Work Permissions = 0640" /etc/MailScanner/MailScanner.conf
sed -i "/^Incoming Queue Dir =/ c\Incoming Queue Dir = /var/spool/postfix/hold" /etc/MailScanner/MailScanner.conf
sed -i "/^Outgoing Queue Dir =/ c\Outgoing Queue Dir = /var/spool/postfix/incoming" /etc/MailScanner/MailScanner.conf
sed -i "/^MTA =/ c\MTA = postfix" /etc/MailScanner/MailScanner.conf
sed -i "/^Quarantine User =/ c\Quarantine User = root" /etc/MailScanner/MailScanner.conf
sed -i "/^Quarantine Group =/ c\Quarantine Group = www-data" /etc/MailScanner/MailScanner.conf
sed -i "/^Quarantine Permissions =/ c\Quarantine Permissions = 0660" /etc/MailScanner/MailScanner.conf
sed -i "/^Quarantine Whole Message =/ c\Quarantine Whole Message = yes" /etc/MailScanner/MailScanner.conf
sed -i "/^Virus Scanners =/ c\Virus Scanners = clamd" /etc/MailScanner/MailScanner.conf
sed -i "/^Monitors for ClamAV Updates =/ c\Monitors for ClamAV Updates = /var/lib/clamav/*.cld /var/lib/clamav/*.cvd" /etc/MailScanner/MailScanner.conf
sed -i "/^Clamd Socket =/ c\Clamd Socket = /var/run/clamav/clamd.ctl" /etc/MailScanner/MailScanner.conf
sed -i "/^Clamd Lock File =/ c\Clamd Lock File = /var/run/clamav/clamd.pid" /etc/MailScanner/MailScanner.conf
sed -i "/^Spam Subject Text =/ c\Spam Subject Text = ***SPAM***" /etc/MailScanner/MailScanner.conf
sed -i "/^Spam Actions =/ c\Spam Actions = deliver store" /etc/MailScanner/MailScanner.conf
sed -i "/^High Scoring Spam Actions =/ c\High Scoring Spam Actions = store delete" /etc/MailScanner/MailScanner.conf
sed -i "/^Non Spam Actions =/ c\Non Spam Actions = deliver store" /etc/MailScanner/MailScanner.conf
sed -i "/^SpamAssassin User State Dir =/ c\SpamAssassin User State Dir = /var/spool/MailScanner/spamassassin" /etc/MailScanner/MailScanner.conf
Make the script executable:
chmod +x mailscanner.sh
and run using
Edit /etc/default/mailscanner. There you will have to set the variable run_mailscanner to 1.

Install and configure Saslauthd:

aptitude install sasl2-bin libsasl2-modules-ldap
edit the configuration:
vi /etc/default/saslauthd

Set MECH_OPTIONS to the ip address of your imap server

And at the bottom change:
OPTIONS="-c -m /var/run/saslauthd"
OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd -r"
Set the correct permissions:
chown postfix /var/spool/postfix/var/run/saslauthdchgrp sasl /var/spool/postfix/var/run/saslauthdchmod 710 /var/spool/postfix/var/run/saslauthd
Start the system:
/etc/init.d/saslauthd start
/etc/init.d/mailscanner start
Check your logs for errors:
tail -f /var/log/mail.log
Check your mail.log (tail –f /var/log/mail.log) and you should see the following:
Jun 13 12:18:23 hoshi MailScanner[26388]: MailScanner E-Mail Virus Scanner version 4.81.4 starting...

Install Baruwa:

First you need to install rabbitmq-server
aptitude install rabbitmq-server
And configure it:
rabbitmqctl add_user baruwa passwordrabbitmqctl add_vhost baruwarabbitmqctl set_permissions -p baruwa baruwa ".*" ".*" ".*"/etc/init.d/rabbitmq-server restart
Now install Baruwa:
aptitude install baruwa
You'll be prompted to setup the baruwa db for logging, including an access username and password, which will be used by the Baruwa scripts. You'll also be prompted to setup the admin user information for the Baruwa frontend.

Edit Baruwa's setings:
vi /usr/share/pyshared/baruwa/settings.py
Change the Quarantine hosturl to your liking:
QUARANTINE_REPORT_HOSTURL = 'http://baruwa-alpha.local'
Setup the Baruwa DB:
baruwa-admin syncdb --noinputbaruwa-admin migratebaruwa-admin syncdb --noinputbaruwa-admin migrate
Set MailScanner to use Baruwa's DB:
vi /etc/MailScanner/MailScanner.conf
You need to make sure that the following options are set:
Always Looked Up Last = &BaruwaSQLIs Definitely Not Spam = &BaruwaWhitelistIs Definitely Spam = &BaruwaBlacklistRequired SpamAssassin Score = &BaruwaLowScoreHigh SpamAssassin Score = &BaruwaHighScore
Enable Baruwa in Apache
a2dissite 000-defaultservice apache2 reload
Restart MailScanner:
/etc/init.d/mailscanner restart
Point your browser to http://hostname_used login with admin user and password and start working. You can now use the interface to add users and process messages, etc.

If you can't login use:
baruwa-admin createsuperuser
to create an initial admin user.

Setup Instructions for Baruwa (on the web insterface):
  • Log into Baruwa as admin and navigate to:
    • --> Settings --> Accounts --> Create Account
  • Once you've created the user account, you'll get two new tabs on that page, Profile Settings and Associated Addresses.
  • Fill out Profile Settings choosing Domain Admin and set a low score of 6 and a high score of 9 and check scan email.
  • Click the + sign under Associated Addresses and enter a domains for which the user is the admin of eg. domain.com. domain.com will show up under Associated Addresses.
  • Click on domain.com and you'll be taken into Domain Information, where you'll be able to setup SMTP delivery information. Go ahead and add the receiving smtp server, or the ip of the receiving smtp server. Select enable and if you use a non-standard port, set it, otherwise use 25. Once you've done that, you can click on the test button next to the pencil, to see if your receiving server will accept the connection.
Now, you can log out as admin, and log in as the user you just setup and mails should start flowing.

The relay_recipients, relay_domains and transports settings in /etc/postfix/main.cf will use the entries you've provided in Baruwa. Therefore, no hash file is required.
The associated queries in the mysql cf files will pull the result in the proper format and feed it to postfix for use.

However, if you'd like to use a hash for any of your config files, use the following as an example of how to setup /etc/postfix/main.cf:
relay_recipient_maps = hash:/etc/postfix/relay_recipients
Of course, you would have to create the hash file(s), populate it and postmap it for postfix to use.

Note: If you do end up using hash for relay_recipients for specific domains, you'll have to remove that domain from /etc/postfix/access. All other domain users can still be verified using look_ahead

Also, if you need to do mx lookups, you'll have to edit /etc/postfix/mysql-transports.cf query to look like:
concat('smtp:', mail_hosts.address, ':', port) 'transport'The [ and ] were removed to allow MX lookups.

Install and configure SPF:

aptitude install postfix-policyd-spf-perl
Then we edit /etc/postfix/master.cf
vi /etc/postfix/master.cf
and add the following stanza at the end:
policy unix - n n - - spawn user=nobody argv=/usr/sbin/postfix-policyd-spf-perl
Then restart Postfix:
/etc/init.d/postfix restart

Install and Configure FuzzyOcr:

FuzzyOCR has some prerequisites like ocrad and gocr that we can install like this:
apt-get install fuzzyocr3 netpbm gifsicle libungif-bin gocr ocrad libstring-approx-perl libmldbm-sync-perl libdigest-md5-perl libdbd-mysql-perl imagemagick tesseract-ocr
So FuzzyOCR is now installed, now we need to configure it. FuzzyOCR's configuration file is /etc/spamassassin/FuzzyOcr.cf.
In that file almost everything is commented out. We open that file now and make some modifications:
vi /etc/spamassassin/FuzzyOcr.cf
Uncomment the following lines:
focr_global_wordlist /etc/spamassassin/FuzzyOcr.wordsfocr_preprocessor_file /etc/spamassassin/FuzzyOcr.prepsfocr_scanset_file /etc/spamassassin/FuzzyOcr.scansets<focr_enable_image_hashing 3focr_digest_db /etc/spamassassin/FuzzyOcr.hashdbfocr_db_hash /etc/spamassassin/FuzzyOcr.dbfocr_db_safe /etc/spamassassin/FuzzyOcr.safe.dbfocr_bin_helper convert, tesseract
Comment out the path:
#focr_path_bin /usr/local/netpbm/bin:/usr/local/bin:/usr/bin
We will be storing the image hashes in a mysql database to improve on performance such that images that we have already scanned do not get scanned again as OCR is a resource intense activity.
Create MySQL Database:

The sql script creates the database for fuzzyocr:
vi FuzzyOcr.mysql
CREATE DATABASE FuzzyOcr;use FuzzyOcr;
CREATE TABLE `Hash` ( `key` varchar(255) NOT NULL default '', `basic` varchar(64) NOT NULL default '', `fname` varchar(160) NOT NULL default '', `ctype` varchar(64) NOT NULL default '', `ftype` smallint(5) unsigned NOT NULL default '0', `match` int(10) unsigned NOT NULL default '0', `input` int(10) unsigned NOT NULL default '0', `check` int(10) unsigned NOT NULL default '0', `score` decimal(8,3) NOT NULL default '0.000', `dinfo` text NOT NULL, PRIMARY KEY (`key`) );
CREATE TABLE `Safe` ( `key` varchar(255) NOT NULL default '', `basic` varchar(64) NOT NULL default '', `fname` varchar(160) NOT NULL default '', `ctype` varchar(64) NOT NULL default '', `ftype` smallint(5) unsigned NOT NULL default '0', `match` int(10) unsigned NOT NULL default '0', `input` int(10) unsigned NOT NULL default '0', `check` int(10) unsigned NOT NULL default '0', `score` decimal(8,3) NOT NULL default '0.000', `dinfo` text NOT NULL, PRIMARY KEY (`key`) );
GRANT SELECT,UPDATE,INSERT,DELETE ON FuzzyOcr.* TO fuzzyocr@localhost IDENTIFIED BY 'fuzzyocr';
Create the DB:
mysql -p < FuzzyOcr.mysql
Edit the configuration file:
vi /etc/spamassassin/FuzzyOcr.cf
Enable the following lines:
focr_mysql_db FuzzyOcrfocr_mysql_hash Hashfocr_mysql_safe Safefocr_mysql_user fuzzyocrfocr_mysql_pass fuzzyocrfocr_mysql_host localhostfocr_mysql_port 3306focr_mysql_socket /var/run/mysqld/mysqld.sock
Setup FuzzyOcr Database Cleaner script:
vi /usr/sbin/fuzzy-cleanmysql

#Script to clean out mysql tables of data. Default is to leave data in Safe for 1 day and Hash for 10 days.
use Getopt::Long;
use DBI;
use MLDBM qw(DB_File Storable);
my %Files = (
db_hash => '/var/lib/fuzzyocr/FuzzyOcr.db',
db_safe => '/var/lib/fuzzyocr/FuzzyOcr.safe.db',
use DBI;
$database = "FuzzyOcr";
$hostname = "localhost";
$socket = "/var/run/mysqld/mysqld.sock";
$port = "3306";
$username = "fuzzyocr";
$password = 'password';
# defaults
my $cfgfile = "/etc/spamassassin/FuzzyOcr.cf";
my %App;
my %age;
$age{'age'} = 10*24; # 10 days
$age{'hash'} = $age{'age'};
$age{'safe'} = 0;
my $help = 0;
my $verbose = 0;
GetOptions( \%age,
'config=s' => \$cfgfile,
'help' => \$help,
'verbose' => \$verbose,
if ($help) {
print "Usage: fuzzy-cleanmysql [Options]\n";
print "\n";
print "Available options:\n";
print "--age=i Global age in hours to keep in db\n";
print "--config=s Specify location of FuzzyOcr.cf\n";
print " Default: /etc/spamassassin/FuzzyOcr.cf\n";
print "--hash=i Number of hours old to keep in Hash db\n";
print "--safe=i Number of hours old to keep in Safe db\n";
print "--verbose Show more informations\n";
print "\n";
exit 1;
# Convert hours to seconds
$age{'age'} *= 60 * 60;
$age{'hash'} *= 60 * 60;
$age{'safe'} *= 60 * 60;
$age{'safe'} = $age{'safe'} ? $age{'safe'} : $age{'age'};
# Read custom paths from FuzzyOcr.cf
my $app_path = q(/usr/local/netpbm/bin:/usr/local/bin:/usr/bin);
open CONFIG, "< $cfgfile" or warn "Can't read configuration file, using defaults...\n";
while () {
if ($_ =~ m/^focr_bin_(\w+) (.+)/) {
$App{$1} = $2;
printf "Found custom path \"$2\" for application \"$1\"\n" if $verbose;
if ($_ =~ m/^focr_path_bin (.+)/) {
$app_path = $1;
printf "Found new path: \"$1\"\n" if $verbose;
if ($_ =~ m/^focr_enable_image_hashing (\d)/) {
$App{hashing_type} = $1;
printf "Found DB Hashing\n" if ($verbose and $1 == 2);
printf "Found MySQL Hashing\n" if ($verbose and $1 == 3);
if ($_ =~ m/^focr_mysql_(\w+) (.+)/) {
$MySQL{$1} = $2;
printf "Found MySQL option $1 => '$2'\n" if $verbose;
if ($_ =~ m/^focr_threshold_max_hash (.+)/) {
$App{max_hash} = $1;
printf "Updated Thresold{max_hash} = $1\n" if $verbose;
close CONFIG;
# make shure we have this threshold set
$App{max_hash} = 5 unless defined $App{max_hash};
# search path for bin_util unless already specified in configuration file
foreach my $app (@bin_utils) {
next if defined $App{$app};
foreach my $d (split(':',$app_path)) {
if (-x "$d/$app") {
$App{$app} = "$d/$app";
sub get_ddb {
my %dopts = ( AutoCommit => 1 );
my $dsn = "DBI:mysql:database=$database";
if (defined $socket) {
$dsn .= ";mysql_socket=$socket";
} else {
$dsn .= ";host=$hostname";
$dns .= ";port=$port" unless $port == 3306;
printf "Connecting to: $dsn\n" if $verbose;
return DBI->connect($dsn, $username, $password,\%dopts) or die("Could not connect!");
if ($App{hashing_type} == 3) {
my $ddb = get_ddb();
if ($ddb) {
my $sql;
foreach my $ff (sort keys %Files) {
$ff =~ s/db_//;
$sqlbase = "FROM $MySQL{$ff} WHERE $MySQL{$ff}.\`check\` < ?";
my $timestamp = time;
$timestamp = $timestamp - $age{$ff};
$sql = "DELETE $sqlbase";
if ( $verbose ) {
printf "Delete from Table $MySQL{$ff}\n";
print "$sql, $timestamp\n";
print "Timestamp is ", scalar(localtime($timestamp)), "\n";
print "That's $age{$ff} seconds earlier than now.\n";
print "\n";

Make the script executable:
chmod +x /usr/sbin/fuzzy-cleanmysql
And add it to cron
vi /etc/crontab
@weekly /usr/sbin/fuzzy-cleanmysql &> /dev/null #FuzzyOcr DB cleaner
Filtering PDF, XLS and Phishing Spam with ClamAV (Sanesecurity Signatures):
There is currently a lot of spam where the spam "information" is attached as .pdf or .xls files, sometimes also hidden inside a .zip file. While these spam mails are not easy to catch with e.g. SpamAssassin or a Bayes filter, the ClamAV virus scanner can catch them easily when it is fed with the correct signatures as ClamAV is built to scan mail attachments.
Install it with:
aptitude install clamav-unofficial-sigs
Edit it's configuration file:
vi /usr/share/clamav-unofficial-sigs/conf.d/00-clamav-unofficial-sigs.conf
Change Path to clamd.pid:
Reload after update:
Work Directory:
And once you're done with the configuration, set the following to yes:
Now we run the update script to check if the download works:
Add it to cron:
00 04 * * * root /usr/sbin/clamav-unofficial-sigs &> /dev/null

Grey listing with Postgrey:

apt-get install postgrey libnet-rblclient-perl
Enable it in postfix:
postconf -e "grey_policy = check_policy_service inet:"
Reload postfix configuration:
sudo /etc/init.d/postfix reload
You may need to modify the whitelisting of certain domains or users. To do so, use the files in /etc/postgrey/.

The default delay is 300 seconds (5 minutes), to change this value edit /etc/default/postgrey. Add --delay=N to the POSTGREY_OPTS:
POSTGREY_OPTS="--inet=10023 --delay=60"


Create the script:
vi /etc/cron.daily/kam.sh
Add the following content:


 # Original version modified by Andrew MacLachlan (andrew@gdcon.net)
 # Added additional MailScanner restarts on inital restart failure
 # Made script run silently for normal (successful) operation
 # Increased UPDATEMAXDELAY to 900 from 600
 # Insert a random delay up to this value, to spread virus updates round
 # the clock. 1800 seconds = 30 minutes.
 # Set this to 0 to disable it.
 if [ -f /opt/MailScanner/var/MailScanner ] ; then
 . /opt/MailScanner/var/MailScanner
 if [ "x$UPDATEMAXDELAY" = "x0" ]; then
 logger -p mail.info -t KAM.cf.sh Delaying cron job up to $UPDATEMAXDELAY seconds
 perl -e "sleep int(rand($UPDATEMAXDELAY));"
 # JKF Fetch KAM.cf
 #echo Fetching KAM.cf...
 cd /etc/mail/spamassassin
 rm -f KAM.cf
 wget -O KAM.cf http://www.peregrinehw.com/downloads/SpamAssassin/contrib/KAM.cf > /dev/null 2>&1
 if [ "$?" = "0" ]; then
 #echo It completed and fetched something
 if ( tail -10 KAM.cf | grep -q '^#.*EOF' ); then
 # echo It succeeded so make a backup
 cp -f KAM.cf KAM.cf.backup
 echo ERROR: Could not find EOF marker
 cp -f KAM.cf.backup KAM.cf
 echo It failed to complete properly
 cp -f KAM.cf.backup KAM.cf
 #echo Reloading MailScanner and SpamAssassin configuration rules
 /etc/init.d/mailscanner reload > /dev/null 2>&1
 if [ $? != 0 ] ; then
 echo "MailScanner reload failed - Retrying..."
 /etc/init.d/mailscanner force-reload
 if [ $? = 0 ] ; then
 echo "MailScanner reload succeeded."
 echo "Stopping MailScanner..."
 /etc/init.d/mailscanner stop
 echo "Waiting for a minute..."
 perl -e "sleep 60;"
 echo "Attemping to start MailScanner..."
 /etc/init.d/mailscanner start
And make it executable:
chmod +x /etc/cron.daily/kam.sh


Create the script:
vi /usr/sbin/update_scamnailer
Note: Content is in https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B9cN15Q3pKnwNmU2NTY2ZDQtZTdiNS00ZjM4LWI0MDktM2ZhNGYzOTY0NTZk&authkey=CKOFkqIJ&hl=en

and make it executable:
chmod +x /usr/sbin/update_scamnailer
Add it to cron:
@daily /usr/sbin/update_scamnailer &> /dev/null #Update Scamnailer

Firewalling the SpamSnake with Firehol:

Firehol is a stateful iptables packet filtering firewall configurator. It is abstracted, extensible, easy and powerful. It can handle any kind of firewall, but most importantly, it gives you the means to configure it, the same way you think of it.

Install Firehol
apt-get install firehol
Firehol Settings:
vi /etc/default/firehol
and change the following:
START_FIREHOL=YESvi /etc/firehol/firehol.conf
and add the following:
server_zabbix_ports="tcp/10050 udp/10050"client_zabbix_ports="default 10050"
version 5 # Accept all client traffic on any interface interface any internet protection strong server "icmp ping ICMP ssh http https telnet webmin dns dcc snmp smtp smtps submission zabbix mysql" accept client all accept
Restart firehol:
service fireholl restart
And that's it

Possibly Related Posts

No comments:

Post a Comment