Fighting spam with SpamAssassin, procmail and greylisting

On my private server we use a number of measures to stop and prevent spam from arriving in the users inboxes:

• postgrey (greylisting) to delay arrival (hopefully block lists will be up to date in time to stop unwanted mail, also some senders do not retry)
• SpamAssasin to block mails by scoring different aspects of the emails. Newer versions of it has URIBL (domain based, for links in the emails) in addtition to the tradional RBL (IP based) block lists. Which works better. I also created my own URIBL block list which you can use, dbl.fupp.net.
• Procmail. For users on my server, I recommend this procmail rule:

  :0
  * ^X-Spam-Status: Yes
  .crapbox/

  It will sort emails that has a score indicating it is spam into mailbox "crapbox".
• blocking unwanted and dangerous attachments, particularly for Windows users.

A few recent changes and improvements

The last days I have done a few changes, which might not be visible to you, but I'll list it here anyway:

- updated to FreeBSD 7.2-STABLE, so that we have ZFS version 13 instead of version 6, which was the first ZFS version imported to FreeBSD.

- upgraded Apache from 1.3 to 2.2.11 ITK. I also configured KeepAlive with a short timeout (it's faster and cheaper to take more requests per connection), and set a max limit for simultaneous clients on specific virtual hosts where this had been a problem (ITK lets us set this per virtual host and not just for the whole server).

- fixed the serial console to the server. I also use a Real Weasel in use, but it's nice to have a non screen-scraping console free of bugs.

- fixed the startup scripts starting Amavis (the spam prevention system in use) so that they actually work. There were some issues here due to the order things need to happen: 1) get ZFS filesystems mounted, 2) mount devfs filesystems, 3) start the Totem jail, 4) start Amavis. Recently this has made mail deliveries freeze for hours or even days until I discovered everything was not up and running.

- upgraded the Totem "backup" server, which normally just does DNS and MX (receive emails) and gives me the console access to the Totem machine. But this machine could now run the Totem jail if necessary, at least for some time. But it's a little slow (it's an Intel Atom system).

- moving of the Nagios monitoring completed. It was running from a machine I needed to discontinue. In the process I wasn't paying too much attention, which also helped lead do mail delivery freezes. But that is over now.

In other words: things could be faster and more stable, and it'll be easier for me to fix and keep things running well.

Next question up is: do we remove our Squirrelmail webmail, and go for the new one (Roundcube)? I suppose we might need to change IMAP server and/or mailbox format then, due to performance issues. I'll think about it.

Improvements, and a real weasel

I have doubled the amount of RAM on the server, going from 1 to 2 GB. This should definitely help on the hangs. Also, I have added management again, by using a Real Weasel PCI card. If there is any trouble with the machine now, I should have a fair chance to fix it from home or whereever I am.






I took some pictures of the current server setup. On top you can see the backup server for DNS/MX, which also controls the 4-port USB serial adapter, for management.








Serial cables:

Maintenance today

I am adding more RAM, and fixing management of the PC so that I can do out-of-band management. This means I can fix it more easily from home/remote location if something is not working.

Update

Ok, so we still have hangs. The blog neppe.no (which uses a lot of memory for PHP) and ZFS are the culprits. I had to undo some "fixes" to have neppe.no working at all. Until we have neppe.no using less memory, or more memory in the machine, we are going to have some occasional hangs. When the hangs happens, sometimes processes die. Because of this I have extended the monitoring on Totem to also check things like the mail queue length, and that Amavis/Postgrey are up and running. Should help to keep things up and running, and mail flowing.

Getting closer on fixing the hangs

It seems the recent server hangs are related to rewrite rules in Apache and the generation of one big PHP page several times in parallel. This makes the server use a lot of swap within a short time, and for the users -- hang. I have made some changes to the setup so that it hopefully doesn't happen again.

Server hangs

Arrgh. This morning it happened again. The whole server freezed from 07:50 to 08:15. I'll be looking into why this is happening. But it might take a little while before I get any results I'm afraid.