Fupp

A few recent changes and improvements

2009-07-23T13:46:00.000-07:00

The last days I have done a few changes, which might not be visible to you, but I'll list it here anyway:

- updated to FreeBSD 7.2-STABLE, so that we have ZFS version 13 instead of version 6, which was the first ZFS version imported to FreeBSD.

- upgraded Apache from 1.3 to 2.2.11 ITK. I also configured KeepAlive with a short timeout (it's faster and cheaper to take more requests per connection), and set a max limit for simultaneous clients on specific virtual hosts where this had been a problem (ITK lets us set this per virtual host and not just for the whole server).

- fixed the serial console to the server. I also use a Real Weasel in use, but it's nice to have a non screen-scraping console free of bugs.

- fixed the startup scripts starting Amavis (the spam prevention system in use) so that they actually work. There were some issues here due to the order things need to happen: 1) get ZFS filesystems mounted, 2) mount devfs filesystems, 3) start the Totem jail, 4) start Amavis. Recently this has made mail deliveries freeze for hours or even days until I discovered everything was not up and running.

- upgraded the Totem "backup" server, which normally just does DNS and MX (receive emails) and gives me the console access to the Totem machine. But this machine could now run the Totem jail if necessary, at least for some time. But it's a little slow (it's an Intel Atom system).

- moving of the Nagios monitoring completed. It was running from a machine I needed to discontinue. In the process I wasn't paying too much attention, which also helped lead do mail delivery freezes. But that is over now.

In other words: things could be faster and more stable, and it'll be easier for me to fix and keep things running well.

Next question up is: do we remove our Squirrelmail webmail, and go for the new one (Roundcube)? I suppose we might need to change IMAP server and/or mailbox format then, due to performance issues. I'll think about it.

Improvements, and a real weasel

2008-09-27T16:33:00.001-07:00

I have doubled the amount of RAM on the server, going from 1 to 2 GB. This should definitely help on the hangs. Also, I have added management again, by using a Real Weasel PCI card. If there is any trouble with the machine now, I should have a fair chance to fix it from home or whereever I am.

I took some pictures of the current server setup. On top you can see the backup server for DNS/MX, which also controls the 4-port USB serial adapter, for management.

Serial cables:

Maintenance today

2008-09-27T04:30:00.001-07:00

I am adding more RAM, and fixing management of the PC so that I can do out-of-band management. This means I can fix it more easily from home/remote location if something is not working.

Update

2008-09-26T04:00:00.000-07:00

Ok, so we still have hangs. The blog neppe.no (which uses a lot of memory for PHP) and ZFS are the culprits. I had to undo some "fixes" to have neppe.no working at all. Until we have neppe.no using less memory, or more memory in the machine, we are going to have some occasional hangs. When the hangs happens, sometimes processes die. Because of this I have extended the monitoring on Totem to also check things like the mail queue length, and that Amavis/Postgrey are up and running. Should help to keep things up and running, and mail flowing.

Getting closer on fixing the hangs

2008-09-23T05:24:00.000-07:00

It seems the recent server hangs are related to rewrite rules in Apache and the generation of one big PHP page several times in parallel. This makes the server use a lot of swap within a short time, and for the users -- hang. I have made some changes to the setup so that it hopefully doesn't happen again.

Server hangs

2008-09-14T01:29:00.000-07:00

Arrgh. This morning it happened again. The whole server freezed from 07:50 to 08:15. I'll be looking into why this is happening. But it might take a little while before I get any results I'm afraid.

Downtime today

2008-09-10T14:37:00.000-07:00

The last weeks we've had some hangs, lasting upto and hour or two, which I still have now been able to track down. It could be related to the fact that we are using ZFS now. In an attempt to avoid the hangs, I upgraded the mother OS (on the physical server) today. Unfortunately it did not come up after a reboot, which happened during a busy day at work. It took some time before I could look at it and also to find out what was wrong. According to Nagios, we were down from 12:46 to 20:52. And yes, I do get an SMS message if the server goes down.