Pieter’s Programming Blog

I bought (together with my brother) a new server. The old one is definitely ready for retirement: 120.000 visits, 1.600.000 pages and 50.000.000 hits (not counting frequent Google crawls, integration with SMS services and Nieuwsblad.be) for pets.be in a month was a bit too much for 1Gb RAM on a hyperthreaded processor which also runs some other websites and now my koopjeszoeker.be site which definitely needs more memory and faster disks.

The investment wasn’t small, but should be worth it: 2 servers, each with 2 quad-core cpu’s and 4GB RAM, all in one unit. I ordered the server on a tuesday morning and could pick it up the same evening. 3 weeks without free time later, the server is ready to be shipped from under my bed (the noise!) to the data center. Ubuntu, Varnish, Apache 2, Tomcat, MySQL, Subversion, CVS, Firehol, … all is installed and (a little bit) tested.

Those dreaded “server busy” messages should be gone soon and koopjeszoeker.be will be ready to go out of beta! (Jay!)

So, if one day I have my new dual quad-core server, what do I install on it? Fedora made maintenance on my current server a bit hard because I had to go through long steps to go from one core to the next every 6 months (and sometimes a trip to Brussels to press the reset button when I messed up).

Ubuntu seems easy to install and has long support for the 6.06 version (till 2011).

On the other hand, CentOS seems reasonable too, since I know of some bigger companies who use it in production. I personally don’t know any companies running Ubuntu (I’m sure there are).

Has anybody any experience with the Ubuntu server version? I already installed it on an old computer at home, which worked ok, but what with multi-core processors?

I’m seriously considering to buy a new server. My current server (Pentium 4, 3 Ghz with 1 Gb RAM) is currently a bit too busy to be healthy.

A double dual quad core (2 servers in 1 unit with each 2 quad-core cpu’s, in total 32 Ghz processing power) may be a little bit overkill (4000 €).

I’m however seriously considering a dual quad core setup with 8 Gb RAM which should be enough to handle the load for the next year(s). I looked up some information to find comparisons between a faster single core cpu and a slower dual or quad core cpu. The conclusion was that for desktops a single faster cpu is sometimes better (because most desktop application are not multi-threaded), but for servers that are mostly multi-process systems you get slower response times but also higher throughput. Since the response times are not really the problem, I think the multiple core setup will be the best choice.

I’m still not sure if I should install Xen for virtualization or not. A benefit would be that I can install MySQL on one virtual server and assign it 4 processors for example. Squid, Apache, Tomcat, Postfix, CVS, … can all get their own virtual instance. But wouldn’t such a virtual-server-per-process setup be a bit hard to maintain?

I’m not sure if virtualization would really give me any benefit, besides the fact that I can isolate some processes (like Postfix and CVS) that shouldn’t be affected when the websites are under heavy load. On the other hand, it seems a bit of a waste to reserve one cpu for these processes that really don’t require so much cpu time.

After 303 days of uptime, I decided to reboot my onthoo.com server. The load was getting a bit too high, especially since my brothers website pets.be was mentioned in some national newspapers (Het Laatste Nieuws and La Meuse).

This was the last uptime message after a long time without any reboots:

16:22:13 up 303 days, 21:43, 1 user, load average: 0.31, 12.81, 57.29

Memory was constantly at 800 MB used, without any significant processes running. After the reboot, it was only 250 MB…

Nevertheless a good uptime for this server, which has to handle quite a load these days.

I still have to figure out what exactly went wrong, since the server didn’t respond to http, ssh nor smtp. After some hours, everything came up like nothing had happened. In the Apache logs I found a lot of OutOfMemory errors, maybe the server was just constantly swapping without any time left for handling connections.

I tweaked some Apache parameters, but apparently this wasn’t enough. If anyone knows of a way how to prevent Apache from taking too much memory, please let me know!

I upgraded my server to Fedora Core 5 with the aid of the excellent guide on http://www.brandonhutchinson.com.

I already used this guide to upgrade from FC2 to FC3 and to FC4. Then I had problems because of an installation problem that I got from the beginning: my /boot directory was differently mapped at boot time then at runtime of the OS (because of the RAID). So although I updated the kernel in the /boot dir, it wasn’t seen at startup and gave compatibility problems (it read the /boot directory from the other disk in the RAID).

The upgrade went fine, but I got a lot of config files that are saved as .rpmnew and that I am now trying to set these correctly again. But hey, if you really want Apache 2.2 and MySQL 5 (like me), it’s normal to expect some work…

Today I got an e-mail from my daily logwatch that I could not process the log files because they were to big. A bit surprised about this sudden load on my server, I took a look at the files and discovered that my mail.log was 300 MB… Apparently a spammer had found a way to abuse my server to send spam.

I tested my server with various sites to see if it was an open relay, but all these tests failed. I used http://abuse.net/relay.html, but this isn’t a complete test. A better one is http://www.ordb.org/submit/, but this one will record your server in its database when your server is an open relay. This database is used by spam filters and mail servers to reject incoming mail.
You can use this ordb.org site if you add the following to postfix main.cf:
smtpd_client_restrictions = reject_rbl_client relays.ordb.org

Since I could not find how the spammer was abusing my server, I blocked the ip responsible for sending the e-mails by following this post: http://www.linuxquestions.org/questions/history/277040

iptables -I INPUT -s 83.217.36.171/255.255.255.255 -j DROP

I also set this parameter in /etc/postfix/main.cf, since I only use my server to send mail from within squirrelmail (= webmail) or from a script in a cgi-bin dir:

mynetworks_style = host

I think the problem is related to a cgi script that is used by one of the sites that I host and that sends e-mail. I renamed the script to make sure this wouldn’t happen again (my apache logs showed 404 errors from spiders that look for all kinds of cgi programs, probably in order to abuse them). I will also investigate how this cgi program can be abused and if there is an update available for it.

The damage isn’t that great, since I think approximately 200.000 messages were put in the mail queue, but I couldn’t find one that wasn’t rejected by the receiving server (my spamassassin filter had already marked them as spam before sending them out).

Anyway, I feel like being robbed…

I finally managed to set my servers clock right automatically with ntpdate. I did this by following this guide. A list of publicly available servers is available here.

Finally, I installed a spam filter on my mail server. I did this by following Integrated Spamd In Postfix.

I needed to create a “.spamassassin” directory in my mail users directory in order to get rid of the warnings in /var/log/maillog.