How to lower the load average on a server with more than 50% in 10 seconds

$uptime
... load average: 0.54, 0.40, 0.36

$sudo vim /etc/fstab
(add the noatime option)

# /dev/sda3
UUID=8623d9e3... / ext3 defaults,errors=remount-ro,noatime 0 1

$sudo mount -a
...
$uptime
... load average: 0.24, 0.16, 0.17

So far this completely unscientific proof.

More info about the noatime option.

Blocking bad bots

Today I blocked some bad bots that were spidering some of my sites. Most notably Custo, which downloads your entire site.

An interesting solution is posted here (I used the mod_rewrite option). You can test this by changing your user agent in Firefox.

This guy seems to be following bad bots.

I added Java, Nutch, Jakarta, Vagabondo and an empty bot name to the list of bad bots.

Ubuntu install²

Below a write-down of the steps I performed to install 2 ubuntu 7.10 servers (webserver + mysql database server) with IPMI.

Reminders:
start services with
- /etc/init.d/mysql start (or stop)
- /etc/init.d/apache2 start (or stop or reload or force-reload)
- Connect to database server: mysql -u root -p --host

Not done:
- SSL on Apache
- DenyHosts (prevent ssh dictionary attacks): see http://packages.ubuntu.com/edgy/net/denyhosts and http://denyhosts.sourceforge.net/

Steps
=====

OS installation
- Change router so desktop gets ip address in 192.168.111.x range
- Connect to ipmi (192.168.111.111) and change ip to 192.168.1.200. Mount ubuntu iso by opening console in browser (http://192.168.1.200) and click on the diskette icon in the right top and choose "mount iso".
- Go to console and reboot.
- Follow instruction steps for Ubuntu (check keyboard: Belgian keyboard gives problems with IPMI, but virtual keyboard and ssh work ok.).
- Remove virtual drive (iso) and reboot.

apt-get
- Remove (comment) line in /etc/apt/sources.list with a dependency on the cd-rom

ntpd
- sudo apt-get install ntp
- sudo vi /etc/ntp.conf; add
server be.pool.ntp.org
server pool.ntp.org

Apache 2
- /etc/init.d/apache2 start (or stop or reload or force-reload)
- sudo a2enmod deflate
- sudo a2enmod headers
- sudo a2enmod expires
- set up name-based virtual hosting following http://httpd.apache.org/docs/2.0/vhosts/name-based.html

PHP
- Copy contents from http://cvs.php.net/viewvc.cgi/php-src/php.ini-recommended?revision=1.179.2.11.2.23.2.1 to php.ini and set variables as needed
- sudo apt-get install php5-gd

eAccelerator
- sudo apt-get update
- sudo apt-get install php5-dev (needed for phpize5)
- Install eaccelerator following the guide from http://eaccelerator.net/wiki/InstallFromSource and http://2bits.com/articles/installing-eaccelerator-0-9-5-1-on-ubuntu-feisty-7-04.html
- Don't forget:
- mkdir /var/cache/eaccelerator
- chmod 644 /var/cache/eaccelerator
- the login credentials for control.php are in the file itself (admin/eAccelerator): change them!

Varnish
- sudo apt-get install varnish --> error in dpkg (see http://ubuntuforums.org/archive/index.php/t-438794.html)
- sudo apt-get install libc6-dev
- sudo apt-get remove varnish
- sudo apt-get install varnish
- Management: telnet 127.0.0.1 6082 (see http://varnish.projects.linpro.no/wiki/ManagementPort) (exit with ctrl+] and quit)
- varnishstat shows statistics
- Varnish by default listens on http://127.0.0.1:6081/
- sudo vi /etc/default/varnish: change 6081 to 80
- sudo vi /etc/apache2/ports.conf: change 80 to 81
- sudo vi /etc/varnish/vcl.conf
- varnish vcl docs: "man vcl"
- Test response headers with GET -ed http://www.example.com
- Show varnish logs in apache format: sudo varnishncsa

MySQL
- /etc/init.d/mysql start (or stop)
- Connect to database server: mysql -u root -p --host onthoo2
- MySQL data is located under /var/lib/mysql/
- Change mysql root password:
mysql> USE mysql;
mysql> UPDATE user SET Password=PASSWORD('new-password') WHERE user='root';
mysql> FLUSH PRIVILEGES;

CVS
- sudo apt-get install cvs

SVN
- sudo apt-get install subversion
- pipe through varnish

Java
- sudo apt-get install sun-java6-jdk
- See https://help.ubuntu.com/community/Java

Tomcat
- Disable port 8080, enable AJP on port 8009
- Change proxy allow in apache2:
<Proxy *>
AddDefaultCharset off
Order deny,allow
#Deny from all
Allow from webserver
</Proxy>

ImageMagick (compile from source for newer version 6.3.7)
- Download sources from ImageMagick website
- sudo apt-get install libjpeg62
- sudo apt-get install libjpeg62-dev
- sudo ldconfig
- cd ImageMagick-6.3.7
- sudo ./configure --prefix=/usr
- sudo make
- sudo make install

Google sitemap generator
- See https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html
- Modified log path from /var/log/httpd/ to /var/log/apache2/
- Added crontabs to /etc/cron.daily
- Changed python2.4 to python2.5 in crontabs

AWStats
- sudo apt-get install awstats
- sudo apt-get install libgeo-ipfree-perl
- vi awstats.conf.local:
LogFormat=1
AllowToUpdateStatsFromBrowser=1
AllowFullYearView=3
- vi awstats.www.pets.be.conf:
LogFile="/var/log/apache2/www.pets.be/access.log"
SiteDomain="www.pets.be"
HostAliases="localhost 127.0.0.1 dierenasielen.be dierenasiel.be refugespouranimaux.be pups.be asiel.org"
- sudo /usr/lib/cgi-bin/awstats.pl -config=www.pets.be -update

Exim (mail sender)
- See https://help.ubuntu.com/7.10/server/C/exim4.html
- Don't forget to run 'sudo update-exim4.conf' when the wizard has finished

Logwatch
- sudo apt-get install logwatch
- default conf is in /usr/share/logwatch/
- add /etc/logwatch/conf/logwatch.conf:
# Default person to mail reports to. Can be a local account or a
# complete email address.
MailTo = logwatch@example.com
# Default person to mail reports from. Can be a local account or a
# complete email address.
MailFrom = root@example.com

Firehol (firewall) on both servers
- sudo apt-get install firehol
- sudo vi /etc/firehol/firehol.conf:
webserver:

         version 5
         interface eth0 internet
         client all accept
         server http accept
         server ssh accept

database server:

          version 5
          clients="webserver"
          interface eth+ internet
                client all accept
                server mysql accept src "$clients"
                server ssh accept

- Patch for iptables warnings (see https://bugs.launchpad.net/ubuntu/+source/firehol/+bug/78017):
- sudo sed 's/%q/%b/g' /lib/firehol/firehol > TMPFILE && sudo mv TMPFILE /lib/firehol/firehol
- sudo chmod 744 /lib/firehol/firehol
- sudo firehol start
- Check with "sudo iptables -L"

IPMI
- change admin username and password
- require https
- use Java Sun plugin
- set invalid login retries and retry timout
- setup ipmitool if you need to change the ip address on which ipmi listens (see https://help.ubuntu.com/community/IPMI):
- sudo apt-get install ipmitool
- cd /usr/share/ipmitool
- sudo cp ipmi.init.basic ipmi.init.ubuntu
- sudo vi ipmi.init.ubuntu (and remove the if loop around the 'modprobe ipmi_si # try new module name' line) - see https://bugs.launchpad.net/ubuntu/+source/ipmitool/+bug/110992
- change ip address: sudo ipmitool -I open lan set 1 ipaddr 85.158.x.x
- change default gateway: sudo ipmitool -I open lan set 1 defgw ipaddr 85.158.x.x
- change netmask sudo ipmitool -I open lan set 1 netmask 255.255.255.x
- browse to http(s)://85.158.x.x

Backup
- See http://ubuntuforums.org/showthread.php?t=35087
- Complete backup of the system:
vi /var/backups/fullimage:
tar cvpzf /backup/fullimage.tgz --exclude=/proc --exclude=/lost+found
--exclude=/backup/ --exclude=/mnt --exclude=/sys /
- Restore backup with "tar xvpfz /backup/fullimage.tgz -C /"
- Special backup scripts in /var/backups
- crontab -e:
0 3 * * * /var/backups/backup-all

New server almost complete

I bought (together with my brother) a new server. The old one is definitely ready for retirement: 120.000 visits, 1.600.000 pages and 50.000.000 hits (not counting frequent Google crawls, integration with SMS services and Nieuwsblad.be) for pets.be in a month was a bit too much for 1Gb RAM on a hyperthreaded processor which also runs some other websites and now my koopjeszoeker.be site which definitely needs more memory and faster disks.

The investment wasn't small, but should be worth it: 2 servers, each with 2 quad-core cpu's and 4GB RAM, all in one unit. I ordered the server on a tuesday morning and could pick it up the same evening. 3 weeks without free time later, the server is ready to be shipped from under my bed (the noise!) to the data center. Ubuntu, Varnish, Apache 2, Tomcat, MySQL, Subversion, CVS, Firehol, ... all is installed and (a little bit) tested.

Those dreaded "server busy" messages should be gone soon and koopjeszoeker.be will be ready to go out of beta! (Jay!)

Ubuntu or CentOS or ...

So, if one day I have my new dual quad-core server, what do I install on it? Fedora made maintenance on my current server a bit hard because I had to go through long steps to go from one core to the next every 6 months (and sometimes a trip to Brussels to press the reset button when I messed up).

Ubuntu seems easy to install and has long support for the 6.06 version (till 2011).

On the other hand, CentOS seems reasonable too, since I know of some bigger companies who use it in production. I personally don't know any companies running Ubuntu (I'm sure there are).

Has anybody any experience with the Ubuntu server version? I already installed it on an old computer at home, which worked ok, but what with multi-core processors?

What type of new server should I choose?

I'm seriously considering to buy a new server. My current server (Pentium 4, 3 Ghz with 1 Gb RAM) is currently a bit too busy to be healthy.

A double dual quad core (2 servers in 1 unit with each 2 quad-core cpu's, in total 32 Ghz processing power) may be a little bit overkill (4000 €).

I'm however seriously considering a dual quad core setup with 8 Gb RAM which should be enough to handle the load for the next year(s). I looked up some information to find comparisons between a faster single core cpu and a slower dual or quad core cpu. The conclusion was that for desktops a single faster cpu is sometimes better (because most desktop application are not multi-threaded), but for servers that are mostly multi-process systems you get slower response times but also higher throughput. Since the response times are not really the problem, I think the multiple core setup will be the best choice.

I'm still not sure if I should install Xen for virtualization or not. A benefit would be that I can install MySQL on one virtual server and assign it 4 processors for example. Squid, Apache, Tomcat, Postfix, CVS, ... can all get their own virtual instance. But wouldn't such a virtual-server-per-process setup be a bit hard to maintain?

I'm not sure if virtualization would really give me any benefit, besides the fact that I can isolate some processes (like Postfix and CVS) that shouldn't be affected when the websites are under heavy load. On the other hand, it seems a bit of a waste to reserve one cpu for these processes that really don't require so much cpu time.

Website hosting

After 303 days of uptime, I decided to reboot my onthoo.com server. The load was getting a bit too high, especially since my brothers website pets.be was mentioned in some national newspapers (Het Laatste Nieuws and La Meuse).

This was the last uptime message after a long time without any reboots:

16:22:13 up 303 days, 21:43, 1 user, load average: 0.31, 12.81, 57.29

Memory was constantly at 800 MB used, without any significant processes running. After the reboot, it was only 250 MB...

Nevertheless a good uptime for this server, which has to handle quite a load these days.

I still have to figure out what exactly went wrong, since the server didn't respond to http, ssh nor smtp. After some hours, everything came up like nothing had happened. In the Apache logs I found a lot of OutOfMemory errors, maybe the server was just constantly swapping without any time left for handling connections.

I tweaked some Apache parameters, but apparently this wasn't enough. If anyone knows of a way how to prevent Apache from taking too much memory, please let me know!

Fedora Core 5

I upgraded my server to Fedora Core 5 with the aid of the excellent guide on http://www.brandonhutchinson.com.

I already used this guide to upgrade from FC2 to FC3 and to FC4. Then I had problems because of an installation problem that I got from the beginning: my /boot directory was differently mapped at boot time then at runtime of the OS (because of the RAID). So although I updated the kernel in the /boot dir, it wasn't seen at startup and gave compatibility problems (it read the /boot directory from the other disk in the RAID).

The upgrade went fine, but I got a lot of config files that are saved as .rpmnew and that I am now trying to set these correctly again. But hey, if you really want Apache 2.2 and MySQL 5 (like me), it's normal to expect some work...

Open relay

Today I got an e-mail from my daily logwatch that I could not process the log files because they were to big. A bit surprised about this sudden load on my server, I took a look at the files and discovered that my mail.log was 300 MB... Apparently a spammer had found a way to abuse my server to send spam.

I tested my server with various sites to see if it was an open relay, but all these tests failed. I used http://abuse.net/relay.html, but this isn't a complete test. A better one is http://www.ordb.org/submit/, but this one will record your server in its database when your server is an open relay. This database is used by spam filters and mail servers to reject incoming mail.
You can use this ordb.org site if you add the following to postfix main.cf:
smtpd_client_restrictions = reject_rbl_client relays.ordb.org

Since I could not find how the spammer was abusing my server, I blocked the ip responsible for sending the e-mails by following this post: http://www.linuxquestions.org/questions/history/277040

iptables -I INPUT -s 83.217.36.171/255.255.255.255 -j DROP

I also set this parameter in /etc/postfix/main.cf, since I only use my server to send mail from within squirrelmail (= webmail) or from a script in a cgi-bin dir:

mynetworks_style = host

I think the problem is related to a cgi script that is used by one of the sites that I host and that sends e-mail. I renamed the script to make sure this wouldn't happen again (my apache logs showed 404 errors from spiders that look for all kinds of cgi programs, probably in order to abuse them). I will also investigate how this cgi program can be abused and if there is an update available for it.

The damage isn't that great, since I think approximately 200.000 messages were put in the mail queue, but I couldn't find one that wasn't rejected by the receiving server (my spamassassin filter had already marked them as spam before sending them out).

Anyway, I feel like being robbed...

Server time accuracy

I finally managed to set my servers clock right automatically with ntpdate. I did this by following this guide. A list of publicly available servers is available here.

SpamAssassin

Finally, I installed a spam filter on my mail server. I did this by following Integrated Spamd In Postfix.
I needed to create a ".spamassassin" directory in my mail users directory in order to get rid of the warnings in /var/log/maillog.

"XXX unknowon type 0"

I get a JK2 message in Tomcat's catalina.out: "XXX unknowon type 0" (it IS "unknowon" and not "unknown"). Googling didn't help, so for the moment, I don't know what to do about this. There's isn't anything wrong with the site, so I am really clueless about this.

IP Alias

I tried to add an ip address (/etc/sysconfig/network-script/ifcfg-eth0:0) to my server, but when I did "service network restart", the network service went down but didn't restart. Consequence: a trip to Brussels for setting the network settings back to correct values. Took me three hours to get there and back, but since I stopped in Leuven and ate in "The Ribs" there, my day wasn't completely spoiled after all.

Hardening Fedora

My server is only up and running for one week and already there have been attempts to hack it. More specifically, this was mailed to me with logwatch:


Unknown Entries:
authentication failure; logname= uid=0 euid=0 tty=NODEVssh
ruser= rhost=66-195-18-169.dimenoc.com : 12 Time(s)

Failed logins from these:
admin/password from ::ffff:66.195.18.169: 4 Time(s)
guest/password from ::ffff:66.195.18.169: 2 Time(s)
root/password from ::ffff:66.195.18.169: 6 Time(s)
root/password from ::ffff:81.165.72.99: 1 Time(s)
test/password from ::ffff:66.195.18.169: 4 Time(s)
user/password from ::ffff:66.195.18.169: 2 Time(s)

Illegal users from these:
admin/none from ::ffff:66.195.18.169: 4 Time(s)
admin/password from ::ffff:66.195.18.169: 4 Time(s)
guest/none from ::ffff:66.195.18.169: 2 Time(s)
guest/password from ::ffff:66.195.18.169: 2 Time(s)
test/none from ::ffff:66.195.18.169: 4 Time(s)
test/password from ::ffff:66.195.18.169: 4 Time(s)
user/none from ::ffff:66.195.18.169: 2 Time(s)
user/password from ::ffff:66.195.18.169: 2 Time(s)


Today I disabled root logins on my server (so don't bother anymore!) and used key-based authentication by following these tips.

New server

Finally, my new server is installed in the LCL datacenter in Diegem, Belgium in a rack provided by capslock.be co-location.
Now I've waited two days, the DNS servers have finally picked up the change of ip and now almost anyone is using the new server.
One thing that bothered me a lot, was the configuration of the mail server. Did some tuning with /etc/hosts and stuff in order to avoid losing mails while the DNS was changed but not picked up by everyone.
After a lot of Linuxing and Googling, I found that I needed two deamons: a mail server (like postfix) and an interface to access the mails (like dovecot). Simple if you see it know, but I spend a lot of time figuring out why postfix wouldn't let me read my mails, and dovecot wouldn't accept them...
Also added SquirrelMail, a webmail package that allows me to read my mail from anywhere.

A very simple but effective way of speeding up the serving of web pages.
If .gif, .jpg or .js files are the most frequently visited files in your weblogs (webalizer, awstats or whatever), you know something's wrong!

Add this to your httpd.conf file in apache (or in a separate file with "Include conf.d/*.conf" for example). This will avoid the browser to check for each page if there is a new version of the .css or .js files available.

Of course, you can increase the time limit, but remember that if you ever want to update your site (you should!), many users will have old versions of the files and this will propably give (amongst others) javascript errors. One solution for this is to add a version number to your .css and .js files like "my-javascript-1.0.js".


ExpiresActive On
ExpiresByType text/css "access plus 1 hour"
ExpiresByType application/x-javascript "access plus 1 hour"
ExpiresByType image/gif "access plus 1 hour"
ExpiresByType image/jpeg "access plus 1 hour"
ExpiresByType image/png "access plus 1 hour"


Every time my Fedora server starts up, I got a "MySQL server failed" error. The problem was the ping command in /etc/init.d/mysqld that uses UNKNOWN_USER in the "mysqladmin ping" command.
Since I removed all anonymous users as a security measure, this didn't work anymore. So I changed that to a user that has ping rights (mysqladmin ping -u pinguser --password=somePassword).

I installed Fedora on my server and got a "smartd failed" at startup and shutdown. I edited /etc/smartd.conf so "/dev/hda" became "/dev/sda" (I have S-ATA disks).
Also, running "smartctl -i /dev/hda" indicated that SMART was not supported by my disks. Apparently, this must be enabled in the BIOS.
I did "chkconfig smartd off" and the smartd deamon doesn't load at startup anymore.