Google Apps E-mail Storage Reaches 7 GB

My Google Apps Standard Edition e-mail account (like GMail, but I can use my own hostname instead of gmail.com) reached 7 GB. I just noticed it today, but it may already be there for a few days since I don't watch the GMail counter continuously.

The Thrash folder however still shows
No conversations in the Trash. Who needs to delete when you have over 2000 MB of storage?!

7 GB that is, if you use the same metrics hard disk manufacturers use. It's still not really 7 GB though.

Looks like I still have some space left after 4 years of e-mailing (I didn't import my old hotmail and student accounts I had before that):
You are currently using 472 MB (6%) of your 7007 MB.

I used a little trick (don't tell Google!) to import my e-mail from my old server. When you purchase a Premier Edition account, you have better tools to migrate your e-mail. After 15 days (and a successful migration) I downgraded my Premier Edition to a Standard Edition for free (you have 30 days to try it).

How to lower the load average on a server with more than 50% in 10 seconds

$uptime
... load average: 0.54, 0.40, 0.36

$sudo vim /etc/fstab
(add the noatime option)

# /dev/sda3
UUID=8623d9e3... / ext3 defaults,errors=remount-ro,noatime 0 1

$sudo mount -a
...
$uptime
... load average: 0.24, 0.16, 0.17

So far this completely unscientific proof.

More info about the noatime option.

Internal Refactoring

For my 10-day visit to Tokyo, Kyoto, Nara, Hakone and Chiba (my brother-in-law's wedding), I needed to refactor my internal progamming a bit to avoid OutOfMemoryExceptions.

// This class is package protected to avoid 
// external programs messing up.
class Brain {

public void handleEvent(MeetNewPersonEvent event) {
  ...
  if (getLocation().equals(Locations.JAPAN)) {
    fireEvent(new BowEvent(event));
  }
}

public void handleEvent(BowEvent event) {
    // Avoid infinite loop.  The problem is the 
    // 'esteemed higher' part, the person
    // for whom you're bowing may think the same.
    if (event.isPersonEsteemedHigher() 
        && !event.hasBowedTooMuch()) {
      bow();
    } else {
      nod();
    }
}

protected void bow() {
  lookSincere();
  smile();
  bendForward();
}

I also needed to reprogram the eating subroutines.

public void handleEvent(FeelingHungryEvent event) {
...
  if (getLocation().equals(Locations.JAPAN)) {
     // This was a tricky one to handle, 
     // the implementation is left 
     // as an exercise to the reader.
     uploadChopsticksRoutine();
  }
}

The RunForTrainEvent and especially the WaitForTrainEvent could be canceled out since public transportation is much better than the location I originally wrote it for (Belgium).

public void handleEvent(RunForTrainEvent event) {
  if (getLocation().equals(Locations.JAPAN)) {
     stopRunning();
     relax();
     Thread.sleep(5*Timer.MINUTE);
  }
}

Finally, I needed to handle the RunningNoseEvent (extends HasColdEvent) better.

public void handleEvent(RunningNoseEvent event) {
...
  if (getLocation().equals(Locations.JAPAN)) {
    // Blowing your nose in public is NOT DONE 
    // in Japan.  This is considered 
    // a bit the same as burping.  
    // Public humiliation is your part when
    // this is not checked.
    dipNose();
  } else {
    blowNose();
  }
}

protected Location getLocation() {
  ...
  if (isCurrentLocationUnknown()) {
    if (bodyTallerThanMostOthers() 
            && friendlyPeople() && metroEvery2Minutes()
            && dressCode.equals(Dresscodes.COSTUME)
            && eatingCode.equals(EatingCodes.SHOPSTICKS)) {
       return Locations.JAPAN;
    }
  }
}

} // End class

This is released under an Apache license. Please notify me if these changes are of any use to you.

Blocking bad bots

Today I blocked some bad bots that were spidering some of my sites. Most notably Custo, which downloads your entire site.

An interesting solution is posted here (I used the mod_rewrite option). You can test this by changing your user agent in Firefox.

This guy seems to be following bad bots.

I added Java, Nutch, Jakarta, Vagabondo and an empty bot name to the list of bad bots.

GWT: follow-up

This is a follow-up post on Why I dumped GWT.

First of all, I had a really long week + weekend and I was tired when I wrote the post. I apologize for the rather in-your-face title. I should have chosen a more subtitle wording, especially since I really appreciate all the work that developers donate in their free time to open source projects.

I did't expect my blog post would end up as the main article on ongwt.com. I use my blog mainly to communicate with colleagues about my work. It probably has something to do with my recent switch to feedburner. Hooray for feedburner!

Like I wrote, I like GWT and its approach. It's really nice to see how intelligent the development team approached and solved the problem at hand. The image handling (sprites), js compression and http round-trip optimisations are really clever.

I'll start by describing how my site came to what it is now. The site started as a playground for me. I wanted to try the latest new thing (ajax!) and so I first started with scriptaculous. I didn't succeed in getting the layout right with pure css (after all, I'm only a Java developer) and I stumbled upon GWT. The mail app demo is really nice and this gave me the idea to start a site that searches on-line marketplaces and lets users treat the classifieds as e-mail: with the possibility to delete, mark items as read/unread and star them. Much like Google Reader.

For this, GWT was the perfect match. Everything went as I expected it to do, sometimes with some cursing about why my onclick events were not fired and why a non-existing background image in css stopped the hosted mode to work, but all-in-all it was very good.

After some (positive) discussion with my other half, I wanted the work I put in it to give me some return-on-investment (money!). It turned out after some basic user testing (she sitting at the keyboard and I shouting "why would you do that?" and "that's not meant to be used like that") that the whole idea was too complex for a standard user (no offense to my super-intelligent girlfriend) who stumbles upon my site. So week after week I removed some of the functionality to make the page less overwhelming. Until I finally found myself using GWT only for the autocompleter, which clearly wasn't the intention of the GWT framework. This, together with the remarks I gave in the previous post (adsense, analytics and seo) made me decide to temporarily stop developing with GWT.

I expect to start again with GWT once the site "gains some momentum", and then I will re-enable those more complex features which should be easier than with mootools. I'll probably ask some advice from a usability expert about how to design the page with all this functionality without overwhelming first-time users. And I will check out MyGWT and GWT-Ext more thoroughly.

Why I dumped GWT

I've used GWT for over half a year now on koopjeszoeker.be. Two weeks ago I decided to stop development with GWT and go with plain HTML and mootools for the autocompleter. I've used mootools already a lot and I'm really getting the hang of it.

Why? Why did I spend all this time developing in GWT and why did I decided to stop?

First of all, GWT is a fantastic framework for doing web development. I think it's the best tool at the moment if you want to build the next GMail or an intranet application. For all those slow and lousy web interfaces (for timesheets, CMS, ...), GWT could come to the rescue. But my site is completely different.

Some of the reasons below are not really related to GWT, but more to using ajax in general. It is my opinion however, that these problems are easier to solve with 'standard' javascript libraries like mootools, prototype, dwr or scriptaculous since these have a nice way to add some ajax to certain DOM elements. For example, in GWT I had to subclass the autocompleter textbox so I could attach it to an input field that already existed in the HTML. Maybe all of this could by solved if GWT had constructors that accept a DOM id too.

SEO

I'm entering in a highly competitive segment where SEO is really important. Since most of the html is build with GWT, you end up with a pretty empty page for Google. I added some noscript tags, but this was not really helpful.

Adsense

Another problem were my adsense banners. Since I didn't have a lot of content on the page, the banners were sometimes off topic. An even bigger problem was that the banners stayed the same when people searched for different keywords (since the ajax refresh didn't trigger an adsense refresh). I solved this by doing the search with a page refresh instead of an ajax call. The ajax part of the site was limited to sorting, faceting, i18n and displaying tips.

Google Analytics

I'm also using Google Analytics. Although no real evidence exists, it would be naive to think that Google isn't using this data. But because of the ajax calls, I don't get as many pageviews as a static version of competing sites. Every visitor is seen as doing 1 page visit, while he may have browsed several pages. This makes my bounce rate in Google Analytics really high. This can't be good for my Google rankings.
In Belgium we have CIM Metriweb, a kind of archaic tracking system that is used when marketeers look for sites that have many hits. I'm not currently using this, but this thing depends on pageviews if you want the big guys to donate to your site.

What now?

I wanted a fully functional HTML version, where GWT was injected in some places to replace the full page loads with ajax calls. However, I couldn't find an easy way to do this. And once I succeeded, I found that I had almost no code left in GWT that was worth using it instead of mootools. So now, after a lot of research and experimenting, I decided that I'll go for the plain-old html way and spiced up some parts with ajax (like the "so 2007" textbox autocompleter).

I discovered the Blueprint CSS framework (version 0.7 now has semantic classes) and CSS sprites. I've used Kuler and read a lot about CSS tips and tricks. I even read a bit about usability.

And since I spend 3 hours a day on the train, I have time to redesign the site. Using blueprint, it really was easy and the result is a much better looking, stable, fast site. Check the homepage: it only has 1 css, 1 javascript, 1 gif and 1 jpeg, but there are 25 images! Ah, the magic of blueprint, sprites and jawr...


Update: please see GWT follow-up

Compress Javascript and CSS with Jawr

Today I used the nice Jawr taglib which compresses javascript and css files. There's enough information on the Jawr website about how to configure everything, so I won't write about this.

Things to remember are:

• Better structuring / versioning of your development javascript and css versions while still publishing them as 1 file


• Gzip support for compliant browsers


• Give the css and js files cache headers 'until the sun explodes'


• When you deploy a new version of your site, a new css and js version will be downloaded by the browser

Net result: our YSlow score went from 49 to 69!

2 Things I want in CSS 3

I've done some html/css restyling lately and there are some things I would like to see added to CSS 3. The process to request some changes to be incorporated into CSS 3 is a bit overwhelming to me, so I just post them here and hope they will be picked up by someone.

CSS variables

CSS variables would be nice. I want a way so I can easily change all colors in my CSS with one adjustment, not by searching for the color in the file and replacing it with the new value. I also think this would increase readability of the file.
This would allow to define recurring parts of the layout in the CSS file like this:

var backgroundcolor : #FFFFFF;
var border: 1px solid #CCCCCC;

.container {
background-color: $backgroundcolor;
border: $border;
}

.navbar{
border: $border;
}


Path variables

If I could rename a path selector to a variable, I could remove a lot of classes from the html and still be able to easily change the css.
Take this example:

.navbar ul li a {
text-decoration: none;
}

.navbar ul li a:hover {
text-decoration: underline;
}

This would become:

var listItem: .navbar ul li;

$listItem a {
text-decoration: none;
}

$listItem a:hover {
text-decoration: underline;
}


These examples look simple, but my experience is that a lot of the same values can be found in many CSS files. Wouldn't it be nice if we have a block of variable definitions at the top so we only have to specify once that the color of all borders should be changed?

This would also make it easier to let users (on a blog for example) override the colors with their own stylesheet, which overrides the variables with their settings.

The only way I know off to achieve this at the moment is by generating the CSS with a templating framework like JSP or Velocity (or why not PHP), but this seems like overkill to me.

So, anyone with the power to move the W3C board, go on (and let me know of the results)!

Website Performance tuning with Firebug and YSlow

Today I discovered a cool plugin for Firefox: YSlow.

In combination with Firebug, it allows you to quickly get a report about performance issues with your site, like too many css or javascript files, missing cache headers and much more.

I got a score of 66 for koopjeszoeker.be! Not much to improve there, besides switching to a CDN, but I don't think this is something that will happen very soon (maybe with Amazon S3?).

Ubuntu install²

Below a write-down of the steps I performed to install 2 ubuntu 7.10 servers (webserver + mysql database server) with IPMI.

Reminders:
start services with
- /etc/init.d/mysql start (or stop)
- /etc/init.d/apache2 start (or stop or reload or force-reload)
- Connect to database server: mysql -u root -p --host

Not done:
- SSL on Apache
- DenyHosts (prevent ssh dictionary attacks): see http://packages.ubuntu.com/edgy/net/denyhosts and http://denyhosts.sourceforge.net/

Steps
=====

OS installation
- Change router so desktop gets ip address in 192.168.111.x range
- Connect to ipmi (192.168.111.111) and change ip to 192.168.1.200. Mount ubuntu iso by opening console in browser (http://192.168.1.200) and click on the diskette icon in the right top and choose "mount iso".
- Go to console and reboot.
- Follow instruction steps for Ubuntu (check keyboard: Belgian keyboard gives problems with IPMI, but virtual keyboard and ssh work ok.).
- Remove virtual drive (iso) and reboot.

apt-get
- Remove (comment) line in /etc/apt/sources.list with a dependency on the cd-rom

ntpd
- sudo apt-get install ntp
- sudo vi /etc/ntp.conf; add
server be.pool.ntp.org
server pool.ntp.org

Apache 2
- /etc/init.d/apache2 start (or stop or reload or force-reload)
- sudo a2enmod deflate
- sudo a2enmod headers
- sudo a2enmod expires
- set up name-based virtual hosting following http://httpd.apache.org/docs/2.0/vhosts/name-based.html

PHP
- Copy contents from http://cvs.php.net/viewvc.cgi/php-src/php.ini-recommended?revision=1.179.2.11.2.23.2.1 to php.ini and set variables as needed
- sudo apt-get install php5-gd

eAccelerator
- sudo apt-get update
- sudo apt-get install php5-dev (needed for phpize5)
- Install eaccelerator following the guide from http://eaccelerator.net/wiki/InstallFromSource and http://2bits.com/articles/installing-eaccelerator-0-9-5-1-on-ubuntu-feisty-7-04.html
- Don't forget:
- mkdir /var/cache/eaccelerator
- chmod 644 /var/cache/eaccelerator
- the login credentials for control.php are in the file itself (admin/eAccelerator): change them!

Varnish
- sudo apt-get install varnish --> error in dpkg (see http://ubuntuforums.org/archive/index.php/t-438794.html)
- sudo apt-get install libc6-dev
- sudo apt-get remove varnish
- sudo apt-get install varnish
- Management: telnet 127.0.0.1 6082 (see http://varnish.projects.linpro.no/wiki/ManagementPort) (exit with ctrl+] and quit)
- varnishstat shows statistics
- Varnish by default listens on http://127.0.0.1:6081/
- sudo vi /etc/default/varnish: change 6081 to 80
- sudo vi /etc/apache2/ports.conf: change 80 to 81
- sudo vi /etc/varnish/vcl.conf
- varnish vcl docs: "man vcl"
- Test response headers with GET -ed http://www.example.com
- Show varnish logs in apache format: sudo varnishncsa

MySQL
- /etc/init.d/mysql start (or stop)
- Connect to database server: mysql -u root -p --host onthoo2
- MySQL data is located under /var/lib/mysql/
- Change mysql root password:
mysql> USE mysql;
mysql> UPDATE user SET Password=PASSWORD('new-password') WHERE user='root';
mysql> FLUSH PRIVILEGES;

CVS
- sudo apt-get install cvs

SVN
- sudo apt-get install subversion
- pipe through varnish

Java
- sudo apt-get install sun-java6-jdk
- See https://help.ubuntu.com/community/Java

Tomcat
- Disable port 8080, enable AJP on port 8009
- Change proxy allow in apache2:
<Proxy *>
AddDefaultCharset off
Order deny,allow
#Deny from all
Allow from webserver
</Proxy>

ImageMagick (compile from source for newer version 6.3.7)
- Download sources from ImageMagick website
- sudo apt-get install libjpeg62
- sudo apt-get install libjpeg62-dev
- sudo ldconfig
- cd ImageMagick-6.3.7
- sudo ./configure --prefix=/usr
- sudo make
- sudo make install

Google sitemap generator
- See https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html
- Modified log path from /var/log/httpd/ to /var/log/apache2/
- Added crontabs to /etc/cron.daily
- Changed python2.4 to python2.5 in crontabs

AWStats
- sudo apt-get install awstats
- sudo apt-get install libgeo-ipfree-perl
- vi awstats.conf.local:
LogFormat=1
AllowToUpdateStatsFromBrowser=1
AllowFullYearView=3
- vi awstats.www.pets.be.conf:
LogFile="/var/log/apache2/www.pets.be/access.log"
SiteDomain="www.pets.be"
HostAliases="localhost 127.0.0.1 dierenasielen.be dierenasiel.be refugespouranimaux.be pups.be asiel.org"
- sudo /usr/lib/cgi-bin/awstats.pl -config=www.pets.be -update

Exim (mail sender)
- See https://help.ubuntu.com/7.10/server/C/exim4.html
- Don't forget to run 'sudo update-exim4.conf' when the wizard has finished

Logwatch
- sudo apt-get install logwatch
- default conf is in /usr/share/logwatch/
- add /etc/logwatch/conf/logwatch.conf:
# Default person to mail reports to. Can be a local account or a
# complete email address.
MailTo = logwatch@example.com
# Default person to mail reports from. Can be a local account or a
# complete email address.
MailFrom = root@example.com

Firehol (firewall) on both servers
- sudo apt-get install firehol
- sudo vi /etc/firehol/firehol.conf:
webserver:

         version 5
         interface eth0 internet
         client all accept
         server http accept
         server ssh accept

database server:

          version 5
          clients="webserver"
          interface eth+ internet
                client all accept
                server mysql accept src "$clients"
                server ssh accept

- Patch for iptables warnings (see https://bugs.launchpad.net/ubuntu/+source/firehol/+bug/78017):
- sudo sed 's/%q/%b/g' /lib/firehol/firehol > TMPFILE && sudo mv TMPFILE /lib/firehol/firehol
- sudo chmod 744 /lib/firehol/firehol
- sudo firehol start
- Check with "sudo iptables -L"

IPMI
- change admin username and password
- require https
- use Java Sun plugin
- set invalid login retries and retry timout
- setup ipmitool if you need to change the ip address on which ipmi listens (see https://help.ubuntu.com/community/IPMI):
- sudo apt-get install ipmitool
- cd /usr/share/ipmitool
- sudo cp ipmi.init.basic ipmi.init.ubuntu
- sudo vi ipmi.init.ubuntu (and remove the if loop around the 'modprobe ipmi_si # try new module name' line) - see https://bugs.launchpad.net/ubuntu/+source/ipmitool/+bug/110992
- change ip address: sudo ipmitool -I open lan set 1 ipaddr 85.158.x.x
- change default gateway: sudo ipmitool -I open lan set 1 defgw ipaddr 85.158.x.x
- change netmask sudo ipmitool -I open lan set 1 netmask 255.255.255.x
- browse to http(s)://85.158.x.x

Backup
- See http://ubuntuforums.org/showthread.php?t=35087
- Complete backup of the system:
vi /var/backups/fullimage:
tar cvpzf /backup/fullimage.tgz --exclude=/proc --exclude=/lost+found
--exclude=/backup/ --exclude=/mnt --exclude=/sys /
- Restore backup with "tar xvpfz /backup/fullimage.tgz -C /"
- Special backup scripts in /var/backups
- crontab -e:
0 3 * * * /var/backups/backup-all