[Dovecot] Poor pop3 over nfs performance

[Mark Zealey]

> rc11 had this fix which may be relevant:
> 
> 	- dovecot-uidlist is now recreated if it results in 
> file shrinking
> 	  over 25%.

Not sure this fix is relevent; but I have upgraded to 1.1.1

> With v1.1 INDEX=MEMORY shouldn't matter that much anymore, since it
> should cache the virtual message size to dovecot-uidlist.
> 
> 1) Do you have ,W=<size> in maildir filenames? Apparently not?

No; yesterday though I did switch exim to delivering through lda -
surprisingly though there was not much effect on the graphs.

> 2) Do you have W=<size> added to dovecot-uidlist?

I have found something here which I believe is a bug; will detail below

> 3) Are your POP3 users download + delete or do they leave mail on the
> server?

A mixture; I believe at least 80% will download+delete; probably closer
to 95% but we do have some very large mail stores - most of these would
be accessed by a combination of pop and imap.

> You should be able to use dotlock_use_excl=yes.
...
> dotlock is the slowest locking method. Did you try how well 
> fcntl would
> have worked?

Not tried fcntl; I don't really trust nfs locking. I have enabled the
excl in the config; again not much change.

Now what I discovered in testing a particularly problematic user with an
800mb cur directory (under v1.1.1; I have never used v1.0 on this mbox
but it was converted with your courier convert script); initially they
had the following:

drwx------ 2 exim exim   2048 Apr 11  2007 courierimapkeywords
-rw-r--r-- 1 exim exim     47 Apr 22 15:56 courierimapsubscribed
-rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb
-rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist
drwx------ 2 exim exim 499712 Jul  2 10:35 cur
-rw------- 1 exim exim  20912 Jul  2 10:51 dovecot.index
-rw------- 1 exim exim  33056 Jun 17 17:34 dovecot.index.cache
-rw------- 1 exim exim  31336 Jul  2 10:37 dovecot.index.log
-rw------- 1 exim exim 158465 Jul  2 10:13 dovecot-uidlist
drwx------ 2 exim exim   4096 Jul  2 10:29 new
-rw------- 1 exim exim     23 Jun 24 22:49 subscriptions
drwx------ 2 exim exim   2048 Jul  2 10:42 tmp

As you can see, the cache file is very old, and the uidlist file is
behind the last time that the logged in (which was 10:35). When I log in
over pop and do a stat/list, it re-reads everything in cur/new but
didn't update the dovecot-uidlist file or the index.cache file (the log
file is updated though). Dovecot-uidlist file does not have the extra W=
tags in it. I removed the dovecot.index.cache file and performed the
same login/stat/list operation and logged out, and suddenly the
dovecot-uidlist file was updated to the new v1.1 format, and the
directory listing looks like:

total 1068
drwx------ 2 exim exim   2048 Apr 11  2007 courierimapkeywords
-rw-r--r-- 1 exim exim     47 Apr 22 15:56 courierimapsubscribed
-rw-r--r-- 1 exim exim 142493 Jun 23 19:57 courierimapuiddb
-rw-r--r-- 1 exim exim 195309 Jun 24 18:52 courierpop3dsizelist
drwx------ 2 exim exim 499712 Jul  2 10:35 cur
-rw------- 1 exim exim  20912 Jul  2 10:53 dovecot.index
-rw------- 1 exim exim  31428 Jul  2 10:53 dovecot.index.log
-rw------- 1 exim exim 175508 Jul  2 10:53 dovecot-uidlist
drwx------ 2 exim exim   4096 Jul  2 10:29 new
-rw------- 1 exim exim     23 Jun 24 22:49 subscriptions
drwx------ 2 exim exim   2048 Jul  2 10:42 tmp

This looks like a bug to me; I will have a look in the source and see if
I can find anything, but I thought I'd send this email as you'll be able
to find the bug faster than I can.

Thanks,

Mark