[Dovecot] Locking foolishness - Better info

16 Sep 2003

      I realized that my last message about locking gave bad information -- or
rather, very little information.
Here's my setup:

Home directories and user login information are located on an Xserve
running Mac OS X 10.2.6.  Home directories are split between 2 different
directories -- /Users and /Students.  Home dirs are exported via NFS to the
mail server.  User login info is shared via LDAP to the mail server and to
Windows machines in classrooms.

The mail server is an old beige G3 running Debian Woody (linux 2.4.20)
kernel.  It's completely up-to-date.  Homedirs are mounted at
/Network/Servers/network.uphamscornercs.net/Users and
/Network/Servers/network.uphamscornercs.net/Students, in keeping with the OS
X Server userdir name scheme.  The only user with a shell on the server is
root, other than that all user information is pulled from LDAP using PAM.

Postfix is setup to deliver mail to maildir:~/Maildir, and dovecot is set to
read mail from the same directory.  This is working without any problem.

Clients
Entourage works fine to access mailboxes -- it reads mail without a problem.
However, it pops up an error message intermittently that reads "Mail could
not be read: Internal Error [<datestamp>]".  This message doesn't seem to
interfere with reading mail, it's just an annoyance.

Squirrelmail doesn't work to access mailboxes.  It logs in fine, but gives
the following error message in all windows that require mailbox access:
"ERROR : Could not complete request.
Query: <imap query string>
Reason Given: Internal error [<datestamp>]"
In both cases (Entourage and Squirrelmail), the Internal Error is
accompanied by the following error message in the /var/log/mail.log file:
"imap(marlier): Sep 16 08:40:45 dovecot:Error:
/Network/Servers/network.uphamscornercs.net/Users/marlier/Maildir/.subscript
ions failed with subscription file file_wait_lock(): Invalid argument"
So, it looks to me like something isn't working with locking.  I'm not sure
whether the problem is with the NFS mount, or with dovecot, or with
permissions, or something else, but I'd like to get it fixed.  I love the
server, and if I can get this licked I think I've got a long-term setup.
Relevant sections of /usr/local/etc/dovecot.conf:
Use mmap() instead of read() to read mail files. read() seems to be a bit
faster with my Linux/x86 and it's better with NFS, so that's the default.
mail_read_mmaped = no
Copy mail to another folders using hard links. This is much faster than
actually copying the file. This is problematic only if something modifies
the mail in one folder but doesn't want it modified in the others. I don't
know any MUA which would modify mail files directly. IMAP protocol also
requires that the mails don't change, so it would be problematic in any
case.
If you care about performance, enable it.
maildir_copy_with_hardlinks = no
Check if mails' content has been changed by external programs. This slows
down things as extra stat() needs to be called for each file. If changes
are
noticed, the message is treated as a new message, since IMAP protocol
specifies that existing message are immutable.
maildir_check_content_changes = yes
Which locking methods to use for locking mbox. There's three available:
dotlock: Create <mailbox>.lock file. This is the oldest and most NFS-safe
solution. If you want to use /var/mail/ like directory, the
users
will need write access to that directory.
fcntl  : Use this if possible. Works with NFS too if lockd is used.
flock  : May not exist in all systems. Doesn't work with NFS.

You can use both fcntl and flock too; if you do the order they're declared
with is important to avoid deadlocks if other MTAs/MUAs are using both
fcntl
and flock. Some operating systems don't allow using both of them
simultaneously, eg. BSDs. If dotlock is used, it's always created first.
mbox_locks = dotlock
Should we create dotlock file even when we want only a read-lock? Setting
this to yes hurts the performance when the mailbox is accessed
simultaneously
by multiple processes, but it's needed for reliable reading if no other
locking methods are available.
mbox_read_dotlock = yes
Maximum time in seconds to wait for lock (all of them) before aborting.
mbox_lock_timeout = 300
If dotlock exists but the mailbox isn't modified in any way, override the
lock file after this many seconds.
mbox_dotlock_change_timeout = 30
umask to use for mail files and directories
umask = 0077
Drop all privileges before exec()ing the mail process. This is mostly
meant for debugging, otherwise you don't get core dumps. Note that setting
this to yes means that log file is opened as the logged in user, which
might not work. It could also be a small security risk if you use single
UID
for multiple users, as the users could ptrace() each others processes
then.
mail_drop_priv_before_exec = no
Thanks in advance for any help you can give.

Ian

[Dovecot] Locking foolishness - Better info

Ian Marlier

Relevant sections of /usr/local/etc/dovecot.conf:

Use mmap() instead of read() to read mail files. read() seems to be a bit

faster with my Linux/x86 and it's better with NFS, so that's the default.

Copy mail to another folders using hard links. This is much faster than

actually copying the file. This is problematic only if something modifies

the mail in one folder but doesn't want it modified in the others. I don't

know any MUA which would modify mail files directly. IMAP protocol also

requires that the mails don't change, so it would be problematic in any

If you care about performance, enable it.

Check if mails' content has been changed by external programs. This slows

down things as extra stat() needs to be called for each file. If changes

noticed, the message is treated as a new message, since IMAP protocol

specifies that existing message are immutable.

Which locking methods to use for locking mbox. There's three available:

dotlock: Create <mailbox>.lock file. This is the oldest and most NFS-safe

solution. If you want to use /var/mail/ like directory, the

will need write access to that directory.

fcntl : Use this if possible. Works with NFS too if lockd is used.

flock : May not exist in all systems. Doesn't work with NFS.

You can use both fcntl and flock too; if you do the order they're declared

with is important to avoid deadlocks if other MTAs/MUAs are using both

and flock. Some operating systems don't allow using both of them

simultaneously, eg. BSDs. If dotlock is used, it's always created first.

Should we create dotlock file even when we want only a read-lock? Setting

this to yes hurts the performance when the mailbox is accessed

by multiple processes, but it's needed for reliable reading if no other

locking methods are available.

Maximum time in seconds to wait for lock (all of them) before aborting.

If dotlock exists but the mailbox isn't modified in any way, override the

lock file after this many seconds.

umask to use for mail files and directories

Drop all privileges before exec()ing the mail process. This is mostly

meant for debugging, otherwise you don't get core dumps. Note that setting

this to yes means that log file is opened as the logged in user, which

might not work. It could also be a small security risk if you use single

for multiple users, as the users could ptrace() each others processes

then. mail_drop_priv_before_exec = no