[Dovecot] Investigating suspected cache corruption
Hi All,
I'm running Dovecot 1.1.11 for a site of about 7000 active users with
a Squirrelmail frontend. We migrated from Courier (big shout to
everyone who worked on the wonderful part of the Dovecot wiki that
deals with Courier migration) and I have been extremely happy with the
performance gains Dovecot has given us. Not only is the IMAP part of
Squirrelmail faster, but our same hardware is under far less stress
than it was before the migration.
To date, the only problem we have experienced is a seemingly random
but significant slowdown in Squirrelmail's loading of the messages in
a user's Inbox. Each time, I have been able to fix the problem by
deleting the user's dovecot.index, dovecot.index.cache and
dovecot.index.log files in the user's INDEX directory for their
dovecot Inbox.
I tried having users experiencing this problem log in to a test
Squirrelmail box with mail_debug on, but I didn't see anything out of
the ordinary and they report the same slowness on the test server. I
don't see anything but normal login/logout IMAP lines in the
production dovecot logs.
I have hesitated a bit in posting to the list since this seems to be
happening to a small percentage of users and it technically may be a
problem with Squirrelmail, but the fact that I'm able to fix the
problem by nuking Dovecot's index files has me concerned - does anyone
have any suggestions for troubleshooting? My first idea when I get
another one of these is to do a telnet IMAP session to Dovecot on one
of the Squirrelmail boxes and see if I can get it to slow down when
retrieving messages from the Inbox.
We are using Maildir over NFS with the control/index files stored on
NFS as well. We're doing FS quota over NFS using rquotad to query the
NFS server for each user's quota. Here's the output of dovecot -n:
1.1.11: /etc/dovecot.conf
Warning: fd limit 1024 is lower than what Dovecot can use under full
load (more than 1536). Either grow the limit or change
login_max_processes_count and max_mail_processes settings
OS: Linux 2.6.18-128.1.1.0.1.el5 x86_64 Red Hat Enterprise Linux
Server release 5.3 (Tikanga) login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(pop3): /usr/libexec/dovecot/pop3-login login_process_per_connection: no login_processes_count: 10 login_max_processes_count: 512 verbose_proctitle: yes first_valid_uid: 0 mail_location: maildir:~/Maildir:CONTROL=/Mail/mailhome/ %u/.dovecot:INDEX=/Mail/mailhome/%u/.dovecot mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(pop3): /usr/libexec/dovecot/pop3 mail_plugins(default): quota imap_quota mail_plugins(imap): quota imap_quota mail_plugins(pop3): mail_plugin_dir(default): /usr/lib64/dovecot/imap mail_plugin_dir(imap): /usr/lib64/dovecot/imap mail_plugin_dir(pop3): /usr/lib64/dovecot/pop3 imap_client_workarounds(default): outlook-idle delay-newmail imap_client_workarounds(imap): outlook-idle delay-newmail imap_client_workarounds(pop3): imap_logout_format(default): bytes(in/out)=%i/%o imap_logout_format(imap): bytes(in/out)=%i/%o imap_logout_format(pop3): bytes=%i/%o pop3_uidl_format(default): %08Xu%08Xv pop3_uidl_format(imap): %08Xu%08Xv pop3_uidl_format(pop3): UID%u-%v pop3_client_workarounds(default): pop3_client_workarounds(imap): pop3_client_workarounds(pop3): outlook-no-nuls oe-ns-eoh pop3_logout_format(default): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(imap): top=%t/%p, retr=%r/%b, del=%d/%m, size=%s pop3_logout_format(pop3): top(count/bytes)=%t/%p, retr(count/bytes)=%r/ %b, del(deleted/pre-deleted)=%d/%m, mailbox-size(bytes)=%s namespace: type: private prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: mechanisms: plain login passdb: driver: pam userdb: driver: passwd plugin: quota: fs:Mail Quota:user
Many thanks, David Warden
On Wed, 2009-10-14 at 16:17 -0400, David Warden wrote:
To date, the only problem we have experienced is a seemingly random
but significant slowdown in Squirrelmail's loading of the messages in
a user's Inbox. Each time, I have been able to fix the problem by
deleting the user's dovecot.index, dovecot.index.cache and
dovecot.index.log files in the user's INDEX directory for their
dovecot Inbox.
First thing to try next time: Does it help if you only delete dovecot.index.cache file? How large is the cache file? And take a copy of the dovecot.index* files before doing that, I can then ask a few more questions about them.
I tried having users experiencing this problem log in to a test
Squirrelmail box with mail_debug on, but I didn't see anything out of
the ordinary and they report the same slowness on the test server. I
don't see anything but normal login/logout IMAP lines in the
production dovecot logs.
mail_debug doesn't really help with performance debugging.
My first idea when I get
another one of these is to do a telnet IMAP session to Dovecot on one
of the Squirrelmail boxes and see if I can get it to slow down when
retrieving messages from the Inbox.
That sounds like a good idea to try. And use the exact same IMAP commands as Squirrelmail to do it. strace -tt output of of the slow imap process could also show something interesting.
participants (2)
-
David Warden
-
Timo Sirainen