[Dovecot] Corrupted index cache files
I see these errors more often than I'd like from Dovecot-1.1.2 on Mac
OS X 10.5.4 (names and numbers elided):
Corrupted index cache file %s: Corrupted virtual size for uid=%d: %d ! = %d Corrupted index cache file %s: Broken virtual size for mail UID %d Corrupted index cache file %s: used_file_size too large
How bad are these? What should I look for to find out why they happen?
Other details: File system is HFS+ Mail store is maildir Hardware is Intel (Mac Pro)
Thanks.
On Aug 8, 2008, at 9:59 PM, Mike Abbott wrote:
I see these errors more often than I'd like from Dovecot-1.1.2 on
Mac OS X 10.5.4 (names and numbers elided):
One or more users?
As long as Dovecot detects the corruption, the errors are transparent
to users since Dovecot fixes them automatically.
Post your dovecot -n output? Can you reproduce these easily with my
imaptest tool? http://imapwiki.org/ImapTest
One problem with HFS+ is that hard links are more or less buggy. But
v1.1's default settings should include dotlock_use_excl=yes. You maybe
should set maildir_copy_with_hardlinks=no, but that shouldn't cause
this bug.
One or more users?
Many different users.
Post your dovecot -n output?
Here's some of it. Not very enlightening.
1.1.2: /etc/dovecot/dovecot.conf
base_dir: /var/run/dovecot/ verbose_proctitle: yes first_valid_uid: 6 last_valid_uid: 50000 first_valid_gid: 6 last_valid_gid: 50000 mail_access_groups: mail mail_privileged_group: mail mail_location: maildir:~/mail
Can you reproduce these easily with my imaptest tool? http://imapwiki.org/ImapTest
Some of them. When running imaptest I see these dovecot errors:
Corrupted index cache file %s: record continues outside its allocated
size
Corrupted index cache file %s: record points outside file
Corrupted index cache file %s: used_file_size too large
Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d -> %d Error: user%d[%d]: <...>: Header CC changed Error: user%d[%d]: <...>: Header DELIVERED-TO changed Error: user%d[%d]: <...>: Header FROM changed Error: user%d[%d]: <...>: Header IN-REPLY-TO changed Error: user%d[%d]: <...>: Header MESSAGE-ID changed Error: user%d[%d]: <...>: Header REFERENCES changed Error: user%d[%d]: <...>: Header SUBJECT changed Error: user%d[%d]: <...>: Header SUBJECT changed Error: user%d[%d]: <...>: Header TO changed Error: user%d[%d]: UID %d changed Message-Id Error: user%d[%d]: UID=%d INTERNALDATE changed Error: user%d[%d]: uid=%d <...>: BODY changed Error: user%d[%d]: uid=%d <...>: BODYSTRUCTURE changed Error: user%d[%d]: uid=%d <...>: BODY[%d] size changed Error: user%d[%d]: uid=%d <...>: BODY[HEADER] size changed Error: user%d[%d]: uid=%d <...>: BODY[TEXT] size changed Error: user%d[%d]: uid=%d <...>: BODY[] size changed Error: user%d[%d]: uid=%d <...>: ENVELOPE changed Error: user%d[%d]: uid=%d <...>: RFC822.SIZE size changed
Changing maildir_copy_with_hardlinks makes no difference.
You say that the dovecot errors are harmless because dovecot fixes
them, but still there must be some downside, if only a performance
hit. Any other info I can provide to help you figure this out?
On Aug 13, 2008, at 6:09 PM, Mike Abbott wrote:
mail_access_groups: mail mail_privileged_group: mail
You probably won't need either of these. And there's no point in
setting them to the same value.
How soon? With what kind of imaptest parameters? I can't reproduce
this on my Macbook (OS X 10.5.4, HFS+).
Plus I see LOTS of errors from imaptest itself: Error: UIVALIDITY changed: %d -> %d
Did you test with a single user or multiple users? imaptest's support
for testing multiple users is currently a bit broken, but this
shouldn't happen with a single user..
These shouldn't be happening as long as you use a test mbox file that
has unique Message-ID headers. Were you using the recommended dovecot-
crlf as the input?
Yes, there's a performance hit since all the cached data is lost.
How soon? With what kind of imaptest parameters? I can't reproduce
this on my Macbook (OS X 10.5.4, HFS+).
I ran imaptest for just a few minutes and saw all these errors in that
time. Default imaptest parameters except for user/host names etc.
This error happens with clients=1, clients=10 and clients=100.
I passed in your dovecot-crlf file to imaptest, but the message store
was cloned from elsewhere first. So you're saying that dovecot only
works with special hand-crafted input, and not with real-world mail?
Hmmm.... :) :)
Actually I do see plenty of duplicate Message-ID headers in the mail
store (not from your input file). I'll investigate that further.
Thanks for the tip.
On Aug 13, 2008, at 7:12 PM, Mike Abbott wrote:
I mean the user parameter, do you use user=something or user=something
%d or similar to specify one/multiple different users where imaptest
logs on to?
Dovecot doesn't care about any of that, but imaptest's state tracking
requires that Message-ID header uniquely identifies a mail.
So the imaptest run was done with a mailbox that already contained
other data? Can you break if if you try with an empty mailbox where
only imaptest starts appending new mails?
I re-ran imaptest on an empty mail store, single client, multiple
users, using your dovecot-crlf input file, for a couple hours. Here's
the distribution of errors that imaptest reports:
100 Error: user%d[%d]: <...>: Header DELIVERED-TO changed
167 Error: user%d[%d]: <...>: Header CC changed
417 Error: user%d[%d]: <...>: Header TO changed
438 Error: user%d[%d]: <...>: Header IN-REPLY-TO changed
445 Error: user%d[%d]: <...>: Header REFERENCES changed
485 Error: user%d[%d]: <...>: Header FROM changed
503 Error: user%d[%d]: <...>: Header SUBJECT changed
544 Error: user%d[%d]: <...>: Header MESSAGE-ID changed
764 Error: user%d[%d]: UID=%d INTERNALDATE changed
823 Error: user%d[%d]: UID %d changed
1040 Error: user%d[%d]: uid=%d <...>: BODY changed
1052 Error: user%d[%d]: uid=%d <...>: BODYSTRUCTURE changed
1546 Error: user%d[%d]: uid=%d <...>: ENVELOPE changed
1833 Error: user%d[%d]: uid=%d <...>: RFC822.SIZE size changed
3493 Error: user%d[%d]: uid=%d <...>: BODY[] size changed
3691 Error: user%d[%d]: uid=%d <...>: BODY[HEADER] size changed
3695 Error: user%d[%d]: uid=%d <...>: BODY[TEXT] size changed
3880 Error: user%d[%d]: uid=%d <...>: BODY[%d] size changed
376365 Error: UIVALIDITY changed: %d -> %d
During that time dovecot reported no corruption errors.
What does it mean that imaptest is having such trouble?
On Aug 14, 2008, at 9:56 AM, Mike Abbott wrote:
What do you need to do to get corruption errors? Is multiple clients
enough?
What does it mean that imaptest is having such trouble?
I think it just means that imaptest's support for multiple users is
broken. I always test it only using a single test user.
On 8/13/2008, Mike Abbott (michael.abbott@apple.com) wrote:
Post your dovecot -n output?
Here's some of it. Not very enlightening.
Why just some of it? Its not like its that much, and what you may *think* is irrelevant, may in fact be *very* relevant...
--
Best regards,
Charles
participants (3)
-
Charles Marcus
-
Mike Abbott
-
Timo Sirainen