Hi Dovecot Users,
I’ve configured dovecot dsync replication and I see troubles in the logs and get user complaints which I can’t explain. I found similar threads on this mailinglist, but I couldn’t find a solution anywhere. Does anybody have dsync running without problems on a high volume mailserver?
I see the following logs, examples given:
Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: /var/mail/zoechi/dovecot-uidlist: Duplicate file entry at line 10395: 1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377 (uid 41092 -> 41093) - retrying by re-reading from beginning
with this one I’m not sure - it might be that this is completely OK because due to replication UIDs clash. Maybe that’s OK, but I couldn’t find a confirmation.
Feb 17 18:16:49 dovecot dovecot: imap(zoechi): Warning: Maildir /var/mail/zoechi: Expunged message reappeared, giving a new UID (old uid=41092, file=1487350019.M138380P28563.dovecot.wogri.at,S=18930,W=19377)
This one is definitely a problem, deleted messages re-appear in the mailbox.
I made sure that the 2 hosts doing replication have different hostnames. I run 2.2.27 (from debian-jessie backports). Config below.
Thanks for hints (or pointers to other example configs where dsync works without problems)
dovecot -n
2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.16 (fed8554)
OS: Linux 4.8.14 x86_64 Debian 8.7 ext4
auth_verbose = yes debug_log_path = /var/log/dovecot.debug doveadm_password = # hidden, use -P to show it first_valid_gid = 106 first_valid_uid = 104 hostname = localhost last_valid_gid = 106 last_valid_uid = 104 mail_gid = dovecot mail_location = maildir:/var/mail/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader namespace { list = children location = virtual:/var/mail/%n/virtual prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.1:12345 quota = maildir:User quota quota_rule = *:storage=5G quota_rule2 = Trash:storage=+200M quota_rule3 = Spam:ignore quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } pop3_deleted_flag = $POP3Deleted postmaster_address = postmaster@wogri.at protocols = " imap lmtp sieve pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap { process_limit = 1024 } service lmtp { inet_listener lmtp { port = 2003 } unix_listener lmtp { user = dovecot } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/dovecot/server.cert.pem ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota } protocol pop3 { mail_plugins = quota fts fts_lucene virtual notify replication } local 2a00:XXX::/64 { doveadm_password = # hidden, use -P to show it }
On 17 Feb 2017, at 21.37, Wolfgang Hennerbichler <wogri@wogri.com> wrote:
There seems to be something weird with using Maildir and replication. Haven't had time to debug it and it's likely not an easy bug to fix, so for now the solution would be to use only sdbox/mdbox with replication.
On 20/02/2017 9:09 PM, Timo Sirainen wrote:
I have experienced this quite a few times. It is tricky to reliably reproduce. I am also using replication with Maildir.
This appears to be a timing/locking issue of some sort. One way to trigger this seems to be to have a new mail come in and be delivered, and then as soon as it appears in Thunderbird, delete it straight away.
Thunderbird is set to move mails to Trash when they are deleted and expunge the Inbox on exit.
My replication partner sees the new mail has come in, and then it pushes it back to the original. Meanwhile the one on the original has already been deleted by the client so the mail re-appears back on the original again.
The latency to my replication partner is 150ms or so but this problem might become easier to reproduce if the delay is increased.
I can't say that this is definitely how to reproduce it all of the time but when I have experience this, that's roughly the sequence of events.
Reuben
On 20 Feb 2017, at 11:09, Timo Sirainen <tss@iki.fi> wrote:
Thank you, I can confirm that after the migration to sdbox I don’t see those errors anymore. I could not find good documentation for converting maildir to sdbox on the fly, so I wrote a script. If there’s interest I can share this.
wogri
On Feb 22, 2017, at 15:51, Wolfgang Hennerbichler <wogri@wogri.com> wrote: Thank you, I can confirm that after the migration to sdbox I don’t see those errors anymore.
Turns out that message still re-appear although all my mailboxes are converted to sdbox. The most likely case for a message to re-appear is that an unread message in the inbox is deleted. On the next imap sync it re-appears (only sometimes, not always). Although this time the server does not log any errors. This is my current dovecot config:
dovecot -n
2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.16 (fed8554)
OS: Linux 4.9.4 x86_64 Debian 8.7 ext4
auth_verbose = yes debug_log_path = /var/log/dovecot.debug doveadm_password = # hidden, use -P to show it first_valid_gid = 106 first_valid_uid = 104 hostname = localhost last_valid_gid = 106 last_valid_uid = 104 mail_gid = dovecot mail_location = sdbox:/var/mail/sdbox/%n mail_plugins = quota fts fts_lucene virtual notify replication mail_temp_dir = /var/lib/dovecot/tmp mail_uid = dovecot managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext editheader namespace { list = children location = virtual:/var/mail/virtual/%n prefix = virtual. separator = . } namespace inbox { inbox = yes list = yes location = mailbox "Deleted Messages" { auto = subscribe special_use = \Trash } mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { auto = no special_use = \Sent } mailbox Spam { auto = subscribe special_use = \Junk } mailbox Trash { special_use = \Trash } prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { default_language = de fts = lucene fts_lucene = whitespace_chars=@. mail_replica = tcp:172.16.1.2:12345 quota = count:User quota quota_rule = *:storage=6G quota_rule2 = Trash:storage=+200M quota_rule3 = Spam:ignore quota_vsizes = yes quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u sieve = /etc/sieve/%n.sieve sieve_default = /etc/sieve/default.sieve sieve_dir = ~/sieve sieve_extensions = +editheader } pop3_deleted_flag = $POP3Deleted postmaster_address = postmaster@wogri.at protocols = " imap lmtp sieve pop3" service aggregator { fifo_listener replication-notify-fifo { user = dovecot } unix_listener replication-notify { user = dovecot } } service doveadm { inet_listener { port = 12345 } } service imap-postlogin { executable = script-login /usr/local/bin/mail-location.sh } service imap { process_limit = 1024 } service lmtp { inet_listener lmtp { port = 2003 } unix_listener lmtp { user = dovecot } user = dovecot } service managesieve-login { inet_listener sieve { port = 4190 } service_count = 1 } service pop3 { process_limit = 1024 } service quota-warning { executable = script /usr/local/sbin/quota-warning.sh unix_listener quota-warning { user = dovecot } user = dovecot } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0600 } } ssl = required ssl_cert = </etc/dovecot/server.cert.pem ssl_key = # hidden, use -P to show it userdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } protocol lmtp { mail_plugins = quota sieve virtual notify replication } protocol lda { mail_plugins = quota fts fts_lucene virtual notify replication sieve } protocol imap { mail_max_userip_connections = 50 mail_plugins = quota fts fts_lucene virtual notify replication imap_quota } protocol pop3 { mail_plugins = quota fts fts_lucene virtual notify replication } local 2a00:YYYY:XXXX::/64 { doveadm_password = # hidden, use -P to show it }
wogri
participants (3)
-
Reuben Farrelly
-
Timo Sirainen
-
Wolfgang Hennerbichler