November 2021 - dovecot

mail_location for doveadm?
by Grant Keller 05 Nov '21

05 Nov '21

Hello, I am working on getting doveadm commands working from a director to a backend server and I am hitting a bit of a roadblock. We use a homebrew hashing algorithm for mail_location, which we set for users in postlogin scripts. Works fine except I get the following when I run doveadm on a user: # doveadm -D expunge -u gvk mailbox INBOX.Trash SENTBEFORE 1-Jan-2021Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message) Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message) doveadm(gvk): Debug: auth-master: passdb lookup(gvk): Started passdb lookup doveadm(gvk): Debug: auth-master: conn unix:/var/run/dovecot/director-userdb: Connecting doveadm(gvk): Debug: auth-master: conn unix:/var/run/dovecot/director-userdb (pid=21984,uid=0): Client connected (fd=11) doveadm(gvk): Debug: auth-master: passdb lookup(gvk): auth PASS input: user=gvk proxy=y proxy_refresh=450 host=c.lab.imapd.sonic.net hostip=184.23.168.77 doveadm(gvk): Debug: auth-master: passdb lookup(gvk): Finished passdb lookup (user=gvk proxy=y proxy_refresh=450 host=c.lab.imapd.sonic.net hostip=184.23.168.77) doveadm(gvk): Error: remote(c.lab.imapd.sonic.net:1842): mail_location not set and autodetection failed: Mail storage autodetection failed with home=/home2/22/25/gvk doveadm(gvk): Debug: auth-master: conn unix:/var/run/dovecot/director-userdb (pid=21984,uid=0): Disconnected: Connection closed (fd=11) I don't see anything like a doveadm-postlogin service in the documentation, so how would I go about making this work. Grant Keller System Operations grant.keller(a)sonic.com

2 1

Memory leaks in dovecot
by Joan Moreau 04 Nov '21

04 Nov '21

Hi Anyone can help on those memory leaks since 2.3.17 ? These came after adding the fts_mail_user_init(user, FALSE, &error) and fts_mail_user_deinit(user) function calls to avoid breaking due to change in the API Thank you for help doveadm: Warning: Event 0x5563f3f2b740 leaked (parent=0x5563f3f2b330): mail-user.c:78 doveadm: Warning: Event 0x5563f3f2b330 leaked (parent=(nil)): mail-storage-service.c:1359 doveadm: Warning: Event 0x5563f4009ef0 leaked (parent=0x5563f3f4cf80): mail-index.c:67 doveadm: Warning: Event 0x5563f40096c0 leaked (parent=0x5563f3f4cf80): fs-api.c:32 doveadm: Warning: Event 0x5563f40092b0 leaked (parent=0x5563f3f4cf80): mail-storage.c:430 doveadm: Warning: Event 0x5563f3f4cf80 leaked (parent=0x5563f3f21f00): mail-user.c:78 doveadm: Warning: Event 0x5563f3f21f00 leaked (parent=(nil)): mail-storage-service.c:1359 doveadm: Warning: Event 0x5563f3f2d080 leaked (parent=0x5563f3f234f0): mail-index.c:67 doveadm: Warning: Event 0x5563f3f2c820 leaked (parent=0x5563f3f234f0): fs-api.c:32 doveadm: Warning: Event 0x5563f3f2c300 leaked (parent=0x5563f3f234f0): mail-storage.c:430 doveadm: Warning: Event 0x5563f3f234f0 leaked (parent=0x5563f3f4e1d0): mail-user.c:78 doveadm: Warning: Event 0x5563f3f4e1d0 leaked (parent=(nil)): mail-storage-service.c:1359 doveadm: Warning: Event 0x5563f3f24d20 leaked (parent=0x5563f3fd58b0): mail-index.c:67 doveadm: Warning: Event 0x5563f3f24620 leaked (parent=0x5563f3fd58b0): fs-api.c:32 doveadm: Warning: Event 0x5563f3f23e30 leaked (parent=0x5563f3fd58b0): mail-storage.c:430 doveadm: Warning: Event 0x5563f3fd58b0 leaked (parent=0x5563f3fd54a0): mail-user.c:78 doveadm: Warning: Event 0x5563f3fd54a0 leaked (parent=(nil)): mail-storage-service.c:1359 doveadm: Warning: Event 0x5563f3fd6df0 leaked (parent=0x5563f3fcc8a0): mail-index.c:67 doveadm: Warning: Event 0x5563f3fd65c0 leaked (parent=0x5563f3fcc8a0): fs-api.c:32 doveadm: Warning: Event 0x5563f3fd5dd0 leaked (parent=0x5563f3fcc8a0): mail-storage.c:430 doveadm: Warning: Event 0x5563f3fcc8a0 leaked (parent=0x5563f3f1c990): mail-user.c:78 doveadm: Warning: Event 0x5563f3f1c990 leaked (parent=(nil)): mail-storage-service.c:1359 doveadm: Warning: Event 0x5563f3f16a20 leaked (parent=0x5563f3fbb710): mail-index.c:67

2 1

Dovecot v2.3.17 released
by Aki Tuomi 04 Nov '21

04 Nov '21

We are pleased to release v2.3.17 of Dovecot. Please note that 2.3.17 release will be the last one to support Debian/Stretch since Bullseye is now supported. https://dovecot.org/releases/2.3/dovecot-2.3.17.tar.gz https://dovecot.org/releases/2.3/dovecot-2.3.17.tar.gz.sig Binary packages in https://repo.dovecot.org/ Docker images in https://hub.docker.com/r/dovecot/dovecot Regards Aki Tuomi Open-Xchange oy --- * Dovecot now logs a warning if time seems to jump forward at least 100 milliseconds. * dict: Lines logged by the dict process now contain the dict name as the prefix. * lib-index: mail_cache_fields, mail_always_cache_fields and mail_never_cache_fields now verifies that the listed header names are valid. Especially the UTF8 "–" character has sometimes been wrongly used instead of the ASCII "-". + *-login: Added login_proxy_rawlog_dir setting to capture rawlogs between proxy and backend. + dict: The server process now keeps the last 10 idle dict backends cached for maximum of 30 seconds. Practically this acts as a connection pool for dict-redis and dict-ldap. Note that this doesn't affect dict-sql, because it already had its own internal cache. + doveadm: New stats add/remove commands added to support changing the metrics configuration on runtime. + lazy_expunge: Added lazy_expunge_exclude settings to disable lazy_expunge for specific folders. \Special-use flags can be used as folder names. + lib-lua: Added a new helper function dovecot.restrict_global_variables() to disable or enable defining new global variables. - LAYOUT=index List index rebuild was missing. - LAYOUT=index: Duplicate GUIDs were not detected. - acl: When using acl_ignore_namespace Dovecot attempted to access or create dovecot-acl-list even when the namespace should have been ignored. For virtual namespaces this could have yielded errors about "Read-only file system" or "Permission denied". - auth: Setting the "master" passdb field to empty value would cause proxying to fail with an authentication error. Now an empty "master" field is ignored. - doveadm-server: Duplicate error lines were sent for failed commands. This didn't normally cause visible problems, except when using wildcards in usernames or -A parameter to go through multiple users. - doveadm-server: Logs written by doveadm-server were often missing log prefixes, especially mail_log_prefix for mail commands. Logs sent to doveadm TCP client were also missing log prefixes. - doveadm: v2.3 regression: batch command always crashes. - doveadm: v2.3.11 regression: Commands failed if ssl_cert or ssl_key files weren't readable by the user running doveadm, even though doveadm didn't actually use these settings - imap-hibernate: Process may crash at deinit: Panic: file ioloop.c: line 928 (io_loop_destroy): assertion failed: (ioloop->cur_ctx == NULL). - imap: Using imap_fetch_failure=no-after can cause assert-crash with some IMAP commands if reading the mail fails (e.g. wrong cached mail size). Fixes: Panic: file index-mail-headers.c: line 198 (index_mail_parse_header_init): assertion failed: (!mail->data.header_parser_initialized) - imap: v2.3.10 regression: When using INDEXPVT to enable private \Seen flags (for shared or public namespaces) the STORE command did not send untagged replies for the \Seen flag changes. - imap: v2.3.15 regression: If PREVIEW/SNIPPET is not the final FETCH option in the command, the IMAP FETCH response is broken. - imap: v2.3.15 regression: MOVE command leaks mailbox if it can't be opened and crashes at deinit: Panic: file mail-user.c: line 229 (mail_user_deinit): assertion failed: ((*user)->refcount == 1). - imapc: Copying nonexistent mail via imapc could have crashed. Fixes: Panic: file mail-storage.c: line 2385 (mailbox_transaction_commit_get_changes): assertion failed: (ret < 0 || seq_range_count(&changes_r->saved_uids) == save_count || array_count(&changes_r->saved_uids) == 0). - indexer: v2.3.15 regression: Process crashes if indexer-client disconnects while it's waiting for command reply. This happened for example if IMAP SEARCH triggered long fts indexing and the IMAP client disconnected while waiting for the reply. - indexer: v2.3.15 regression: Process may have crashed in some situations. - indexer: v2.3.15 regression: indexer-worker processes may not have reached the process_limit in some situations, possibly even using just one indexer-worker process even though there were many indexing requests queued. - lib-compression: Reading lz4 compressed mdbox mails may crash. Fixes: Panic: file istream.c: line 345 (i_stream_read_memarea): assertion failed: (!stream->blocking). - lib-compression: bench-compress crashes due to xz being read-only. - lib-lua: Fix linking libdict_lua for non-GNU linkers when Lua support is disabled. - lib-mail: There was no limit on how large an email header name could be. Processable header names are now limited to 1000 bytes. - lib-oauth2: Dovecot disallowed JWT tokens if their validity time was older than token creation time (nbf < iat). - lib-storage: Reduce memory footprint of certain storage operations. - lib-storage: When listing mailboxes with storage name escape characters (^ or .) as part of the mailbox name, the listing could show corrupted mailbox names. Due to an issue in handling escaped parent folders, the listing of other mailbox names would become corrupted by prepending parts of the previously listed mailboxes parent folder as prefix to the actual mailbox names. The corruption can occur when using LAYOUT=INDEX and maildir or obox, or when using the listescape plugin. - mail-crypt: Fix "-O" argument for "doveadm mailbox cryptokey password" command to be a boolean, and not expect a string. - submission-login: Add support for not authenticating to next hop in submission proxying. - submission-login: EHLO was not sent again after XCLIENT when doing submission proxying. - virtual: Mailboxes do not correctly detect underlying mailboxes getting re-created even though they have a different UIDVALIDITY or GUID.

9 13

2.3.17 broken on CentOS8 / bug
by TG Servers 03 Nov '21

03 Nov '21

7 14

disable sieve replication with dsync
by David Mandelberg 03 Nov '21

03 Nov '21

Hi, I think I got dsync working for mail, but I'm getting these errors: Error: sieve: file storage: mkdir_parents_chgrp(/usr/local/etc/mail/mandelberg.org/david/sieve/tmp) failed: Read-only file system I'm managing sieve files externally, and I don't want any part of dovecot writing to them. Is there any way to configure dsync to not replicate them? I looked at doveadm-sync(1) to see about adding a flag to replication_dsync_parameters, but the closest thing I see is -x for excluding mailboxes. Or maybe -o? Is there a way to specify a nested setting to -o, so I could do something like `plugin { sieve = }` for dsync?

1 0

Re: full-text index and search
by Marc 03 Nov '21

03 Nov '21

Is there some info on what to expect how big these indexes can get (% mailbox)? Is there any differences between solr / xapian storage use? https://github.com/slusarz/dovecot-fts-flatcurve/

2 1

Re: limiting cpu time of dovecot/imap
by Marc 03 Nov '21

03 Nov '21

> > What would be the best way to 'throttle' dovecot/imap processes of specific users? > > > > Can you please elaborate a bit how you would like to ‘throttle’ the processes? Give it less CPU time? > > That you can probably achieve by returning service/imap/executable="/usr/bin/nice -n 10 /usr/libexec/dovecot/imap" > from users for the throttled users. > Thanks Sami, I was experimenting with some external script that applied cpulimit and ionice to processes. But this suggestion is of course much nicer.

1 0

RE: health check passthrough not 100% in combination with haproxy
by Marc 03 Nov '21

03 Nov '21

I was not having the exit trap and the sleep+kill, I don't really get why it is necessary to have these. Looks a bit inefficient to me having a sleep and to kill your own process. But I am not really a bash expert. > > which of course does not make sense... can you try if this works? I'm guessing your script probably looks pretty much like this already... > > #!/bin/bash > timeout=10 > trap "exit 0" QUIT > { > sleep $timeout > kill -3 $$ 2>/dev/null > } & > read -r input > exit_code=$? > if [ ${exit_code} -eq 0 ]; then > echo -n -e "HTTP/1.1 200 OK\r\n" > echo -n -e "Content-Length: 0\r\n" > echo -n -e "\r\n" > fi > exit 0

1 0

dsync error: Received unexpected input d != N
by David Mandelberg 03 Nov '21

03 Nov '21

Hi, I'm trying to set up replication with dsync, and I'm getting errors like this: dsync-local(david(a)mandelberg.org)<...>: Error: dsync(...): Received unexpected input d != N I tried the `doveadm -D sync` command from https://doc.dovecot.org/configuration_manual/replication/#administration and its output looked like: ... dsync-local(david(a)mandelberg.org)<...>: Debug: brain M: Locking done by remote (local hostname=..., remote hostname=...) dsync-local(david(a)mandelberg.org)<...>: Debug: brain M: Local mailbox tree: ... [line above repeated many times for different mailboxes] dsync-local(david(a)mandelberg.org)<...>: Error: dsync(...): Received unexpected input d != N dsync-local(david(a)mandelberg.org)<...>: Error: dsync(...): Received unexpected input d != N dsync-local(david(a)mandelberg.org)<...>: Debug: auth-master: conn unix:/run/dovecot/auth-userdb (pid=47097,uid=0): Disconnected: Connection closed (fd=8) I'm a bit stuck. Does anybody have any ideas what's going on? The output of `doveconf -n` is attached.

1 1

Auth-Trouble
by Hanns Mattes 03 Nov '21

03 Nov '21

Hi, I've installed Dovecot on a freshly installed machine running Freebsd 13.4. Configuration was copied from an earlier installation, which worked perfectly, until I screwed an update. AFAICS things are working, except replication. I see tons of Errors on the remote and the local machine Oct 31 16:15:30 freebsd dovecot[3248]: doveadm(xxx(a)xx.de): Fatal: connect(213.239.197.36:54321) failed: Interrupted system call and some Oct 31 16:47:33 freebsd dovecot[5509]: auth: cram-md5(xxx(a)xx.de,176.199.241.57,<xSyR+afPn9ywx/E5>): Password mismatch Oct 31 16:47:28 freebsd dovecot[5509]: doveadm(xxx(a)yy.de): Fatal: connect(213.239.197.36:54321) failed: Connection refused and we also get Oct 31 18:17:17 freebsd dovecot[934]: imap(xxx(a)wxxx.de)<1649><lWFvO6nPDeewx/E5>: Panic: file array.c: line 10 (array_idx_modifiable_i): assertion failed: (idx < array->buffer->used / array->element_size) Oct 31 18:17:17 freebsd dovecot[934]: imap(xxx(a)wxxx.de)<1649><lWFvO6nPDeewx/E5>: Fatal: master: service(imap): child 1649 killed with signal 6 (core not dumped - https://dovecot.org/bugreport.html#coredumps - set service imap { drop_priv_before_exec=yes }) not to forget Oct 31 18:18:33 freebsd dovecot[934]: doveadm(xxx@xxx): Fatal: connect(213.239.197.36:54321) failed: Connection refused Users are authenticating with ldap. I'm clueless, and I don't have any clue, if it is a misconfiguration of dovecot or my freebsd-install. Any ideas appreciated Here is the output of doveconf -n # 2.3.17 (e2aa53df5b): /usr/local/etc/dovecot/dovecot.conf # Pigeonhole version 0.5.17 (054dddfa) # OS: FreeBSD 13.0-RELEASE-p4 amd64 # Hostname: freebsd.bruecko.de auth_mechanisms = plain login digest-md5 cram-md5 apop auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890+.-_@ auth_verbose = yes doveadm_password = # hidden, use -P to show it doveadm_port = 54321 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes lda_original_recipient_header = X-Original-To lmtp_save_to_detail_mailbox = yes mail_location = mdbox:~/mdbox mail_plugins = " quota fts fts_xapian trash zlib notify replication acl" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { hidden = no ignore_on_failure = no inbox = no list = children location = mdbox:%%h/mdbox prefix = shared/%%u/ separator = / subscriptions = yes type = shared } namespace { location = mdbox:/virtualmail/public:INDEXPVT=%h/mdbox/Public prefix = Public/ separator = / subscriptions = yes type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = INBOX/ separator = / } passdb { args = /usr/local/etc/dovecot/master-users driver = passwd-file master = yes pass = yes } passdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile acl_anyone = allow acl_shared_dict = file:/virtualmail/shared-mailboxes.db fts = xapian fts_autoindex = yes fts_autoindex_exclude = \Trash fts_enforced = yes fts_xapian = partial=3 full=20 verbose=0 mail_replica = tcps:mail.bruecko.de quota = dict:User quota::file:%h/dovecot-quota quota_exceeded_message = Storage quota for this account has been exceeded, please try again later. quota_grace = 250M quota_rule = *:storage=2500M quota_rule2 = INBOX/Trash:storage=+10%% quota_status_nouser = DUNNO quota_status_overquota = 552 5.2.2 Mailbox is full / Mailbox ist voll quota_status_success = DUNNO quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=80%% quota-warning 80 %u quota_warning3 = storage=75%% quota-warning 75 %u sieve = ~/dovecot.sieve sieve_dir = ~/sieve sieve_global_dir = /virtualmail sieve_max_actions = 0 sieve_max_redirects = 128 sieve_max_script_size = 0 sieve_quota_max_scripts = 0 trash = /usr/local/etc/dovecot/dovecot-trash.conf.ext } postmaster_address = admin(a)bruecko.de protocols = imap pop3 lmtp sieve replication_dsync_parameters = -d -N -l 30 -U -x Public service aggregator { fifo_listener replication-notify-fifo { user = vmail } unix_listener replication-notify { user = vmail } } service auth { unix_listener /var/spool/postfix/private/auth { mode = 0666 } unix_listener auth-userdb { group = vmail user = vmail } } service decode2text { executable = script /usr/local/libexec/dovecot/decode2text.sh unix_listener decode2text { mode = 0666 } user = dovecot } service doveadm { inet_listener { port = 54321 ssl = yes } vsz_limit = 0 } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service indexer-worker { vsz_limit = 2 G } service lmtp { inet_listener lmtp { address = 127.0.0.1 port = 24 } user = vmail } service managesieve-login { inet_listener sieve { port = 4190 } inet_listener sieve_deprecated { port = 2000 } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service quota-status { client_limit = 1 executable = quota-status -p postfix inet_listener { port = 12340 } } service quota-warning { executable = script /usr/local/bin/quota-warning.sh unix_listener quota-warning { user = vmail } user = vmail } service replicator { process_min_avail = 1 unix_listener replicator-doveadm { mode = 0666 } } ssl_cert = </usr/local/etc/dovecot/fullchain17.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_client_ca_dir = /etc/ssl/certs ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes userdb { driver = prefetch } userdb { args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext driver = ldap } verbose_proctitle = yes protocol lmtp { mail_plugins = " quota fts fts_xapian trash zlib notify replication acl sieve" } protocol lda { mail_plugins = " quota fts fts_xapian trash zlib notify replication acl sieve" } protocol imap { mail_max_userip_connections = 100 mail_plugins = " quota fts fts_xapian trash zlib notify replication acl imap_quota imap_zlib imap_acl" } Regards Hanns

2 7