February 2012 - dovecot

Re: [Dovecot] [Dovecot 2.1] imap service crash when using some clients
by interfaSys sàrl 16 Feb '12

16 Feb '12

Hello Pascal, I've just tested rc6 and get almost the same results. imap still crashes, but Horde still works. I forgot to mention that this is on FreeBSD 8 using gcc 4.6 and LTO. And contrary to what I said in my initial email, it doesn't seem to be ACL plugin related. imap crashes even if it's not in the plugins list. Here is the backtrace: # gdb /usr/libexec/dovecot/imap /path/to/core Core was generated by `imap'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/dovecot/libdovecot-storage.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/libdovecot-storage.so.0 Reading symbols from /usr/lib/dovecot/libdovecot.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/libdovecot.so.0 Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/librt.so.1 Reading symbols from /usr/lib/dovecot/lib01_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib01_acl_plugin.so Reading symbols from /usr/lib/dovecot/lib02_imap_acl_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib02_imap_acl_plugin.so Reading symbols from /usr/lib/dovecot/lib10_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib10_quota_plugin.so Reading symbols from /usr/lib/dovecot/lib11_imap_quota_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib11_imap_quota_plugin.so Reading symbols from /usr/lib/dovecot/lib11_trash_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib11_trash_plugin.so Reading symbols from /usr/lib/dovecot/lib20_autocreate_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib20_autocreate_plugin.so Reading symbols from /usr/lib/dovecot/lib20_fts_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib20_fts_plugin.so Reading symbols from /usr/lib/dovecot/lib20_zlib_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib20_zlib_plugin.so Reading symbols from /lib/libz.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.5 Reading symbols from /usr/lib/libbz2.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libbz2.so.4 Reading symbols from /usr/lib/dovecot/lib21_fts_squat_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib21_fts_squat_plugin.so Reading symbols from /usr/lib/dovecot/lib30_imap_zlib_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib30_imap_zlib_plugin.so Reading symbols from /usr/lib/dovecot/lib90_antispam_plugin.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/dovecot/lib90_antispam_plugin.so Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x00000000105c64e7 in imap_match () from /usr/lib/dovecot/libdovecot.so.0 (gdb) bt full #0 0x00000000105c64e7 in imap_match () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #1 0x00000000104b4bf6 in mailbox_list_iter_next () from /usr/lib/dovecot/libdovecot-storage.so.0 No symbol table info available. #2 0x0000000000416b99 in cmd_list_continue.6853 () No symbol table info available. #3 0x000000000042166d in cmd_list_full () No symbol table info available. #4 0x0000000000000000 in ?? () No symbol table info available. #5 0x8080808080808080 in ?? () No symbol table info available. #6 0x0000000000000004 in ?? () No symbol table info available. #7 0x0000000010a3f1a8 in ?? () No symbol table info available. #8 0x0000000010aa4390 in ?? () No symbol table info available. #9 0x0000000010a3f174 in ?? () No symbol table info available. #10 0x00007fffffffe9a0 in ?? () No symbol table info available. #11 0x0000000010596e4f in p_strdup () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #12 0x0000000010a22200 in ?? () No symbol table info available. #13 0x0000000010aa4458 in ?? () No symbol table info available. #14 0x0000000010aa4340 in ?? () No symbol table info available. #15 0x0000000010aa4498 in ?? () No symbol table info available. #16 0x0000000110a3f040 in ?? () No symbol table info available. #17 0x0000000010a3f128 in ?? () No symbol table info available. #18 0x0000000010a3e840 in ?? () No symbol table info available. #19 0x0000000010a3f040 in ?? () No symbol table info available. #20 0x0000000010a22200 in ?? () No symbol table info available. #21 0x0000000000000001 in ?? () No symbol table info available. #22 0x0000000010a22200 in ?? () No symbol table info available. #23 0x0000000000000000 in ?? () No symbol table info available. #24 0x000000000041d320 in client_command_input.12433 () No symbol table info available. #25 0x00000000004203ca in client_handle_input () No symbol table info available. #26 0x0000000010a574e0 in ?? () No symbol table info available. #27 0x00000000105ae219 in i_stream_read () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #28 0x000000000041d13f in client_input () No symbol table info available. #29 0x00000000105ce276 in io_loop_call_io () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #30 0x00000000105c9ea3 in io_loop_handler_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #31 0x00000000105c9f78 in io_loop_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #32 0x00000000105c9fb3 in master_service_run () from /usr/lib/dovecot/libdovecot.so.0 No symbol table info available. #33 0x00000000004158bb in main () No symbol table info available. Regards, Olivier > On 01/18/2012 02:03 AM interfaSys sàrl wrote: > >/ Hello, > />/ > />/ I've just noticed that when Horde is connecting to Dovecot 2.1, it > />/ crashes the imap service if Dovecot is configured to use the ACL plugin. > />/ I'm not sure what's so special about the command Horde sends, but it > />/ shouldn't make Dovecot crash. Everything is fine when using Thunderbird. > />/ > />/ Here is the message in Dovecot's logs > />/ "Fatal: master: service(imap): child 89974 killed with signal 11 (core > />/ not dumped)" > />/ > />/ The message says that the core is not dumped, even though I did add > />/ drop_priv_before_exec=yes to my config file. > / > dovecot stop > ulimit -c unlimited > dovecot > > Now connect with Horde and let it crash. > > >/ I've tried connecting to the pid using gdb, but the process just hangs > />/ as soon as I'm connected. > />/ > / > continue > [wait for the crash] > bt full > detach > quit > > > Regards, > Pascal

2 7

[Dovecot] v2.1.0 released
by Timo Sirainen 16 Feb '12

16 Feb '12

http://dovecot.org/releases/2.1/dovecot-2.1.0.tar.gz http://dovecot.org/releases/2.1/dovecot-2.1.0.tar.gz.sig Here's finally the the stable v2.1 release. I expect this release to be quite stable, since it's already being used in several large installations. Changes since rc7: FS layout related crashfix, two man pages added. The biggest changes since v2.0: * Plugins now use UTF-8 mailbox names rather than mUTF-7: acl, autocreate, expire, trash, virtual * auth_username_format default changed to %Lu. If you really want case sensitive usernames, set it back to empty. * Solr full text search backend changed to use mailbox GUIDs instead of mailbox names, requiring reindexing everything. solr_old backend can be used with old indexes to avoid reindexing, but it doesn't support some newer features. * Expire plugin: Only go through users listed by userdb iteration. Delete dict rows for nonexistent users, unless expire_keep_nonexistent_users=yes. * Temporary authentication failures sent to IMAP/POP3 clients now includes the server's hostname and timestamp. This makes it easier to find the error message from logs. * dsync was merged into doveadm. There is still "dsync" symlink pointing to "doveadm", which you can use the old way for now. The preferred ways to run dsync are "doveadm sync" (for old "dsync mirror") and "doveadm backup". + imapc (= IMAP client) storage allows using a remote IMAP server to be used as storage. This allows using Dovecot as a smart (caching) proxy or using dsync to do migration from remote IMAP server. + Mailbox indexing via queuing indexer service (required for Lucene) + Lucene full text search (FTS) backend rewritten with support for different languages + FTS finally supports "OR" search operation + FTS supports indexing attachments via external programs + IMAP FUZZY extension, supported by Lucene and Solr FTS backends + IMAP SPECIAL-USE extension to describe mailboxes + Mailbox list indexes + Statistics tracking via stats service. Exported via doveadm stats. + Autocreate plugin creates/subscribes mailboxes physically only when the mailbox is opened for the first time. Mailbox listing shows the autocreated mailboxes even if they don't physically exist. + Password and user databases now support default_fields and override_fields settings to specify template defaults/overrides. + SCRAM-SHA-1 authentication mechanism by Florian Zeitz + LDAP: Allow building passdb/userdb extra fields from multiple LDAP attributes by using %{ldap:attributeName} variables in the template. + Improved multi-instance support: Track automatically which instances are started up and manage the list with doveadm instance commands. All Dovecot commands now support -i <instance_name> parameter to select the instance (instead of having to use -c <config path>). See instance_name setting. + auth: Implemented support for Postfix's "TCP map" sockets for user existence lookups. - listescape plugin works perfectly now

3 7

[Dovecot] There is a way to know if a email has been expunged ?
by Alex Baule 16 Feb '12

16 Feb '12

Hy Everyone...and Timo ! There is a way to know if a email was expunged (deleted from hard disk) inside a plugin ? I rewrite the expunge function, but the real expunge function is void, i can't know if was really expunged. There is a way to know this ?

2 8

[Dovecot] Any possibility of running query after sucessful login?
by Jacek Osiecki 16 Feb '12

16 Feb '12

Hi, I'm using finally dovecot 2.0 after quite a smooth upgrade from 1.0 :) with virtual mailboxes stored in sql database. Sometimes I need to check if a mailbox has been used recently or not. Checking for logins in dovecot logs sometimes means searching through gigabytes of info... I was just wondering if there is any possibility of running another query after successful login - just to fill some extra field like "last_login"? So far my only idea is to replace user/password query with procedure like login(user,password) that would also return a line as required by user/password query. However, some mysql implementations (for example, standard php-mysql) can't handle procedures which return values - so will it work in dovecot? Or maybe some other ideas how to do that? Greetings, -- Jacek Osiecki joshua(a)ceti.pl GG:3828944 I don't want something I need. I want something I want.

2 4

[Dovecot] index synchronization during local delivery
by Andy YB Hu 16 Feb '12

16 Feb '12

Hi there, Now our system will take dovecot as the whole imap back end, and keep the LDA of ours, But our system has our own UID generation mechanism (in dovecot it's incremental from 1), which will be done while local delivery, So when local deliveries, it will first generate the UID, move the mail into maildir, then synchronize the mailbox index, which is part of dovecot LDA does. The question is what API or something else about index synchronization I can integrate into our own LDA program easily? Seems mailbox_copy() would be the possibility? Thanks.

2 7

Re: [Dovecot] 2.1b6 imapc sync error
by Metro Domain Admin 15 Feb '12

15 Feb '12

On Wed, Feb 15, 2012, at 10:50 PM, Timo Sirainen wrote: > On 15.2.2012, at 22.48, Metro Domain Admin wrote: > > > On Wed, Feb 15, 2012, at 09:36 PM, Timo Sirainen wrote: > >> Yes, it's because of "sync". The "Not supported" comes from imapc > >> backend, because Dovecot tried to do some changes to it that weren't > >> possible with IMAP protocol (modseq changes probably). I think the > >> migration still was successful though. > > > > The migration went beautifully otherwise. But are the errors minor > > enough that I can continue with sync? > > Probably better to continue with "backup" just in case. :) > > > I had the idea that I could sync back to the imapc account, as a > > fallback option (and, presumably I'd want to preserve the /tmp/imapc > > directory for this?). > > That's not something that I've tested, and I don't know how well it works > (if at all). > OK, "backup" it is then. Thanks.

1 0

[Dovecot] MySQL server has gone away
by Mark Moseley 15 Feb '12

15 Feb '12

I'm running 2.0.17 and I'm still seeing a decent amount of "MySQL server has gone away" errors, despite having multiple hosts defined in my auth userdb 'connect'. This is Debian Lenny 32-bit and I'm seeing the same thing with 2.0.16 on Debian Squeeze 64-bit. E.g.: Jan 12 20:30:33 auth-worker: Error: mysql: Query failed, retrying: MySQL server has gone away Our mail mysql servers are busy enough that wait_timeout is set to a whopping 30 seconds. On my regular boxes, I see a good deal of these in the logs. I've been doing a lot of mucking with doveadm/dsync (working on maildir->mdbox migration finally, yay!) on test boxes (same dovecot package & version) and when I get this error, despite the log saying it's retrying, it doesn't seem to be. Instead I get: dsync(root): Error: user ...: Auth USER lookup failed dsync(root): Fatal: User lookup failed: Internal error occurred. Refer to server log for more information. Watching tcpdump at the same time, it looks like it's going through some of the mysql servers, but all of them have by now disconnected and are in CLOSE_WAIT. Here's an (edited) example after doing a dsync that completes without errors, with tcpdump running in the background: # sleep 30; netstat -ant | grep 3306; dsync -C^ -u mailbox(a)test.com backup mdbox:~/mdbox tcp 1 0 10.1.15.129:57436 10.1.52.48:3306 CLOSE_WAIT tcp 1 0 10.1.15.129:49917 10.1.52.49:3306 CLOSE_WAIT tcp 1 0 10.1.15.129:35904 10.1.52.47:3306 CLOSE_WAIT 20:49:59.725005 IP 10.1.15.129.35904 > 10.1.52.47.3306: F 1126:1126(0) ack 807 win 1004 <nop,nop,timestamp 312603858 766667259> 20:49:59.725459 IP 10.1.52.47.3306 > 10.1.15.129.35904: . ack 1127 win 123 <nop,nop,timestamp 766667998 312603858> 20:49:59.725568 IP 10.1.15.129.57436 > 10.1.52.48.3306: F 1126:1126(0) ack 807 win 1004 <nop,nop,timestamp 312603858 1842560856> 20:49:59.725779 IP 10.1.52.48.3306 > 10.1.15.129.57436: . ack 1127 win 123 <nop,nop,timestamp 1842561225 312603858> dsync(root): Error: user mailbox(a)test.com: Auth USER lookup failed dsync(root): Fatal: User lookup failed: Internal error occurred. Refer to server log for more information. 10.1.15.129 in this case is the dovecot server, and the 10.1.52.0/24 boxes are mysql servers. That's the same pattern I've seen almost every time. Just a FIN packet to two of the servers (ack'd by the mysql server) and then it fails. Is the retry mechanism supposed to transparently start a new connection, or is this how it works? In connecting remotely to these same servers (which aren't getting production traffic, so I'm the only person connecting to them), I get seemingly random disconnects via IMAP, always coinciding with a "MySQL server has gone away" error in the logs. This is non-production, so I'm happy to turn on whatever debugging would be useful. Here's doveconf -n from the box the tcpdump was on. This box is just configured for lmtp (but have seen the same thing on one configured for IMAP/POP as well), so it's pretty small, config-wise: # 2.0.17: /etc/dovecot/dovecot/dovecot.conf # OS: Linux 3.0.9-nx i686 Debian 5.0.9 auth_cache_negative_ttl = 0 auth_cache_ttl = 0 auth_debug = yes auth_failure_delay = 0 base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot/debug.log default_client_limit = 3005 default_internal_user = doveauth default_process_limit = 1500 deliver_log_format = M=%m, F=%f, S="%s" => %$ disable_plaintext_auth = no first_valid_uid = 199 last_valid_uid = 201 lda_mailbox_autocreate = yes listen = * log_path = /var/log/dovecot/mail.log mail_debug = yes mail_fsync = always mail_location = maildir:~/Maildir:INDEX=/var/cache/dovecot/%2Mu/%2.2Mu/%u mail_nfs_index = yes mail_nfs_storage = yes mail_plugins = zlib quota mail_privileged_group = mail mail_uid = 200 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave mdbox_rotate_interval = 1 days mmap_disable = yes namespace { hidden = no inbox = yes list = yes location = prefix = INBOX. separator = . subscriptions = yes type = private } passdb { args = /opt/dovecot/etc/lmtp/sql.conf driver = sql } plugin { info_log_path = /var/log/dovecot/dovecot-deliver.log log_path = /var/log/dovecot/dovecot-deliver.log quota = maildir:User quota quota_rule = *:bytes=25M quota_rule2 = INBOX.Trash:bytes=+10%% quota_rule3 = *:messages=3000 sieve = ~/sieve/dovecot.sieve sieve_before = /etc/dovecot/scripts/spam.sieve sieve_dir = ~/sieve/ zlib_save = gz zlib_save_level = 3 } protocols = lmtp sieve service auth-worker { unix_listener auth-worker { mode = 0666 } user = doveauth } service auth { client_limit = 8000 unix_listener login/auth { mode = 0666 } user = doveauth } service lmtp { executable = lmtp -L process_min_avail = 10 unix_listener lmtp { mode = 0666 } } ssl = no userdb { driver = prefetch } userdb { args = /opt/dovecot/etc/lmtp/sql.conf driver = sql } verbose_proctitle = yes protocol lmtp { mail_plugins = zlib quota sieve } Thanks!

4 29

[Dovecot] 2.1b6 imapc sync error
by Metro Domain Admin 15 Feb '12

15 Feb '12

I received this error at the end of a migration from imapc to mdbox. Is it because I wanted doveadm 'sync' instead of 'backup'? I didn't get time to do any testing, but since you're pushing for the release, here it is: kerio2:~ adminuser$ /opt/dovecot/bin/doveadm -o imapc_user=jsample -o imapc_password=****** -o imapc_host=*.*.*.* sync -R -u jsample_dc(a)mydomain.com imapc:/tmp/imapc dsync(jsample_dc(a)mydomain.com): Error: Can't update mailbox Archives/2011: Not supported dsync(jsample_dc(a)mydomain.com): Error: Can't update mailbox Archives/2012: Not supported dsync(jsample_dc(a)mydomain.com): Error: Can't update mailbox INBOX: Not supported ...etc, for all mailboxes in the account. This is my userdb entry: jsample_dc@mydomain.com:{***}password:502:6:IMAPC RECIPIENT::/bin/false: kerio2:~ adminuser$ /opt/dovecot/bin/doveconf -n # 2.1.rc6: /opt/dovecot-2.1.rc6/etc/dovecot/dovecot.conf # OS: Darwin 9.8.0 Power Macintosh auth_cache_negative_ttl = 0 auth_cache_size = 1 M auth_debug = yes auth_mechanisms = plain login auth_username_format = %n(a)mydomain.com auth_verbose = yes disable_plaintext_auth = no instance_name = dovecot-2.1.rc6 listen = 192.168.20.40 mail_home = /var/mail/%d/%n mail_location = mdbox:~/mdbox mail_plugin_dir = /opt/dovecot/lib/dovecot mail_plugins = fts fts_squat # I know, this is deprecated in 2.1 mail_privileged_group = mail maildir_very_dirty_syncs = yes namespace inbox { inbox = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = } passdb { args = scheme=CRYPT username_format=%u /opt/dovecot/etc/dovecot/users driver = passwd-file } plugin { fts = squat } postmaster_address = postmaster(a)mydomain.com protocols = imap lmtp service auth { unix_listener /var/spool/postfix/private/auth { group = _postfix mode = 0660 user = _postfix } unix_listener auth-userdb { group = mail mode = 0660 user = adminuser } } service imap { executable = imap postlogin } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = _postfix mode = 0666 user = _postfix } } service postlogin { executable = script-login rawlog -t } service stats { fifo_listener stats-mail { mode = 00 } } ssl_ca = </opt/local/apache2/conf/ssl/gd_bundle.crt ssl_cert = </opt/local/apache2/conf/ssl/server4.crt ssl_key = </opt/local/apache2/conf/ssl/server4.key userdb { args = username_format=%u /opt/dovecot/etc/dovecot/users driver = passwd-file } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 16 } kerio2:~ adminuser$

2 1

[Dovecot] domain quota in sql
by Adam Szpakowski 15 Feb '12

15 Feb '12

Hi, I have a problem with setting up quota usage tracking for domains. Problem is, that domain quota is always based on only one user in domain, the last one on whom behalf any operation was performed. To recreate this situation, one has only to perform: doveadm quota recalc -u USER on a user, and the domain quota takes the value of this user quota. Have anyone a clue what could cause this problem? Quota tracking is set up both for users and domains. For users quota tracking is working fine, data is saved into dbase without problems. We are using dovecot 2.0.15. Quota enforcing (rules) is NOT a part of the problem. Important parts of config: dict { quota = mysql:/etc/dovecot/dovecot-dict-used-quota.conf.ext quota2 = mysql:/etc/dovecot/dovecot-dict-used-quota-domain.conf.ext } plugin { quota = dict:User quota::proxy::quota quota2 = dict:Domain quota:%d:proxy::quota2 } /etc/dovecot/dovecot-dict-used-quota.conf.ext map { pattern = priv/quota/storage table = used_quota username_field = username value_field = bytes } map { pattern = priv/quota/messages table = used_quota username_field = username value_field = messages } /etc/dovecot/dovecot-dict-used-quota-domain.conf.ext map { pattern = priv/quota/storage table = used_quota_domain username_field = domain value_field = bytes } map { pattern = priv/quota/messages table = used_quota_domain username_field = domain value_field = messages } -- Adam Szpakowski

4 10

[Dovecot] Dovecot-2.0.18 and Dovecot-Pigeonwhole-0.2.6
by Ralf Zimmermann 15 Feb '12

15 Feb '12

Hi all, today I have compiled Dovecot-2.0.18 with Dovecot-Pigeonwhole-0.2.6. Now I have following Problem with sieve: ------------------------%<--------------------------------------------- main_script: line 249: error: included personal script 'vacation.sieve' does not exist. main_script: error: validation failed. ------------------------%<--------------------------------------------- I have a definition for 'sieve_dir' and the sieve script 'vacation.sieve' exits in the folder. # Dovecot Sieve Parameters plugin { ... sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } If I use Dovecot-2.0.17 and Dovecot-Pigeonwhole-0.2.5 with the same configuration I can define include :personal "vacation.sieve"; in my main sieve script. All works fine. Greets Ralf

3 3