November 2012 - dovecot

[Dovecot] Commercial features in Dovecot future: Object storage, archive
by Timo Sirainen 13 Nov '12

13 Nov '12

Hi all, Dovecot Oy’s web pages at www.dovecot.fi have been updated. The products page lists two features that will be available for commercial licensing, extending the functionality of the basic open-source version of Dovecot. * Storing emails to (high-latency) object storage, initially supporting Amazon S3, Caringo CAStor and Scality. * Email archive storage. See http://www.dovecot.fi/products/index.html for details. I’ve been developing Dovecot for over 10 years now. For a long time it was my primary motivation in life to create the best IMAP server available :) I think I've pretty much accomplished that by now. The future is looking very bright for Dovecot: we will continue the open source development stronger than ever, but in addition, for the long term it needs some additional licensed components that bring the money to cover the cost for future Dovecot development and to be able to build up the support in a professional way. These new features will be added as plugins on top of Dovecot to extend the functionality. Note that I’m not just randomly choosing which features will be open and which will be licensed. Only some specific features will be licensed where my company is going to make money with partnerships and in other measurable ways.

5 6

[Dovecot] mbox vs. maildir storage block waste
by Christoph Anton Mitterer 13 Nov '12

13 Nov '12

Hi. I recently mentioned in several posts, that I'd tended to use mbox rather than maildir, because you don't loose so much space (due to always allocating full blocks per maildir file and thus per mail). I made some tests of my archive, which consists of some 3,4 million mails at a total of 42GB). Most of these mails are probably normal sized, but there are also some with bigger attachments. For those who are interested here are the results: I used a 53687091200 B image file (via loop device) and tested ext4 only. btrfs is IMHO not yet ready, I have had often issues with XFS (corruptions), reiser4 is more or less dead and reiser3 is said to have issues (see e.g. its wikipedia article, even though it has that mode for small files which would fit nicely). As you see the number of mails increased a bit, cause I tested over several days... but this is only a very small increase so it shouldn't change the numbers a lot. 1) Original mbox archives (right now in Evolution) mbox exact space: 38122676224 (does not include meta-data) mbox guess space: 44625670144 (includes Evolution meta-data which is several GBs) mbox num mails: 3412999 (occurances of From_ lines) In the following: - image file, 1B-blocks, Used_begin, Used_end, Available_begin, Available_end result out of df -B 1 - mdir exact used space is the sum of du -B 1 for each regular file (i.e. each mdir file) - mdir guess used space du -B 1 on the root dir of the filesystem - mdir num mails: find . type -f | wc -l on the root dir of the filesystem 2) EXT4 with 4096 blocks: image file: 53687091200 1B-blocks: 52844687360 Used_begin: 188555264 Used_end: 45198778368 Available_begin: 49971777536 Available_end: 2444972032 mdir exact used space: 44810866688 mdir guess used space: 45010243584 mdir num mails: 3423296 delta: 6.688190464 G delta / mail: 1953 B 3) EXT4 with 2048 blocks: image file: 53687091200 1B-blocks: 50324295680 Used_begin: 82857984 Used_end: 41598846976 Available_begin: 47557083136 Available_end: 6041094144 mdir exact used space: 41323991040 mdir guess used space: 41516007424 mdir num mails: 3425033 delta: 3.201314816 G delta / mail: 934 B 4) EXT4 with 1024 blocks: image file: 53687091200 1B-blocks: 50314834944 Used_begin: 38287360 Used_end: 39909360640 Available_begin: 47592193024 Available_end: 7721119744 mdir exact used space: 39683908608 mdir guess used space: 39871086592 mdir num mails: 3425033 delta: 1.561232384 G delta / mail: 455 B As you can see, the delta per mail is rather close to the statistically expected values of 2048B, 1024B and 512B. In the end I probably changed my opinion. ~7GB of wasted block space for all my mails is actually quite a lot, but in days of cheap disk space it's acceptable. And with mbox one has IMHO the major disadvantage that mailservers (including dovecot) store some meta-data _in_ it (i.e. in the mails themselves) , which I don't like a lot. I still think about reports that mbox is much faster with full text search (which sounds reasonable)... but therefore one needs probably and database backend anyway. HTH, Chris.

7 21

[Dovecot] Invalid Managesieve commands are counted twice
by Christoph Bußenius 12 Nov '12

12 Nov '12

Hi, the Managesieve server closes the connection if it receives an unknown command before authentication: "IMPLEMENTATION" "Dovecot Pigeonhole" "SIEVE" "fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave" "NOTIFY" "mailto" "SASL" "PLAIN" "STARTTLS" "VERSION" "1.0" OK "Dovecot ready." --> BOGUS NO "Error in MANAGESIEVE command received by server." NO "Error in MANAGESIEVE command received by server." BYE "Too many invalid MANAGESIEVE commands." Connection closed by foreign host. Note that only one bogus command has been sent by the client, however the server sends two identical error messages. This seems to be a bug in Pigeonhole 0.3.3. In version 0.2.6, the connection was kept open after the error message. This is actually important to us because we use the "sieveshell" utility which is shipped with the Python "managesieve" package. The managesieve.py module always sends a BOGUS command after the TLS handshake. According to its comments, this is done to work around problems with other server implementations: # Some servers send capabilities after TLS handshake, some # do not. We send a bogus command, and expect a NO. If you # get something else instead, read the extra NO to clear # the buffer. typ, data = self._command('BOGUS') (The full source is at http://pydoc.net/managesieve/0.4.2/managesieve) As a result, sieveshell cannot be used with TLS and a current Dovecot/Pigeonhole server. Cheers, Christoph -- Christoph Bußenius Rechnerbetriebsgruppe der Fakultäten Informatik und Mathematik Technische Universität München +49 89-289-18519 <> Raum 00.05.040 <> Boltzmannstr. 3 <> Garching

2 2

[Dovecot] Help me with IMAP config
by Tibby 12 Nov '12

12 Nov '12

Hello! I have dovecot running. Imap works fine. When i connect with outlook an when i delete a message it gets crossed out but still stays in my mailbox. I want to disable this feature. I want it once its deleted then go to trash folder on the mail server and thats it. Is there an option for this? I'm running dovecot 1.2.15 Thanks!

2 1

[Dovecot] Sieve users script problem.
by Eliezer Croitoru 12 Nov '12

12 Nov '12

I am using Dovecot 2.1.9 which seems to solve older bug in sieve. Now I have another problem with users scripts. I want dovecot to run users scripts but it seems to not even see them. Dovecot runs the default sieve script from home dir but wont see the sieve scripts in the sieve subdirectory and some logs: Nov 10 18:35:54 lda(user(a)domain.local): Debug: Quota root: name=User quota backend=maildir args= Nov 10 18:35:54 lda(user(a)domain.local): Debug: Quota rule: root=User quota mailbox=* bytes=1073741824 messages=0 Nov 10 18:35:54 lda(user(a)domain.local): Debug: Quota rule: root=User quota mailbox=Trash bytes=+32212254 (3%) messages=0 Nov 10 18:35:54 lda(user(a)domain.local): Debug: Quota warning: bytes=1020054732 (95%) messages=0 reverse=no command=quota-warning 95 raw mail user Nov 10 18:35:54 lda(user(a)domain.local): Debug: Quota warning: bytes=214748364 (20%) messages=0 reverse=no command=quota-warning 20 raw mail user Nov 10 18:35:54 lda(user(a)domain.local): Debug: none: root=, index=, control=, inbox=, alt= Nov 10 18:35:54 lda(user(a)domain.local): Debug: Destination address: eliezer(a)domain.local (source: user@hostname) Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: include: sieve_global_dir is not set; it is currently not possible to include `:global' scripts. Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: using the following location for user's Sieve script: /home/vmail/domain.local/eliezer/home/.dovecot.sieve Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: opening script /home/vmail/domain.local/eliezer/home/.dovecot.sieve Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: script binary /home/vmail/domain.local/eliezer/home/.dovecot.svbin successfully loaded Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: binary save: not saving binary /home/vmail/domain.local/eliezer/home/.dovecot.svbin, because it is already stored Nov 10 18:35:54 lda(user(a)domain.local): Debug: sieve: executing script from /home/vmail/domain.local/eliezer/home/.dovecot.svbin Nov 10 18:35:54 lda(user(a)domain.local): Info: sieve: msgid=<XXXX@XXXX>: stored mail into mailbox 'INBOX' So it recognizes the home sir but wont look at the sieve sub directory. My dovecot -n output: # 2.1.9: /etc/dovecot/dovecot.conf # OS: Linux 3.3.8-gentoo x86_64 Gentoo Base System release 2.0.3 ext3 auth_mechanisms = plain login dict { quota = mysql:/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no first_valid_uid = 5000 last_valid_uid = 5000 log_path = /var/log/dovecot.log mail_debug = yes mail_gid = 5000 mail_location = maildir:/home/vmail/%d/%n/Maildir/ mail_plugins = " quota" mail_privileged_group = vmail mail_uid = 5000 namespace { inbox = yes list = yes location = prefix = separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { quota = maildir:User quota quota_rule = *:storage=1G quota_rule2 = Trash:storage=+3%% quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=20%% quota-warning 20 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve sieve_extensions = +include +vnd.dovecot.debug +imapflags +spamtest +spamtestplus +relational +comparator-i;ascii-numeric } postmaster_address = postmastert(a)domain.local service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0660 user = postfix } unix_listener auth-userdb { group = vmail mode = 0600 user = vmail } } service dict { unix_listener dict { group = vmail mode = 0600 user = vmail } } service imap-login { inet_listener imap { port = 143 } inet_listener imaps { port = 993 ssl = yes } } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0660 user = postfix } } service pop3-login { inet_listener pop3 { port = 110 } inet_listener pop3s { port = 995 ssl = yes } } service quota-warning { executable = script /etc/dovecot/quota-warning.sh user = vmail } ssl_ca = </etc/ssl/dca.pem ssl_cert = </etc/ssl/ssl.crt ssl_key = </etc/ssl/ssl.key userdb { args = /etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = " quota sieve" } protocol imap { mail_plugins = " quota imap_quota" } protocol pop3 { pop3_client_workarounds = outlook-no-nuls oe-ns-eoh pop3_uidl_format = %08Xu%08Xv } Thanks, Eliezer

2 2

[Dovecot] Solr 4.0 - lucene - FTS
by Charles Marcus 12 Nov '12

12 Nov '12

Hi Timo, As one who is interested in implementing FTS sometime in the future, I'm curious about what is in store as far as improvements go... Specifically, any plans for implementing immediate/automatic index updates at delivery time? The lack of automatically updated indexes is one downside for its implementation... Also, does the release of Solr 4.0 mean anything for the lucene library used by dovecot? http://www.marketwatch.com/story/lucidworks-congratulates-apache-foundation… Thanks, -- Best regards, Charles

3 6

[Dovecot] Sieve puts incoming message into inbox on any problem with submission_host
by Christian Rohmann 12 Nov '12

12 Nov '12

Hello dovecot-users, I have a question/suggestion regarding the submission_host feature of the lda (either via dovecot-lda binary or lmtp) in combination with sieve. When trying to deliver message to a mailbox and this message has a sieve redirect action applied to it, dovecot is using the configured smtp server to forward the message just fine. Unfortunately, if the server configured for submission is unavailable, returns an error or cannot be found via DNS, sieve will simply put the message into the inbox and be done with it: --- dovecot.log --- dovecot: lda(user1): Error: lmtp client: DNS lookup of idonotexist.example.com failed: Name or service not known dovecot: lda(user1): Error: sieve: msgid=unspecified: failed to redirect message to <user2> (refer to server log for more information) dovecot: lda(user1): sieve: msgid=unspecified: stored mail into mailbox 'INBOX' dovecot: lda(user1): Error: sieve: execution of script /mailboxes/user1/.dovecot.sieve failed, but implicit keep was successful (user logfile /mailboxes/user1/.dovecot.sieve.log may reveal additional details) --- /dovecot.log --- The same applies to vacation messages being sent out. Especially with an (unconditional) redirect action, users don't expect to find messages in their inbox. Also problems with the submission_host could very much be temporary and a little delay in delivering a message is better then putting it somewhere the user doesn't expect a message to be. Is there any way to change the behavior of dovecot or the sieve plugin to tempfail in case a message cannot be sent out? I know that with multiple sieve actions it gets more complicated as there could be corner cases were the first message can be sent via the submission server and another message produced by the same sieve script cannot. I'd love sieve to behave like this: a) if submission host is unreachable (hostname wrong, timeout, ...) -> tempfail b) if submission host returns 4xx error -> tempfail c) if submission host returns 5xx error -> fail(-save) and just drop the message into inbox or even have an option to reject the message completely in this case. In case of a tempfail response, the mta simply queues the message and tries again later. In case of a permanent fail the mta will create a bounce message to in form the sender. Regards and have a great weekend, Christian

2 1

Re: [Dovecot] LDA without lookup as non-root?
by E.B. 11 Nov '12

11 Nov '12

Timo, Sorry I didn't see your response until now >> 3) The interesting part -- I am invoking LDA from Maildrop. See: >> http://thread.gmane.org/gmane.mail.imap.dovecot/65473 > So >> when invoked, Maildrop has already dropped to the destination UID/GID >> and the needed paths are available in the environment. However, using >> as many permutations of calling LDA as I can think of (based on >> http://wiki2.dovecot.org/LDA ), I always get this: >> >> (command line usage error. Command output: lda: Fatal: Couldn't lookup >> our username (uid=2500) ) > > Set USER environment. Sorry, would you mind being more specific? If you see my follow-up posts on this thread, I found a source file with this error message in it (link below) and reading that code, there is no way to avoid this error for non-system users (uid's) if you don't use -d. (looking at the "destination" variable) http://hg.dovecot.org/dovecot-sieve-1.1/raw-rev/7d85833eff96

1 0

[Dovecot] doveadm quota get -A via doveadm-proxy director with LDAP
by Ramon Frontera 10 Nov '12

10 Nov '12

Hi, We are running Dovecot 2.1.10 I want configure dovecot to do execute doveadm commands on the proxy server. We have 1 proxy with director and 2 mail servers. we use LDAP when I try to do doveadm quota get -u user it works fine. The problem is when I try to do doveadm quota get -A ON PROXY I have this result: doveadm(first_user): Error: Proxy is missing destination host doveadm: Error: Failed to iterate through some users Username Quota name Type Value Limit % in 10-auth.conf I have "!include auth-ldap.conf.ext" in auth-ldap.conf.ext I hve: passdb { driver = ldap args = /etc/dovecot/dovecot-ldap.conf.ext } userdb { driver = ldap args = /etc/dovecot/dovecot-ldap.conf.ext } and In dovecot-ldap.conf.ext I have: uris = ldap://ldap:port dn = xxxxx dnpass = yyyyy auth_bind = no ldap_version = 3 base =base scope = subtree user_attrs = attributes user_filter = (mail=%u) pass_attrs = userPassword=password,uid=user=%L$,cn=proxy,=starttls=any-cert pass_filter = (mail=%u) iterate_attrs = uid=user iterate_filter = (uid=*) I don't know where is my error!!! Thanks in advance -- ------------------------------------------ Ramon Frontera Gallardo Universitat de les Illes Balears

3 2

[Dovecot] quota recalc does not work
by Adrian Stoica 09 Nov '12

09 Nov '12

Hi I upgraded dovecot from 2.0 to 2.1 , and now i face the quota recalc problem. All settings are keep in mysql Before , after a change in mailbox.quota field, i run doveadm quota recalc and all was ok , but now that command doesn't do anything. After i change a quota for a user , i have to restart dovecot to get the new quota , or do a "doveadm reload" instead of recalc. And the doveadm quota recalc -A" give me back a lot of "doveadm(username): Info: User no longer exists, skipping" Any thoughts ? # 2.1.10: /usr/dovecot21/etc/dovecot/dovecot.conf # OS: Linux 2.6.35.4 i686 auth_cache_negative_ttl = 1 secs auth_cache_size = 2 M auth_failure_delay = 4 secs auth_mechanisms = plain login digest-md5 cram-md5 auth_worker_max_count = 100 base_dir = /var/run/dovecot/ default_client_limit = 4096 default_internal_user = vmail default_login_user = nobody default_process_limit = 256 deliver_log_format = mail delivery from=%f subject=%s msgid=%m: %$ dict { acl = mysql:/usr/dovecot21/etc/dovecot/dovecot-dict-sql.conf.ext } disable_plaintext_auth = no first_valid_uid = 1 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes listen = * mail_location = maildir:~/Maildir mail_plugins = " quota acl trash notify mail_log" managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { hidden = no list = children location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u prefix = shared/%%u/ separator = / subscriptions = no type = shared } namespace inbox { inbox = yes list = yes location = mailbox Drafts { special_use = \Drafts } mailbox Junk { special_use = \Junk } mailbox Sent { special_use = \Sent } mailbox "Sent Messages" { special_use = \Sent } mailbox Trash { special_use = \Trash } prefix = separator = / subscriptions = yes type = private } passdb { args = /usr/dovecot21/etc/dovecot/dovecot-sql.conf.ext driver = sql } plugin { acl = vfile acl_shared_dict = proxy::acl autocreate = Trash autocreate2 = Drafts autocreate3 = Sent autosubscribe = Trash autosubscribe2 = Drafts autosubscribe3 = INBOX autosubscribe4 = Sent mail_log_events = delete mailbox_delete mailbox_rename mail_log_fields = box subject from msgid size quota = maildir:User quota quota_warning = storage=95%% quota-warning 95 %u quota_warning2 = storage=95%% quota-warning 90 %u quota_warning3 = storage=85%% quota-warning 85 %u sieve = ~/.dovecot.sieve sieve_dir = ~/sieve trash = /usr/dovecot21/etc/dovecot/dovecot-trash.conf.ext } postmaster_address = postmaster(a)dacris.net protocols = imap quota_full_tempfail = yes rejection_subject = Rejected: %s: %r service auth { unix_listener auth-client { mode = 0666 user = vmail } unix_listener auth-userdb { mode = 0666 user = vmail } } service dict { unix_listener dict { user = vmail } } service imap-login { process_min_avail = 4 } service imap-postlogin { executable = script-login /bin/imap.sh unix_listener imap-postlogin { mode = 01232 } } service imap { executable = imap imap-postlogin process_limit = 1024 vsz_limit = 1 G } service quota-warning { executable = script /bin/quota_warning.sh unix_listener quota-warning { user = vmail } user = vmail } ssl_cert = </etc/ssl/certs/dovecot.pem ssl_key = </etc/ssl/private/dovecot.pem userdb { args = /usr/dovecot21/etc/dovecot/dovecot-sql.conf.ext driver = sql } protocol lda { mail_plugins = " quota acl trash notify mail_log sieve autocreate" } protocol imap { imap_client_workarounds = delay-newmail mail_max_userip_connections = 20 mail_plugins = " quota acl trash notify mail_log imap_quota imap_acl" } dovecot-sql.conf.ext: password_query = SELECT password FROM mailbox WHERE \ username = '%Ln' AND domain = '%Ld' AND \ active = '1' user_query = SELECT CONCAT('/srv/vmail/',mailbox.domain,'/%Ln') AS 'home' ,'98' AS 'uid', '98' AS 'gid' , \ CONCAT('*:storage=', mailbox.quota,'M') AS quota_rule ,"Trash:storage=+50M" AS quota_rule2 FROM mailbox WHERE \ mailbox.username = '%Ln' AND mailbox.domain = '%Ld' AND \ mailbox.active = '1' iterate_query = SELECT username AS user , domain FROM mailbox WHERE active = '1'

1 0