December 2013 - dovecot

[Dovecot] SQL as a storage backend
by Zachary Sturgeon 06 Dec '13

06 Dec '13

Hello, I am researching using dovecot with an SQL database as the storage backend. From what I've read, dovecot only supports mbox style file storage, but I'd like to investigate the possibility of using a database instead. According to http://dovecot.org/pipermail/dovecot/2006-June/013818.html, this possibility has been explored before and has generally been accepted that Dovecot is not optimized to handle this. Any suggestions? - Zachary Sturgeon

4 3

[Dovecot] UIDL and namespaces after qmail-pop3d and BincIMAP
by Adrian C. 06 Dec '13

06 Dec '13

Hello, I migrated qmail-pop3d and BincIMAP to Dovecot in the past, and I could always tweak the settings a little to match what old services used to provide, namely namespaces (separator /, prefix INBOX/) and pop3_uidl_format (%f). This time I'm migrating to a large mail system which has its configuration set differently and there's no changing it. pop3_uidl_format = %08Xu%08Xv namespace private {prefix = INBOX. separator = . inbox = yes} Am I corrent in thinking dsync plugin is the only chance I have of getting this mail migrated right? Unfortunately Debian's Dovecot is much much older than required v2.1.4... so I could use some advice from Dovecot experts. Thanks. -- Adrian C. (anrxc) | anrxc..sysphere.org | PGP ID: D20A0618 PGP FP: 02A5 628A D8EE 2A93 996E 929F D5CB 31B7 D20A 0618

1 0

[Dovecot] Design details of high performance dovecot cluster
by JEHERUL 06 Dec '13

06 Dec '13

Dear All, Currently we are running a mail system for 10000 users each 2GB mailbox. At present we have 5 mail servers where users mailboxes are distributed almost uniformly. We have a HP SAN Storage (HP EVA 6400) where mailboxes are stored. We partition the storage into five vg and each is mounted on the 5 servers. We are using postfix and dovecot. The system was running fine but currently we notices high IO load due to IMAP connection. So we are planning to build a dovecot cluster. Please suggest some design for the dovecot cluster keeping in mind to our scenario. (If needed we can buy additional hardware like load balancer). With Thanks and Regards -- Jeherul Islam India

2 3

[Dovecot] SOLR quick and dirty how to - needs review/testing
by Christian Rößner 06 Dec '13

06 Dec '13

Hi, I just wrote a quick and dirty how to. But not sure, if that is done the way it should. So any kind of feedback is welcome. http://www.roessner-network-solutions.com/?p=1127 At least the stuff does not break things here :-) Kind regards -Christian Rößner -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

1 0

[Dovecot] Active Directory LDAP userdb and dovecot
by Simone Ferretti 06 Dec '13

06 Dec '13

Hello everybody, I have a problem with LDAP userdb and dovecot. Let me first explain my LDAP configuration: I got three Active Directory LDAP servers (a.galliera.it, b.galliera.it, c.galliera.it) responding round robin to the name galliera.it. I want to use LDAP for the userdb lookup, so I configured dovecot-ldap-userdb.conf.ext as follow: hosts = galliera.it # round robin base = dc=galliera,dc=it ldap_version = 3 auth_bind=yes auth_bind_userdn = %n(a)galliera.it dn=CN=stampa,CN=Users,DC=galliera,DC=it dnpass=stampa base = DC=galliera,DC=it scope = subtree deref = always user_attrs = sAMAccountName=home=/home/dovecot.galliera.it/%$,=uid=8,=gid=8 user_filter = (&(objectClass=person)(sAMAccountName=%n)) default_pass_scheme=CRYPT With this configuration everything go fine, I can authenticate and lookup my users in the expected way. What follows is the conversation between (one of) the LDAP server(s) and dovecot after a issue of the command $ doveadm user -u <user>@galliera.it : 62.785686 10.0.31.235 -> 10.0.5.0 TCP 74 43053 > ldap [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=536265719 TSecr=0 WS=32 62.786216 10.0.5.0 -> 10.0.31.235 TCP 78 ldap > 43053 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 TSval=0 TSecr=0 SACK_PERM=1 62.786279 10.0.31.235 -> 10.0.5.0 TCP 66 43053 > ldap [ACK] Seq=1 Ack=1 Win=14624 Len=0 TSval=536265719 TSecr=0 62.786394 10.0.31.235 -> 10.0.5.0 LDAP 122 bindRequest(1) "CN=stampa,CN=Users,DC=galliera,DC=it" simple 62.786583 10.0.31.235 -> 10.0.5.0 TCP 74 43054 > ldap [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=536265719 TSecr=0 WS=32 62.786953 10.0.5.0 -> 10.0.31.235 TCP 78 ldap > 43054 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 TSval=0 TSecr=0 SACK_PERM=1 62.787008 10.0.31.235 -> 10.0.5.0 TCP 66 43054 > ldap [ACK] Seq=1 Ack=1 Win=14624 Len=0 TSval=536265719 TSecr=0 62.787039 10.0.31.235 -> 10.0.5.0 LDAP 122 bindRequest(1) "CN=stampa,CN=Users,DC=galliera,DC=it" simple 62.788484 10.0.5.0 -> 10.0.31.235 LDAP 88 bindResponse(1) success 62.788528 10.0.31.235 -> 10.0.5.0 TCP 66 43053 > ldap [ACK] Seq=57 Ack=23 Win=14624 Len=0 TSval=536265719 TSecr=36040952 62.789334 10.0.5.0 -> 10.0.31.235 LDAP 88 bindResponse(1) success 62.789365 10.0.31.235 -> 10.0.5.0 TCP 66 43054 > ldap [ACK] Seq=57 Ack=23 Win=14624 Len=0 TSval=536265720 TSecr=36040952 62.789462 10.0.31.235 -> 10.0.5.0 LDAP 174 searchRequest(2) "DC=galliera,DC=it" wholeSubtree 62.790396 10.0.5.0 -> 10.0.31.235 LDAP 392 searchResEntry(2) "CN=Marco De benedetto,OU=S.S.C. Area sistemistica,OU=S.C. S.I.e.T. - Servizi informatici e telecomunicazioni,OU=Dipartimento di Staff,OU=Direzione generale,DC=galliera,DC=it" | searchResRef(2) | searchResDone(2) success 62.790508 10.0.31.235 -> 10.0.5.0 LDAP 191 searchRequest(3) "CN=Configuration,DC=galliera,DC=it" wholeSubtree 62.791077 10.0.5.0 -> 10.0.31.235 LDAP 168 searchResRef(3) | searchResDone(3) success 62.791172 10.0.31.235 -> 10.0.5.0 LDAP 203 searchRequest(4) "CN=Schema,CN=Configuration,DC=galliera,DC=it" wholeSubtree 62.791838 10.0.5.0 -> 10.0.31.235 LDAP 88 searchResDone(4) success 62.828752 10.0.31.235 -> 10.0.5.0 TCP 66 43054 > ldap [ACK] Seq=427 Ack=473 Win=15680 Len=0 TSval=536265730 TSecr=36040952 Strange things happen instead if I change the 'hosts' configuration variable to one (no matter what) of the domain controllers e.g.: hosts = a.galliera.it base = dc=galliera,dc=it ldap_version = 3 auth_bind=yes auth_bind_userdn = %n(a)galliera.it dn=CN=stampa,CN=Users,DC=galliera,DC=it dnpass=stampa base = DC=galliera,DC=it scope = subtree deref = always user_attrs = sAMAccountName=home=/home/dovecot.galliera.it/%$,=uid=8,=gid=8 user_filter = (&(objectClass=person)(sAMAccountName=%n)) default_pass_scheme=CRYPT Changing this variable to only one of the domain controllers make the doveadm request (issued as above) hang for some seconds and then exit with a Request time out message. This is the conversation record, you can note that there is *more the one* LDAP server involved in the conversation despite the fact that in the conf file only one is specified: 3427.019635 10.0.31.235 -> 10.0.10.0 TCP 74 33963 > ldap [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=537106777 TSecr=0 WS=32 3427.020428 10.0.10.0 -> 10.0.31.235 TCP 78 ldap > 33963 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 TSval=0 TSecr=0 SACK_PERM=1 3427.020489 10.0.31.235 -> 10.0.10.0 TCP 66 33963 > ldap [ACK] Seq=1 Ack=1 Win=14624 Len=0 TSval=537106777 TSecr=0 3427.020562 10.0.31.235 -> 10.0.10.0 LDAP 122 bindRequest(1) "CN=stampa,CN=Users,DC=galliera,DC=it" simple 3427.021894 10.0.31.235 -> 10.0.10.0 TCP 74 33964 > ldap [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=537106778 TSecr=0 WS=32 3427.022474 10.0.10.0 -> 10.0.31.235 LDAP 88 bindResponse(1) success 3427.022498 10.0.31.235 -> 10.0.10.0 TCP 66 33963 > ldap [ACK] Seq=57 Ack=23 Win=14624 Len=0 TSval=537106778 TSecr=33551087 3427.022805 10.0.10.0 -> 10.0.31.235 TCP 78 ldap > 33964 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 TSval=0 TSecr=0 SACK_PERM=1 3427.022868 10.0.31.235 -> 10.0.10.0 TCP 66 33964 > ldap [ACK] Seq=1 Ack=1 Win=14624 Len=0 TSval=537106778 TSecr=0 3427.022996 10.0.31.235 -> 10.0.10.0 LDAP 122 bindRequest(1) "CN=stampa,CN=Users,DC=galliera,DC=it" simple 3427.024537 10.0.10.0 -> 10.0.31.235 LDAP 88 bindResponse(1) success 3427.024574 10.0.31.235 -> 10.0.10.0 TCP 66 33964 > ldap [ACK] Seq=57 Ack=23 Win=14624 Len=0 TSval=537106778 TSecr=33551087 3427.024884 10.0.31.235 -> 10.0.10.0 LDAP 176 searchRequest(2) "DC=galliera,DC=it" wholeSubtree 3427.028078 10.0.10.0 -> 10.0.31.235 LDAP 391 searchResEntry(2) "CN=Simone Ferretti,OU=S.S.C. Area sistemistica,OU=S.C. S.I.e.T. - Servizi informatici e telecomunicazioni,OU=Dipartimento di Staff,OU=Direzione generale,DC=galliera,DC=it" | searchResRef(2) | searchResDone(2) success 3427.028426 10.0.31.235 -> 10.0.5.0 TCP 74 43077 > ldap [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=537106779 TSecr=0 WS=32 3427.028882 10.0.5.0 -> 10.0.31.235 TCP 78 ldap > 43077 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=1 TSval=0 TSecr=0 SACK_PERM=1 3427.028932 10.0.31.235 -> 10.0.5.0 TCP 66 43077 > ldap [ACK] Seq=1 Ack=1 Win=14624 Len=0 TSval=537106780 TSecr=0 3427.029091 10.0.31.235 -> 10.0.5.0 LDAP 80 bindRequest(4) "<ROOT>" simple 3427.029816 10.0.5.0 -> 10.0.31.235 LDAP 88 bindResponse(4) success 3427.029832 10.0.31.235 -> 10.0.5.0 TCP 66 43077 > ldap [ACK] Seq=15 Ack=23 Win=14624 Len=0 TSval=537106780 TSecr=36074586 3427.030001 10.0.31.235 -> 10.0.5.0 LDAP 193 searchRequest(3) "CN=Configuration,DC=galliera,DC=it" wholeSubtree 3427.030980 10.0.5.0 -> 10.0.31.235 LDAP 168 searchResRef(3) | searchResDone(3) success 3427.064639 10.0.31.235 -> 10.0.10.0 TCP 66 33964 > ldap [ACK] Seq=167 Ack=348 Win=15680 Len=0 TSval=537106789 TSecr=33551087 3427.068791 10.0.31.235 -> 10.0.5.0 TCP 66 43077 > ldap [ACK] Seq=142 Ack=125 Win=14624 Len=0 TSval=537106790 TSecr=36074586 Is there anyone that have some ideas of what is going on? My dovecot version is 2.2.9 (5c170e0786f3) running on a debian wheezy 3.2.0-4-amd64. My Active Directory server is a windows server 2003 R2. Follows my doveconf -n: # 2.2.9 (5c170e0786f3): /etc/dovecot/dovecot.conf # OS: Linux 3.2.0-4-amd64 x86_64 Debian 7.2 ext4 auth_cache_size = 8 k auth_debug = yes auth_master_user_separator = * auth_username_chars = abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@+ auth_verbose = yes disable_plaintext_auth = no first_valid_uid = 5 hostname = dovecot.galliera.it last_valid_uid = 100000 lda_mailbox_autocreate = yes lda_mailbox_autosubscribe = yes mail_access_groups = mail mail_debug = yes mail_gid = mail mail_location = maildir:/home/%d/%n/Maildir mail_privileged_group = mail mail_uid = mail maildir_very_dirty_syncs = yes managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave namespace { list = children location = maildir:/var/mail/shared/istituzionali:INDEX=/var/mail/shared/istituzionali/INDEX/%n prefix = Avvisi al personale. subscriptions = yes type = public } namespace inbox { inbox = yes location = mailbox Drafts { auto = subscribe special_use = \Drafts } mailbox Junk { auto = subscribe special_use = \Junk } mailbox Sent { auto = subscribe special_use = \Sent } mailbox "Sent Messages" { auto = subscribe special_use = \Sent } mailbox Trash { auto = subscribe special_use = \Trash } prefix = subscriptions = yes type = private } passdb { args = /etc/dovecot/dovecot-ldap.conf.ext driver = ldap } plugin { acl = vfile fts = squat fts_squat = partial=4 full=10 mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename mail_log_fields = from subject uid box msgid size sieve = ~/.dovecot.sieve sieve_dir = ~/sieve } postmaster_address = postmaster(a)galliera.it protocols = " imap lmtp sieve pop3" service auth { unix_listener /var/spool/postfix/private/auth { group = postfix mode = 0666 user = postfix } unix_listener auth-master { mode = 0600 user = mail } unix_listener auth-userdb { mode = 0777 user = mail } user = root } service lmtp { unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } } ssl = no ssl_cert = </etc/dovecot/dovecot.pem ssl_key = </etc/dovecot/private/dovecot.pem syslog_facility = local5 userdb { args = /etc/dovecot/dovecot-ldap-userdb.conf.ext driver = ldap } protocol imap { mail_max_userip_connections = 30 mail_plugins = fts fts_squat mail_log zlib acl imap_acl notify } protocol pop3 { pop3_uidl_format = %v.%u } Thanks in advance and ciao! Simone

1 0

[Dovecot] Expire plugin and databases
by Mauricio Tavares 06 Dec '13

06 Dec '13

Dumb question: do I need to have a database to run the expire plugin?

2 1

[Dovecot] zlib config questions
by Terry Barnum 05 Dec '13

05 Dec '13

After nearly running out of space I swapped in larger disks and then saw the recent threads about zlib compression. Unfortunately I'm still confused after reading <http://wiki2.dovecot.org/Plugins/Zlib>. In order to compress new email being stored do I only need to change 10-mail.conf to this: mail_plugins = $mail_plugins zlib and 90-plugin.conf to this: plugin { zlib_save_level = 6 zlib_save = gz ... Or do I need to instead (or also?) add it to the list of plugins in 20-imap.conf? A simple 'sudo doveadm reload' to enable? Once enabled, is there an indication in the logs that compression is taking place? Do new mail files have a suffix like .Z to indicate they've been compressed? Any gotchas to be aware of? I'm running macports dovecot 2.2.5. Thanks for any help. -Terry Terry Barnum digital OutPost http://www.dop.com

1 0

[Dovecot] Syslog debug messages
by Bruno Galindro da Costa 05 Dec '13

05 Dec '13

Is there a way to configure dovecot to register in syslog the debug messages enabled by mail_debug=yes as a debug not as information? I need this because I want to send this kind of information only to a remote server. I've configured rsyslog to do this, but dovecot is registering those debug messages as information not as debug (as I expected). This is my rsyslog config: mail.info;\ mail.warning;\ mail.err;\ mail.crit -/var/log/mail.log mail.* :omrelp:192.168.2.162:2514 root@imap:~# dovecot -n # 1.2.9: /etc/dovecot/dovecot.conf Warning: fd limit 1024 is lower than what Dovecot can use under full load (more than 4748). Either grow the limit or change login_max_processes_count and max_mail_processes settings # OS: Linux 2.6.32-44-server x86_64 Ubuntu 10.04.4 LTS xfs log_timestamp: %Y-%m-%d %H:%M:%S protocols: imap imaps managesieve pop3s listen(default): *:143,[::]:143 listen(imap): *:143,[::]:143 listen(pop3): * listen(managesieve): * ssl_listen(default): *:993,[::]:993 ssl_listen(imap): *:993,[::]:993 ssl_listen(pop3): *:2221,[::]:2221 ssl_listen(managesieve): ssl_ca_file(default): /etc/dovecot/ca.crt ssl_ca_file(imap): /etc/dovecot/ca.crt ssl_ca_file(pop3): /etc/dovecot/ca.crt ssl_ca_file(managesieve): ssl_cert_file(default): /etc/dovecot/ufsc.br.crt ssl_cert_file(imap): /etc/dovecot/ufsc.br.crt ssl_cert_file(pop3): /etc/dovecot/ufsc.br.crt ssl_cert_file(managesieve): /etc/ssl/certs/dovecot.pem ssl_key_file(default): /etc/dovecot/ufsc.br.key ssl_key_file(imap): /etc/dovecot/ufsc.br.key ssl_key_file(pop3): /etc/dovecot/ufsc.br.key ssl_key_file(managesieve): /etc/ssl/private/dovecot.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable(default): /usr/lib/dovecot/imap-login login_executable(imap): /usr/lib/dovecot/imap-login login_executable(pop3): /usr/lib/dovecot/pop3-login login_executable(managesieve): /usr/lib/dovecot/managesieve-login login_process_per_connection: no login_process_size: 256 login_max_processes_count: 900 login_max_connections: 1024 max_mail_processes: 2048 mail_max_userip_connections(default): 2048 mail_max_userip_connections(imap): 2048 mail_max_userip_connections(pop3): 10 mail_max_userip_connections(managesieve): 10 mail_privileged_group: mail mail_uid: vmail mail_gid: mail mail_location: maildir:/var/mail/mailboxes/%d/%n:INDEX=/var/mail/indexes/%d/%n mail_debug: yes fsync_disable(default): yes fsync_disable(imap): yes fsync_disable(pop3): yes fsync_disable(managesieve): no mbox_write_locks: fcntl dotlock mail_executable(default): /usr/lib/dovecot/imap mail_executable(imap): /usr/lib/dovecot/imap mail_executable(pop3): /usr/lib/dovecot/pop3 mail_executable(managesieve): /usr/lib/dovecot/managesieve mail_plugins(default): quota imap_quota autocreate trash zlib mail_plugins(imap): quota imap_quota autocreate trash zlib mail_plugins(pop3): quota mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3 mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve managesieve_implementation_string(default): dovecot managesieve_implementation_string(imap): dovecot managesieve_implementation_string(pop3): dovecot managesieve_implementation_string(managesieve): Cyrus timsieved v2.2.13 lda: fsync_disable: no postmaster_address: xxx(a)xxx.xxx.xx mail_plugins: quota sieve auth_socket_path: /var/run/dovecot/auth-master auth default: cache_size: 1024 cache_ttl: 600 cache_negative_ttl: 60 username_chars: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@~ username_format: %Lu passdb: driver: checkpassword args: /etc/dovecot/passdb.v3.1.sh userdb: driver: prefetch userdb: driver: ldap args: /etc/dovecot/dovecot-ldap-userdb.conf socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: mail plugin: quota_exceeded_message: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. quota: maildir:DefaultQuota quota_warning: storage=95%% /usr/local/bin/quota-warning.sh 95 %u quota_warning2: storage=90%% /usr/local/bin/quota-warning.sh 90 %u trash: /etc/dovecot/dovecot-trash.conf sieve: /var/mail/sieve_scripts/%d/%n/.dovecot.sieve sieve_dir: /var/mail/sieve_scripts/%d/%n/sieve mail_log_events: delete expunge copy mailbox_delete mail_log_fields: uid box msgid from subject size vsize autocreate: Spam autosubscribe: Spam -- Att. Bruno Galindro da Costa

2 1

[Dovecot] FTS solr issue with second private namespace
by Andreas Oster 05 Dec '13

05 Dec '13

Hi all, I have just stumbled over an issue with the FTS plugin of dovecot. For our primary namespace, INBOX and its subfolders, searching works without any problems but in a second namespace, which I have created for old/archived mails, searching of the mail text/body does not work. I can see the search request in the tomcat log but no result is created and the mail client (Thunderbird) keeps displaying the serach message without returning any results. Tomcat6 catalina.out: Dez 02, 2013 11:58:00 AM org.apache.solr.core.SolrCore execute INFO: [] webapp=/solr path=/select params={fl=uid,score&sort=uid+asc&q=body:"schiller"&fq=%2Bbox:1b4de60d0a6287522d0c00004424cda4+%2Buser:"testuser"&rows=2} hits=0 status=0 QTime=1 I am currently using Dovecot 2.2.9 and do not have any special FTS config: mail_plugins = quota mailbox_alias fts fts_solr plugin { fts = solr fts_solr = break-imap-search url=http://localhost:8080/solr/ fts_autoindex = yes } Thank you for your kind help best regards Andreas

1 2

[Dovecot] Full text search improvements
by Timo Sirainen 05 Dec '13

05 Dec '13

FTS indexing is something I hear quite often nowadays. I’ve added some hacks to make it work better for some installations, but it’s about time to think about the whole design and how it could be improved for everyone in future. Here are some of my initial thoughts. Currently Dovecot supports 3 full text search engines: Solr, CLucene and Dovecot Squat. CLucene plugin has various features built in, which should have been built in a generic way to work with all the engines (although Solr has most of those already built-in). Squat was abandoned a few years ago in favor of Solr/CLucene, but perhaps it could be brought back to life, since it looks like its index sizes could be smaller than Lucene's. Here's a list of things that should be added to generic Dovecot FTS code to improve all the backends: 1. Support for multiple languages. Use textcat while indexing to guess the language of the indexed data. (Perhaps run it separately for each paragraph to handle multi-language mails? Or at least many emails begin/end with different language than the text in the middle, e.g. "Foo Bar wrote:" is often in various languages.) Index the data using the detected language's stemming and other features. Keep track of which languages have been used in the index, and when searching stem the search words to all the used languages. Since each added language requires additional searches and there's the possibility of wrong detection, the list of allowed languages could be configurable. See also http://ntextcat.codeplex.com/ or at least change textcat to use UTF8. 2. Word stemming. This can be done for many languages with Snowball library. Solr has also implemented several other languages, perhaps its code can be somehow automatically translated to C(++) for use with Dovecot? 3. Don't index language-specific stopwords. We can get the word lists from e.g. Solr. 4. Try to detect compound words and index each part separately for languages that use them. http://wiki.apache.org/solr/LanguageAnalysis#Decompounding suggests two possible ways to do it. 5. Normalize words (e.g. drop diacritics). libicu can be used for this. 6. Drop (Unicode) characters that don't belong to the language? Or especially don't index most of the weird Unicode characters. This would avoid filling the index with unnecessary garbage. 7. Don't index non-text data? For example if there is large block of base64 data or something else that definitely doesn't look like text, it's pretty useless to index it. Then again, we do want to index all kinds of IDs that someone might want to search. This could be a bit difficult to implement well. 8. Index attachments separately, so it would be possible to search only attachments. (Should "SEARCH BODY word1 BODY word2" return matches if word1 and word2 are in different attachments?) 9. Attachments can be translated to indexable UTF-8 text already with fts_decoder setting by doing it via a conversion script. This could also support Apache Tika server directly. 10. It should be configurable which fields are indexed. Body and header would always be separately indexed. Optionally there could be also at least: attachments, From, To, Cc, Bcc and Subject. The From/To/Cc/Bcc could also be indexed together in one "addresses" field. The more fields there are, the larger the index, but better/faster search results. 11. Each indexed mail should have metadata: Mailbox GUID, mail UID and the language the mail was indexed with. For attachments there should also be the MIME part number. When matching results, drop results if returned language doesn't match the query language. Squat ----- Currently Squat index consists of a trie containing all the words and pointer to a file listing all the message UIDs that contain them. Each node in the trie has a pointer to the UIDs, so e.g. with "abc" the "a" node will contain UIDs of all mails that contain the "a" letter (e.g. 1,3-5,10). "ab" node will contain mails that have the "ab" substring. Since the "ab" is a subset of "a", the "ab" won't contain UIDs directly but instead it contains indexes to the "a" list to get a better compression (e.g. UID 3-5,10 -> 2-4 indexes in the "1,3-5,10" list). The "abc" node then similarly refers to the "ab" node's indexes. It's configurable how long words Squat will index. Also substring matching is configurable. By default both are 4 letters, so words longer than 4 letters will be split to 4 letter pieces which are indexed (e.g. "dovecot" -> "dove", "ovec", "veco", "ecot"). When searching these pieces are looked up and the results are merged. It's pretty pointless to do a search for 1-2 letter substrings. Most likely the user wants to find 1-2 letter word instead. Perhaps this is true also for 3 letters? The Squat index could be changed to only add results for the first 1-2 (or 1-3?) letters only for full words, not to word prefixes. This of course would mean that the "ab" referring to "a" UID list would no longer work for the first nodes. Substring searching likely wouldn't work very nicely for stemmed words. So Squat should probably index the full stemmed word and then also index the unstemmed word in the small 4 letter pieces. It should be possible to also disable substring searching entirely. Squat already attempts to reduce disk space by encoding the common characters with less bits than other characters. This is hardcoded for English language though. Each index compression could analyze the most common letters and dynamically use those. Perhaps different languages could even use separate Squat index files to get the most advantage of this? Although if there are a lot of languages it's a bit annoying to have many different index files. Squat currently indexes each Unicode character as one letter, so there can be quite a huge number of different 4 letter words. This substring indexing should probably be disabled for words that contain letters not used by the current language. The UID lists are not compressed very efficiently. See http://www2009.eprints.org/41/1/p401.pdf for alternatives. There is currently one Squat index per mailbox. There should be one per user. This requires adding a separate ID -> { mailbox GUID, UID, language, mime part } mapping file. Squat indexes should be updated in two parts. For old mails there are the large trie+uidlist base files in optimized format. For new mails there are small trie+uidlist files in a format where they are just appended to. When the trie or uidlist becomes large enough, it's first internally sorted/optimized, and then merged with the base files by recreating new optimized base files. This fixes the scalability problem with current large Squat indexes where with large unsorted indexes the sorting could have taken forever. Since substring searches in Squat produce only "maybe matches" UID lists, the mails still need to be opened and searched through. This step could probably also do the same language detection + stemming as is done during indexing to improve the search results. (And this feature could be enabled even if no FTS index is enabled.)

6 9