April 2019 - dovecot - dovecot.org

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

On 4/4/19 6:47 PM, dovecot-request(a)dovecot.org wrote: > For a typical Solr index, 60 seconds is an eternity. Most people aim > for query times of 100 milliseconds or less, and they often achieve > that goal. I'm pretty sure I get these while indexing, not querying. Apr 04 16:44:50 host dovecot[114690]: indexer-worker(me(a)example.com): Error: fts_solr: Indexing failed: Request timed out (Request queued 66.015 secs ago, 1 attempts in 66.005 secs, 63.146 in http ioloop, 0.000 in other ioloops, connected 94.903 secs ago) /Peter

5 8

Fwd: Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 2

Problem with solr working, but not indexing
by TG Servers 14 Apr '19

14 Apr '19

2 1

Re: Solr connection timeout hardwired to 60s
by Peter Mogensen 14 Apr '19

14 Apr '19

sorry... I got distracted half way and forgot to put a meaningfull subject so the archive could figure out the thread. - resending. On 4/14/19 4:04 PM, dovecot-request(a)dovecot.org wrote: >> Solr ships with autoCommit set to 15 seconds and openSearcher set to >> false on the autoCommit.? The autoSoftCommit setting is not enabled by >> default, but depending on how the index was created, Solr might try to >> set autoSoftCommit to 3 seconds ... which is WAY too short. I just run with the default. 15s autoCommit and no autoSoftCommit >> This thread says that dovecot is sending explicit commits.? I see explicit /update req. with softCommit and waitSearcer=true in a tcpdump. >> One thing >> that might be happening to exceed 60 seconds is an extremely long >> commit, which is usually caused by excessive cache autowarming, but >> might be related to insufficient memory.? The max heap setting on an >> out-of-the-box Solr install (5.0 and later) is 512MB.? That's VERY >> small, and it doesn't take much index data before a much larger heap >> is required. I run with SOLR_JAVA_MEM="-Xmx8g -Xms2g" > I looked into the code (version 2.3.5.1): This is 2.2.35. I haven't checked the source difference to 2.3.x I must admit. > I immagine that one of the reasons dovecot sends softCommits is because > without autoindex active and even if mailboxes are periodically indexed > from cron, the last emails received with be indexed at the moment of the > search.? I expect that dovecot has to because of it's default behavior by only bringing the index up-to-date just before search. So it has towait for the index result to be available if there's been any new mails indexed. > 1) a configurable batch size would enable to tune the number of emails > per request and help stay under the 60 seconds hard coded http request > timeout. A configurable http timeout would be less useful, since this > will potentially run into other timeouts on solr side. Being able to configure it is great. But I don't think it solves much. I recompiled with 100 as batch size and it still ended in timeouts. Then I recompiled with 10min timeout and now I see all the batches completing and their processesing time is mostly between 1 and 2 minutes (so all would have failed). To me it looks like Solr just takes too long time to index. This is no small machine. It's a 20 core Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz and for this test it's not doing anything else, so I'm a bit surprised that even with only a few users this takes so long time. /Peter

2 2

Re: Mail account brute force / harassment
by Odhiambo Washington 14 Apr '19

14 Apr '19

On Thu, 11 Apr 2019 at 13:24, Marc Roos via dovecot <dovecot(a)dovecot.org> wrote: > > > Say for instance you have some one trying to constantly access an > account > > > Has any of you made something creative like this: > > * configure that account to allow to login with any password > * link that account to something like /dev/zero that generates infinite > amount of messages > (maybe send an archive of virusses?) > * transferring TB's of data to this harassing client. > > I think it would be interesting to be able to do such a thing. > > Instead of being evil, just use fail2ban to address this problem :-) -- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", grep ^[^#] :-)

10 24

permissions for quota-status
by Andreas Meyer 14 Apr '19

14 Apr '19

Hello! doveconf --version 2.2.36 I want to move to a new server with dovecot but get an error Error: service(quota-status): listen(*, 12340) failed: Permission denied when I define service quota-status { executable = quota-status -p postfix inet_listener { port = 12340 } client_limit = 1 } private]# ll insgesamt 0 srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 anvil srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 bounce srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 defer srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 discard srw-------. 1 postfix postfix 0 14. Apr 15:08 dovecot-lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 error srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 lmtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 local srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxymap srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 proxywrite srw-rw----. 1 postfix postfix 0 14. Apr 14:58 quota-status srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 relay srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 retry srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 rewrite srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 scache srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 smtp srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 tlsmgr srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 trace srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 verify srw-rw-rw-. 1 postfix postfix 0 14. Apr 15:05 virtual Don't know what's wrong. Can someone help? Kind regards Andreas -- PGP-Fingerprint: D392 5D21 0299 63D7 5BAE 4562 1E56 B2EA 81A2 59F1

2 1

nginx configuration to pass x-originating-ip
by André Rodier 14 Apr '19

14 Apr '19

Hello, There is a bug in SOGo, as it sends the original IP after successful login, and not before the login process. I traced the bug to the source code. https://sogo.nu/bugs/view.php?id=2979. Then, in my research, I found this old thread: https://forum.nginx.org/read.php?29,237299,237367#msg-237367 Can I use Nginx as an IMAP proxy to add the missing ID. I suspect this is something that can be achieved? I already have an Nginx for the SOGo front-end, so I would add another one for the IMAP proxy. With a little bit of luck, I should be able to pass the original IP between the two servers Does anyone has a working example? Thanks, André -- André Rodier HomeBox: https://github.com/progmaticltd/homebox

1 0

Post login scripts environment
by André Rodier 14 Apr '19

14 Apr '19

Dear Dovecot users, I am running Dovecot 2.2.27 on Debian Stretch, no issue so far. I wonder if there is a way to pass the remote IP address, in an environment variable, in the post login script. My Post login scripts are working well, except that when the server is accessed through a webmail (Roundcube or SOGo), the remote IP address is systematically 127.0.0.1. The other question I have is, is it possible to pass the user agent of the email client used to access the server? I know this can be easily forged, but I would like to log it. Thanks for your insight. -- André Rodier

2 6

Using userdb/passdb data in director_username_hash
by Mark Moseley 12 Apr '19

12 Apr '19

TL;DR: Can director_username_hash use %{userdb:...} or %{passdb:...} ? ==================================== This is on Ubuntu Precise, running dovecot 2.2.36. It's a fully production, director-ized env, so assume everything is working correctly. Happy to post doveconf if it's relevant but wanted to ask a general question first. I was curious if there's a way to get userdb/passdb data into director_username_hash. Currently, we've got default hashing (on %u). I'm returning a SQL field called 'real_username' (the owner of the mailbox, so almost never the same as %u). I'd like (for mdbox reasons) to hash on that rather than %u. My test SQL is returning (this is just a chunk -- it's duplicated for testing): UserName AS userdb_real_username, UserName AS real_username I can see in my director boxes that it's at least picking up the latter: passdb out: PASS 1 user=tesbox(a)mailbox.com proxy=y real_username=testuser Is it possible to inject 'real_username' into director_username_hash? That is, I'd rather hash on 'testuser' instead of 'testbed'. I've been trying different permutations on my director boxes with no luck. director_username_hash = %{userdb:real_username} director_username_hash = %{passdb:real_username} director_username_hash = %{userdb:userdb_real_username} director_username_hash = %{passdb:userdb_real_username} With any of those settings, every mailbox gets hashed to the same backend, so I'm guessing it's not getting anything useful (i.e. probably resolving to the same empty string and hashing on that -- or perhaps is just hashing on the literal string, e.g. "%{userdb:real_username}" ). And I'm not even sure if director_username_hash has access to any passdb/userdb data. Is there a debug setting that would show what string director is using to do the hashing? Current debug settings are: auth_debug = yes auth_debug_passwords = yes mail_debug = yes but not a peep as to the string that director is hashing on.

2 2

Migrating from external server to Dovecot (doveadm sync issues)
by Sven Strickroth 12 Apr '19

12 Apr '19

Hi, I'm experiencing serious issues when using doveadm sync not working at all or do not properly sync. I wanted to use this for migrating external mailboxes to my server (in order to overcome the time when some servers/clients still use the old server and some already the my server). I tried to find a few steps in order to replicate this: For testing purposes I simulate both the remote and local server on my Dovecot instance. I created remote(a)DOMAIN.TLD and local(a)DOMAIN.TLD both use mdbox and are empty (i.e., the mail_location is still non-existent). Then I send two mails to remote(a)DOMAIL.TLD (Subjects: Test1 and Test2). Start backup remote IMAP account (remote(a)DOMAIN.TLD) to local account (local(a)DOMAIN.TLD) * doveadm -o imapc_user="remote(a)DOMAIN.TLD" -o imapc_password="PW" -o imapc_host="localhost" -o mail_prefetch_count=20 -o imapc_features="rfc822.size fetch-headers" backup -R -u local(a)DOMAIN.TLD imapc: (without this I often got errors such as "Error: Backend didn't update mailbox Trash GUID" on sync runs, which seems to be an unrecoverable error). Now the two mailboxes seem to be the same (can confirm this when logging into the two mailboxes). Now, run first sync: * doveadm -o imapc_user="remote(a)DOMAIN.TLD" -o imapc_password="PW" -o imapc_host="localhost" -o mail_prefetch_count=20 -o imapc_features="rfc822.size fetch-headers" sync -R -u local(a)DOMAIN.TLD imapc: Seems to work and does nothing at the moment. Deliver one message to local(a)DOMAIN.TLD (Subject Test3). * doveadm -o imapc_user="remote(a)DOMAIN.TLD" -o imapc_password="PW" -o imapc_host="localhost" -o mail_prefetch_count=20 -o imapc_features="rfc822.size fetch-headers" sync -R -u local(a)DOMAIN.TLD imapc: Now Test1,2,3 are in both mailboxes (further runs do change nothing). Everything OK up the here. Now deliver one messages to remote(a)DOMAIN.TLD (Test4) and delete one old message (Test3). Deliver two messages to local(a)DOMAIN.TLD (Subjects Test5 and Test6) and delete one message (Test2). In sum both mailboxes look like this: remote(a)DOMAIN.TLD has Test1,2,4 and local(a)DOMAIN.TLD has 1,3,5,6. Now sync: * doveadm -o imapc_user="remote(a)DOMAIN.TLD" -o imapc_password="PW" -o imapc_host="localhost" -o mail_prefetch_count=20 -o imapc_features="rfc822.size fetch-headers" sync -R -u local(a)DOMAIN.TLD imapc: dsync(local(a)DOMAIN.TLD): Warning: Mailbox changes caused a desync. You may want to run dsync again: 3 UIDs changed due to UID conflicts Rerun again and see that mails start duplicating and deletings are not synced: remote(a)DOMAIN.TLD: Test1,2,4,3,5,6,5,6,6,5,3,4 (12 mails) and local(a)DOMAIN.TLD: 1,6,4,3,5,6,4,3,5,6,5 (11 mails) When I re-run again I sometimes see the above error message again, mails keep duplicating in both mailboxes and the mailboxes are not synced at all. Am I using doveadm sync wrongly? Is there a way to use "sync" directly w/o backup from the beginning (i.e., how to get rid of the error "Error: Backend didn't update mailbox Trash GUID"). Context: # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.8 imap_client_workarounds = delay-newmail I can't upgrade to latest Dovecot right now. :( -- Best regards, Sven Strickroth PGP key id F5A9D4C4 @ any key-server

1 0