Implementing secondary quota w/ "Archive" namespace
Hello folks—
My first post, so please be gentle… :-)
I have a client email server using SSDs for primary user mailboxes, but since the number of users keeps growing and they all seem to be very reluctant to delete anything, I’ve implemented an “Archive” namespace that stores its mailboxes on a larger HD RAID. The idea is that, as the users approach their quota, they move messages to the Archive mailboxes to alleviate space in their primary Inbox namespace. This secondary storage part is working well, but I’m having trouble w/ getting the quotas to work right. Here are the basics of the setup:
Mac Pro Server 2012 Mac OS X Server 10.6.8 Dovecot 1.1.20apple0.5
Here is how I’ve configured my namespaces (during testing):
namespace private { separator = / prefix = inbox = yes }
namespace private { separator = / prefix = testArchive/ location = maildir:/Shared Items/MailArchive/%u subscriptions = yes }
My quota research has led me to try this:
quota = maildir:User quota:ns=
quota2 = maildir:ns=testArchive/ quota2_rule = *:storage=20G
The first line is already in the default config, with the exception of the added “:ns=“ at the end. The 2nd line in the examples I saw had a middle component w/ the quota name, but when I tried that, like so:
quota2 = maildir:Archive quota:ns=testArchive/
my server fails and shows this in the logs:
Fatal: IMAP(*): Quota root test backend maildir: Unknown parameter: ns=testArchive/
Any idea why it doesn’t like that? Also, do I need to add a quota_rule for the primary quota? It does not have one normally in the Mac OS X Server config…
Thus far in my testing, I’ve been able to get the 2 quotas to show up in Roundcube and Mac Mail.app. It’s a little messy…the first shows up as “User quota”, the 2nd as “ns=testArchive/“, presumably because I cannot leave the description field in there.
Unfortunately, both quotas show the same amount of space in use. If I drop the primary quota to a mere 4MB for testing, and if I have 5.2MB of messages in a testArchive folder, the space used for “User quota” shows as 5.2MB (>100%), as does the “ns=testArchive/“ quota (which is 20GB). In actuality, the Inbox namespace is really only using a few KB— the 5.2MB is in the testArchive namespace. This means that I cannot move messages between either set of namespaces, and new messages are not delivered. So, the quota trouble here is negating the whole point of having the Archive namespace...
Is there a way to get Dovecot to “see” the 2 quotas as unique/discrete? It seems like I’m close to accomplishing what I want, but just can’t quite get it to cooperate. And that “Unknown parameter” error is bewildering. Any ideas?
Thx, Fred
P.S. I can add my Dovecot config to the thread upon request…didn’t want to make this initial message even longer.
Hey Everybody—
Posted this to the list a couple of months ago, but didn’t get any responses. Is there a better place to ask this question about quota & namespace configuration? Seems like a lot of the discussion here is a little deeper/lower-level than my configuration question, like debugging and development…
Thx, Fred
On Sun, Nov 20, 2016 at 3:28 PM, Fred Turner <fred@c5consulting.net> wrote:
I beat my head against basically the same wall a few years back (and similarly felt like I was almost in reach but could never quite get it working), so I'm highly interested in the same topic. But I'd love to hear from someone smarter than me if this is even possible. I don't mind beating my head against a wall if it's not for no reason.
Can anyone verify if this is even possible? Timo?
On Nov 21, 2016, at 7:34 PM, Fred Turner <fred@c5consulting.net> wrote:
Lol, may actually be a 2010...which is essentially no different than a 2012. Mac OS X 10.6.8 would work just fine on that hardware. And probably 10.5.8 as well!
It is generally not possible o install an older OS on a Mac than it shipped with.
That machine, even if it’s a 2010, will run 10.7, 10.8, 10.8, 10.10, 10.11, or even 10.12.
You might have better luck with a support OS.
Sorry, but that is not correct. 10.7 Lion was not released until July 2011. Mac Pro 2010 would obviously not have included it, but rather 10.6 Snow Leopard. And again, there is negligible difference between 2009-2010-2012 Mac Pros, such that they all will likely run anything from 10.5 onward through 10.11 equally well (actually 10.12 too even on the 2009s with a firmware update), whether Apple explicitly specified support or not. And I am not running an older system than this machine shipped with, as it shipped with 10.6.
In any case, my difficulty here is not related to OS hardware support or lack of, but rather this version of Dovecot (v1.1) not being able to do separate quotas for separate namespaces. Dovecot v2 might be able to, but there had apparently been difficulties with that as well, from I've read.
FT
Yeah, I gradually figured out it wouldn't work yesterday when delving back into this and testing. No separate quotas per namespaces until 2.1 or something, I think?
So, got any suggestions on getting it to work with v2.x? I found an old thread from 2013 by Andreas (I think?) and he didn't seem to quite be able to get it to work. Actually, though, I'd be happy to even be able to apply a quota to the primary Inbox namespace and none to the secondary "Archive" namespace, but my testing on a 10.10 Server wasn't having much success either.
Thanks for the responses and input! Fred
On Mon, Nov 21, 2016 at 6:20 PM, Fred Turner <fred@c5consulting.net> wrote:
So I've been playing with this and I mostly have things working. It's 2.2.26.0, btw. In all the below, both namespaces are working and I can copy/move messages back and forth between them.
One thing that I've not figured out yet (though I'm sure I'm just missing something scouring the docs):
If I move messages between namespaces, it appears to ignore the quotas I've set on them. A *copy* will trigger the quota error. But a *move* just happily piles on to the overquota namespace. Is that normal?
E.g., here's the maildirsize from the 'archive' namespace (with quotas set absurdly low for testing) and I just moved some messages into it from INBOX:
20000S,10C 32252 31 2809 1
and it'll just keep tacking on. As you can see it's over on bytes and # of messages. But it will successfully block a copy. This behavior of ignoring the quota for moves goes in both directions, from INBOX to 'archive' and vice versa.
And note that the values above are what I set, so it *is* seeing the quota just fine (and like I said, when I copy a message, it gets appropriately blocked due to quota).
Is this the normal behavior for message moves?
Oh, and it's definitely a move:
A0004 UID MOVE 180 Archive.archive1..
- OK [COPYUID 1268932143 180 53] Moved UIDs...* 69 EXPUNGE..A0004 OK Move completed (0.042 + 0.000 + 0.041 secs)...
BTW, since I spent a good deal of time before I figured this out, if you're using SQL prefetch, the syntax for overrding the location in passdb password_query becomes (with the example ns of 'archive'):
userdb_namespace/archive/location
instead of
namespace/archive/location
I couldn't for the life of me figure out why dovecot was ignoring 'namespace/archive/location'. Writing this email helped me figure it out, as usual :)
=====================================
doveconf -n:
2.2.26 (54d6540): /etc/dovecot/dovecot.conf
Pigeonhole version 0.4.16 (fed8554)
OS: Linux 3.14.77 x86_64 Ubuntu 12.04.5 LTS
auth_cache_negative_ttl = 1 mins auth_cache_size = 10 M auth_cache_ttl = 10 mins auth_debug = yes auth_debug_passwords = yes auth_mechanisms = plain login base_dir = /var/run/dovecot/ debug_log_path = /var/log/dovecot/debug.log default_client_limit = 3005 default_internal_user = doveauth default_process_limit = 1500 deliver_log_format = M=%m, F=%f, S="%s" B="%p/%w" => %$ disable_plaintext_auth = no first_valid_uid = 199 imap_capability = +UNSELECT last_valid_uid = 201 listen = * log_path = /var/log/dovecot/mail.log mail_debug = yes mail_location = maildir:~/Maildir mail_nfs_storage = yes mail_privileged_group = mail mail_uid = 200 managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { hidden = no inbox = yes list = yes location = prefix = INBOX. separator = . subscriptions = yes type = private } namespace archive { inbox = no list = children location = maildir:~/Archive prefix = Archive. separator = . subscriptions = yes type = private } passdb { args = /etc/dovecot/include/sql.conf driver = sql } plugin { quota = maildir:User quota quota2 = maildir:User2 quota:ns=Archive. quota2_rule = *:bytes=20000 quota2_rule2 = *:messages=10 quota_rule = *:bytes=400000 quota_rule2 = INBOX.Trash:bytes=+10%% quota_rule3 = *:messages=2000 } protocols = imap sieve service auth-worker { unix_listener auth-worker { mode = 0666 } user = doveauth } service auth { client_limit = 8000 unix_listener login/auth { mode = 0666 } user = doveauth } service imap-login { inet_listener imap { port = 143 } process_min_avail = 20 service_count = 1000 vsz_limit = 256 M } service imap { client_limit = 10 idle_kill = 5 mins process_limit = 2000 process_min_avail = 10 service_count = 1000 vsz_limit = 512 M } service managesieve-login { inet_listener sieve { port = 4190 } } ssl = no userdb { driver = prefetch } userdb { args = /etc/dovecot/include/sql.conf driver = sql } verbose_proctitle = yes protocol imap { mail_max_userip_connections = 10000 mail_plugins = zlib quota imap_quota stats imap_stats } protocol lda { debug_log_path = /var/log/dovecot/lda/lda-quotawarning.log log_path = /var/log/dovecot/lda/lda-quotawarning.log }
On Wed, Nov 23, 2016 at 6:05 PM, Timo Sirainen <tss@iki.fi> wrote:
I appreciate the patch! Esp on a Weds night. I applied and rerolled dovecot, but I can still move messages into the over-quota namespace.
I threw some i_debug's into quota_roots_equal() (and one right at the top), but I don't ever see them in the debug logs. But both "ctx->moving" and "src_box == NULL" are true, so it never calls quota_roots_equal anyway in that patched 'if' clause in quota_check. I threw the following into quota_check and it printed to the debug log for both if's:
if (ctx->moving ) i_debug("quota: quota_check: YES to ctx->moving"); if (src_box == NULL) i_debug("quota: quota_check: YES to src_box == NULL" );
Out of curiosity, in the Quota wiki page, it mentions that 'in theory there could be e.g. "user quota" and "domain quota" roots'. That's also super interesting to me. Does anyone have any experience with that? I.e. any gotchas?
On 24 Nov 2016, at 9.33, Mark Moseley <moseleymark@gmail.com> wrote:
How about this updated patch?
There's no automatic quota recalculation for domain quotas, because it would have to somehow sum up all the users' quotas. Also I think that it still does do the automatic quota recalculation if it gets into a situation where it realizes that quotas are wrong, but it'll then just use the single user's quota as the entire domain quota. So maybe it would work if you externally sum up all the users' quotas and update it to the domain quota in cronjob, e.g. once per hour. I guess it would be also nice if the internal quota recalculation could be disabled and maybe execute an external script to do it (similar to quota-warnings).
On Thu, Nov 24, 2016 at 9:10 PM, Mark Moseley <moseleymark@gmail.com> wrote:
Anything else I can try? I'm not sure how the logic in the quota system works, so I'm not sure what to suggest. What's the gist of the patch (i.e. what's it trying to do that it wasn't before)?
If I can get a handle on that, I can start littering things with debug statements to try to track stuff down.
On 1 Dec 2016, at 2.22, Mark Moseley <moseleymark@gmail.com> wrote:
..
I just messed up the if-check. This one is now committed and should work: https://github.com/dovecot/core/commit/2ec4ab6f5a1172e86afc72c0f29f470d6fd2b... <https://github.com/dovecot/core/commit/2ec4ab6f5a1172e86afc72c0f29f470d6fd2bd9a.diff>
On Thu, Dec 1, 2016 at 4:37 AM, Timo Sirainen <tss@iki.fi> wrote:
that looks good. When I apply it, I get:
quota-storage.c: In function ‘quota_save_finish’: quota-storage.c:337:15: error: ‘struct mail_save_context’ has no member named ‘copy_src_mail’ quota-storage.c:337:51: error: ‘struct mail_save_context’ has no member named ‘copy_src_mail’ make[4]: *** [quota-storage.lo] Error 1
But if I then also apply the previous patch you gave, though it fails in a number of sections:
patch -p1 < ~moseley/diff2
(Stripping trailing CRs from patch.) patching file src/lib-storage/mail-storage-private.h (Stripping trailing CRs from patch.) patching file src/lib-storage/mail-storage.c Hunk #1 succeeded at 2238 (offset -20 lines). Hunk #2 succeeded at 2255 (offset -20 lines). (Stripping trailing CRs from patch.) patching file src/plugins/quota/quota-storage.c Hunk #1 FAILED at 185. Hunk #2 FAILED at 242. Hunk #3 FAILED at 297. 3 out of 3 hunks FAILED -- saving rejects to file src/plugins/quota/quota-storage.c.rej
BUT, it then compiles.
I haven't tested it extensively, but with this latest patch, when I try to move mail to the over-quota Archive mailbox, it correctly fails! Awesome!
participants (4)
-
＠lbutlr
-
Fred Turner
-
Mark Moseley
-
Timo Sirainen