Re: Questions about hardlinks, alternate storage and compression
Hi Javier, thanks for your reply.
I already checked SIS and, while interesting, is not what I want, because:
- it can be difficult to restore a single message/attachment from a backup
- only the attachments, and not the entire messages, are deduped.
Message-based hardlinks really exists for a reason. The good news is that I found _why_ they are not working: it depends from how dovecot and its sieve plugin (pigenhole) interact. Basically, if I define anything for the before_sieve and after_sieve variables, dovecot stops creating hardlinks for multiple copies of the same message.
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
Thank you all.
On 13/07/15 11:10, Javier Miguel RodrÃguez wrote:
Search about "single instance storage dovecot". This is what you need.
Regards
Javier
On 27/06/2015 18:18, Gionatan Danti wrote:
Hi all, I have some questions about hardlinks, alternate storage and compression. I already scanned the list for related information and I have an idea of how things works, but I would like to have a definite answer.
System spec:
- CentOS 6.6 x64
- dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version
- sdbox mail store
About hardlinks: when sending the same message to two different recipients, I see that the two u.x files are created as two different files. Diffing them, I see that the only difference is a single char (see [1] for an example). My questions are: a) it is possible to tell dovecot to create a single file + a single hardlink (linkref=2)? As other IMAP servers support that features (eg: Cyrus, CommunigatePro, etc) I am wondering if I missed something in configuring dovecot... b) If it is not possible, can I run a script that compare the various files and substitute equal ones (minus the changing line) with hardlinks, or it will confuse dovecot? As a side note, why the changing line ever exists?
About alternate storage and compression: actually, I use a single mail_location without compression. I would like to have an alternate storage and to enable compression on it only, leaving the main location without compression. I if understand it correctly, it _should_ be done using a command similar to "doveadm -Dv -o "plugin/zlib_save=gz" altmove -uu testuser sentbefore 8d". I'm right thinking that it should work? I will really end with a primary uncompressed mail store and an alternate, zlib-compressed one?
Thank you all and sorry if I did some naive questions.
[1] 63c63 < G2fd0811c64be8e553d970000eaa8309f
G2ed0811c64be8e553d970000eaa8309f
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVaSpX3z1H7kL/d9rAQKuPwf/e4GddZvm/qj9sfAnVgV3H5iC62fnS6Ny /TPaXcuLcN5Tx9slhLTwIx8/GRROUVwLVqKYjaXwQciV2yytBu5vkC0+lowIZGq9 kJAAKPp4h3Ia6SDGhI8E5Im9VGGSpbXyLKR+V3rf1G/sOyvJTITliVe4ckf76xrI c1LGYumW0BGZeNZAAA0lYHZGrgy5meCrL20CMupmahoHsOFw5cA3HhJ/dEBRPlOJ y886BScRh7dWJXyS+PUzPFlbFOeULKvh6fVwCK7b4+aFkfjLedDLew5TThWiblK5 c5+rx0pAh8xVdXGZyQXzPjUl22KbQmGfzv78XWlN2WksCnMVaFXe2g== =3iPP -----END PGP SIGNATURE-----
On 14/07/15 08:17, Steffen Kaiser wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
I am missing something?
[1] Return-Path: <g.danti@assyoma.it> Received: from mail.gruppocrimi.it by mail.gruppocrimi.it (Dovecot) with LMTP id VFA8Fj3OmlUStwAA6qgwnw ; Mon, 06 Jul 2015 20:51:41 +0200 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.gruppocrimi.it (Postfix) with ESMTP id 22AB4A1A85; Mon, 6 Jul 2015 20:51:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at gruppocrimi.it Received: from mail.gruppocrimi.it ([127.0.0.1]) by localhost (mail.gruppocrimi.it [127.0.0.1]) (amavisd-new, port 10024) with LMTP id NC3YcizeDFPO; Mon, 6 Jul 2015 20:51:40 +0200 (CEST) Received: from mr003msb.fastweb.it (mr003msb.fastweb.it [85.18.95.87]) by mail.gruppocrimi.it (Postfix) with ESMTP id 4380DA1A7C; Mon, 6 Jul 2015 20:51:40 +0200 (CEST) Received: from ceres.assyoma.it (93.63.55.57) by mr003msb.fastweb.it (8.5.140.03) id 55501C9F0432631D; Mon, 6 Jul 2015 20:51:40 +0200 Received: by ceres.assyoma.it (Postfix, from userid 48) id B7B912643B4; Mon, 6 Jul 2015 20:51:39 +0200 (CEST) To: gionatan.danti@gruppocrimi.it, g.danti@gruppocrimi.it Subject: test invio X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 06 Jul 2015 20:51:39 +0200 From: Gionatan Danti <g.danti@assyoma.it> Organization: Assyoma s.r.l. Message-ID: <ef3149d82ef4350c1c9b6da9c1f03fae@assyoma.it> X-Sender: g.danti@assyoma.it User-Agent: Roundcube Webmail/1.0.5
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tue, 14 Jul 2015, Gionatan Danti wrote:
On 14/07/15 08:17, Steffen Kaiser wrote:
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
You asked about "newer dovecot versions", v2.2 does so.
Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQEVAwUBVaTjzXz1H7kL/d9rAQK1oAf/fdUuBY8kseVEFa5kyXG01cyUjc3RfBNl o0EYm+e2hvoz5B4N96pkbmilYjaCtgUw/qlMnGkzFbmJDwrqOiAhxOG71Aewjvbx q42cXHtw7CsOCr6y+eshNUfU3T20f7wgvyJDqLAOwg/pSP3CjU9m93D2zCqUgDXO MHuDV1zEEljlrxXmtdG8GI5YlwkBqvWXQuPbXr7PhoQ4HTKhvKHWurGvVkfBlg6k cpuy40mSWY3ZXwNDcnHP0o82EezGAdgzDE/EoV4fV0JDvANbTjpwwqE4gMW+wOM+ lUJnMyawkVuvfbB85K/tkK+a0lIVnZOwdUy0RaUcJFeZHXdRsixvIg== =HzBs -----END PGP SIGNATURE-----
On 14/07/15 12:26, Steffen Kaiser wrote:
You asked about "newer dovecot versions", v2.2 does so.
Fair enough :)
So, with v2.2+ the hardlink approach is irremediably gone, at least with LMTP (and without relying to SiS)?
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 14-07-2015 14:44 Gionatan Danti ha scritto:
On 14/07/15 12:26, Steffen Kaiser wrote:
You asked about "newer dovecot versions", v2.2 does so.
Fair enough :)
So, with v2.2+ the hardlink approach is irremediably gone, at least with LMTP (and without relying to SiS)?
Dear list, sorry if I resume this (relatively) old post, but I would like to know if someone has some good suggestions/ideas.
A quick recap... System spec:
- CentOS 6.6 x64
- dovecot-2.0.9-8.el6_6.4.x86_64 RPM package/version
- sdbox mail store
I have two problems:
- hardlinks are not created when the same email is sent to multiple rcpt, nor when sending to the very same user (a non-linked copy is kept both in Inbox and Sent)
- I would like to have an alternate storage and to enable compression on it only, leaving the main location without compression. It is possible?
Relative to the hard-link problem, I found that not using before_sieve and after_sieve solves my problem, so that hardlinks are created correctly. However, this is far from ideal because I lose global sieve rules (global/defaul rules are only applied when no customized rules exist).
Any idea to how to solve these two problem? Thanks.
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Am Dienstag, den 14.07.2015, 12:26 +0200 schrieb Steffen Kaiser:
On Tue, 14 Jul 2015, Gionatan Danti wrote:
On 14/07/15 08:17, Steffen Kaiser wrote:
On Mon, 13 Jul 2015, Gionatan Danti wrote:
On the other hand, private (per-user) sieve file works without interfering with hardlinks. In a similar manner, disabling sieve also permits dovecot to create multiple hardlinks for a single message.
Does someone know if newer dovecot versions change anything in this regard?
LMTP adds Delivered-To header, so all delivered messages are unique and you cannot hardlink messages regardless of Sieve.
If Dovecot LDA adds headers, too, I do not know.
Mmm... I'm using LMTP, but I can't find the "Delivered-To" header. Below you can see an example of successfully hard-linked email [1]
You asked about "newer dovecot versions", v2.2 does so.
I just updated my Dovecot 2.2.13 to the current 2.2.18 This config option was added in the meanwhile:
Which recipient address to use for Delivered-To: header and Received:
header. The default is "final", which is the same as the one given to
RCPT TO command. "original" uses the address given in RCPT TO's ORCPT
parameter, "none" uses nothing. Note that "none" is currently always
used
when a mail has multiple recipients.
#lmtp_hdr_delivery_address = final
Doestn't that mean if you set it to none that no Delived-To: header gets added then?
participants (3)
-
Felix Zielcke
-
Gionatan Danti
-
Steffen Kaiser