[Dovecot] dsync - one or two ways?
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations. Some things it can be used for:
- Initially transfer a mailbox to another server via SSH
- A faster sync done to an existing mailbox, sending only changes
- A superfast sync based on modification sequences.
- Source and destination mailboxes can use different formats (convert-tool will be history)
dsync can handle all kinds of conflicts in mailboxes, handle mailbox deletions, renames, etc. So it's safe to sync even if both source and destination mailboxes have had all kinds of changes.
Now, the question is: Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync? I think in typical cases where you'd think you would want only one-way sync are also the cases where there's no changes coming the other way in any case.
On Fri, 2009-07-17 at 11:59 -0700, Seth Mattinen wrote:
dsync probably wouldn't be such a good idea for this use case anyway. dsync assumes that an IMAP client has accessed the mailboxes in both source and destination, so if there are any UID conflicts (and in the case you describe there would be UID conflicts all the time) it gives new UIDs for the conflicting messages.
On Fri, 2009-07-17 at 12:06 -0700, Seth Mattinen wrote:
So is this targeted as a kind of replication solution then?
Either continuous replication or just one-off user moves/conversions without locking the mailbox and without losing changes done during the move/conversion.
I think even using dsync for backups should be considered as two-way replication. If there really are some new mails in your backup server that aren't in the primary server, it most likely means you just lost some mails on your primary server and you need to get them back.
Sure if you can think of some other uses for it I'd like to hear. But I if you were just planning on copying new mails from a server with completely different mails, I don't really think dsync should be used for that. Wouldn't it be easier to just have the mails forwarded directly there on SMTP level?
On Fri, 2009-07-17 at 15:57 -0400, Charles Marcus wrote:
Ah, but dsync catches those too!
If a message exists in one side but doesn't exist on the other side, dsync asks for a list of recently expunged messages. If it finds the message from there with same IMAP UID and same GUID, it's expunged from the other side too.
Basically the same will work with mailbox deletions and unsubscribes too. It'll just need to store a list of mailbox GUIDs there were deleted/unsubscribed.
On 7/17/2009 4:34 PM, Timo Sirainen wrote:
So, when will 2.0 be in any kind of semi-usable state (by non-programmer types)? ;)
Dunno. And I'm still hoping to break its index files code completely before stabilizing it :)
You're obviously having way too much fun... time for a break... ;)
Seriously though...
Will this be configurable on a per mailbox basis? Or is it intended to be server <-> server / all or nothing?
--
Best regards,
Charles
On Fri, 2009-07-17 at 17:01 -0400, Charles Marcus wrote:
Will this be configurable on a per mailbox basis? Or is it intended to be server <-> server / all or nothing?
You'll need to call it one user at a time in the proper environment (UNIX UID and GID set up right). How to do that and multiple users is your problem. :)
Here's an example command line for it:
dsync -e 'ssh hostname dsync dest' source
Or conversion to dbox (see how v2.0 has Postfix-like -o parameter for overriding settings):
dsync -e 'dsync -o mail_location=dbox:~/dbox dest' source
If only two-way sync is supported those source/dest parameters will go away.
On Jul 17, 2009, at 9:16 PM, Seth Mattinen wrote:
Well, by "continuous" I didn't actually really mean "continuous" but
"periodic" or something :) Meaning you'd have to run dsync every time
after something changes and separately for each user. So you could do
a bit laggy replication using it, but it's not really the permanent or
the most efficient solution for that.
On Jul 17, 2009, at 2:47 PM, Timo Sirainen wrote:
Wow, that will be very handy! Thanks!!
I think a two-way sync would be useful functionality to have,
perhaps tied to a command line option, at least for the sake of
completeness. It would also open up possibilities for having an
"offline" IMAP server, which might be useful for mobile workgroups
and such that may not always have full connectivity.
-Dave-- Dave McGuire Port Charlotte, FL
Timo Sirainen <tss@iki.fi> writes:
This sounds really awesome! How testable is it? My setup would be:
- one server dovecot that is online all the time, filtering email to several
mailboxes with sieve scripts
- the server dovecot would receive changes through a webmail instance
- a dovecot running on my laptop that I would like to be in sync with the server dovecot. I read mail there with Gnus.
Theres only a one user(me) that would have this setup, but there are a handful of other users as well. I have a couple of hundred mailboxes to sync. Can all mailboxes be synced automatically, or do I need to make a list of them for dsync?
-- Joakim Verona
On Fri, 2009-07-17 at 22:31 +0200, joakim@verona.se wrote:
This sounds really awesome! How testable is it?
By "testable" do you mean "is it working now"? It should work except for mailbox deletions and renames. But I haven't really tested it, just ran unit tests. :) And it pretty much requires using Dovecot v2.0 for IMAP so that it can handle expunges properly. And I wouldn't really recommend doing that yet..
Yeah, should work with that setup.
User's all mailboxes are synced automatically.
Timo Sirainen <tss@iki.fi> writes:
Would this setup work then?
- dovecot 1.2 on master and slave machines
- dsync on the slave machine, 2 way sync
- propagate expunges by some other means, like removing maildir mail files with unison or similar?
How would I handle mailbox addition, renaming and deletions?
-- Joakim Verona
On Sat, 2009-07-18 at 01:43 +0200, joakim@verona.se wrote:
Actually there shouldn't be problems with expunges after all. Conflicts just are detected less reliably. For example if:
Server 1 saves message with UID=123 Server 2 saves a different message with UID=123 Server 1 expunges messages UID=123
dsync is run to move changes from server 2 to server 1. It notices that "oh, UID=123 has been expunged from server 1 already" and server 2's UID=123 message isn't copied there. Or if you're syncing changes from 1 to 2, it would expunge the UID=123 from server 2.
If server 1 had been running Dovecot v2.0, the expunge record would have message's GUID stored and dsync would notice that there was a GUID conflict and copy the server 2's message to server 1 as UID=124.
How would I handle mailbox addition, renaming and deletions?
The reason why dsync can't do that without Dovecot v2.0 is because it can't be done reliably without storing extra information.. (And I haven't yet even implemented that to v2.0.) So the answer is: Not easily.
Timo Sirainen <tss@iki.fi> writes:
In my case mail would only arrive on the master. If I interpret you correctly this case will be very simple for dsync and will work already.
Will addition work automatically? Can renaming and deletion be made by renaming/deleting maildir folders manually?
I guess I can do the setup and test it myself :)
-- Joakim Verona
Timo Sirainen wrote:
dsync in Dovecot v2.0 tree is a new utility for syncing a mailbox in two locations.
I am very, very pleased to hear that you are working on this! Just at this moment, I am busy replacing a Dovecot IMAP based server with one with more powerful hardware, and until the tests are complete I want to operate both machines simultaneosly, so synchronisation is a big issue.
Does anyone want dsync to only sync changes from source to destination, instead of doing a full two-way sync?
If you want a decision "one-way XOR two-way", I'd definitely prefer the two-way sync option. If you can offer both sync types, all the better.
BTW, if you have not yet planned to include them, I suggest exception lists. The default would be to sync all messages in all folders for a given list of users, right? It would be nice to be able to configure dsync to achieve the following, too:
"Don't sync folders Trash, Junk, *.Newsletter, Boring.*" "Don't sync messages of a size greater than 2 MB"
What do you think?
-R
On Jul 18, 2009, at 2:20 AM, Ralph Seichter wrote:
This wouldn't be difficult to add.
"Don't sync messages of a size greater than 2 MB"
Why would you want that? Also if you later then wanted to sync it
anyway, it would have to get a new IMAP UID since messages can't be
inserted, and if client had already downloaded that it would have to
do it again. So I don't think this is such a good idea.
Timo Sirainen wrote:
"Don't sync folders Trash, Junk, *.Newsletter, Boring.*"
This wouldn't be difficult to add.
That's good to hear.
"Don't sync messages of a size greater than 2 MB"
Why would you want that?
To save space? I was thinking about which message properties might be useful for filtering, and message size bubbled up first. Lotus Notes/ Domino allows replicating databases with a maximum document size filter, which is useful for low-bandwidth connections or if you can't afford a volume flat rate for your mobile phone.
Well, I wouldn't want a feature to cause trouble. It was just an idea. The folder based filtering however is important to me, so I am glad that you consider it a feasible suggestion.
-R
participants (7)
-
Charles Marcus
-
Christian Felsing
-
Dave McGuire
-
joakim＠verona.se
-
Ralph Seichter
-
Seth Mattinen
-
Timo Sirainen