Thanks Sam

Users not using new mail server yet.

Both old and new using maildir.

Sounds like rsync is the way to go.

Thanks for your help.

James.

Sent from my iPhone XI

On 4 Sep 2018, at 5:41 pm, Sami Ketola <sami.ketola@dovecot.fi> wrote:

...

On 4 Sep 2018, at 10.20, James Brown <jlbrown@bordo.com.au> wrote:

I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:

IMAP-Sync Rsync Doveadmin backup and dsync

What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.

Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.

Does anyone have any advice or examples?

It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server. If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.

imapsync always loses data.

Sami

How do you test the new server?

(Sorry, this seems like it must be obvious.  The server I set up is the first one handling this domain, so testing was easy in that it didn't have history.)

Jeff Abrahamson +33 6 24 40 01 57 +44 7920 594 255

http://p27.eu/jeff/ http://transport-nantes.com/

On 04/09/18 10:49, Reio Remma wrote:

If you have lots of data, you can make an initial pass with rsync whilst the old server is running and another rsync pass (with --delete, use dry run to test) when you've shut down the old server to sync the changes since the initial pass.

Good luck! Reio

On 04/09/2018 11:40, James Brown wrote:

...

Thanks Sam

Users not using new mail server yet.

Both old and new using maildir.

Sounds like rsync is the way to go.

Thanks for your help.

James.

Sent from my iPhone XI

...

On 4 Sep 2018, at 5:41 pm, Sami Ketola <sami.ketola@dovecot.fi> wrote:

...

On 4 Sep 2018, at 10.20, James Brown <jlbrown@bordo.com.au> wrote:

I’d like to migrate to a new server. I only need to transfer the mail store, have already done the users. I’ve seen different recommendations:

IMAP-Sync Rsync Doveadmin backup and dsync

What are the pros and cons of each? What would be best for me? I’m thinking that Doveadmin backup is the way to go.

Both old and new servers are on same network. New server is running Dovecot 2.3.2.1, old server is 2.2.32. macOS X.

Does anyone have any advice or examples? It depends. Are your users already receiving mails to the new server? If yes, then the only way is to use "doveadm sync -1" or data will be lost. Even then data may be lost as UID numbers might already be used in the new server. If your users are not receiving mails to the new server it still depends. Is the mail storage format same in both servers? If yes, then it's probably fastest to use rsync. If not, then you need to use doveadm backup.

imapsync always loses data.

Sami

-- Tervitades Reio Remma

MR Stuudio 25 aastat

*MR Stuudio OÜ* Tondi 17b, 11316, Tallinn Tel +372 650 4808 Mob +372 56 22 00 33 reio@mrstuudio.ee www.mrstuudio.ee

--

Jeff Abrahamson +33 6 24 40 01 57 +44 7920 594 255

http://p27.eu/jeff/ http://transport-nantes.com/

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

imapsync always loses data

never saw this, be carefull by anounce such myths

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

...

On 4 Sep 2018, at 17.47, Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

...

imapsync always loses data

never saw this, be carefull by anounce such myths

It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.

but using "looses data" might others think it also may fail with imap, so be more detailed next time

Sami

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Am 04.09.2018 um 17:18 schrieb Sami Ketola:

...

On 4 Sep 2018, at 18.00, Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

...

...

On 4 Sep 2018, at 17.47, Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

...

imapsync always loses data

never saw this, be carefull by anounce such myths

It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.

but using "looses data" might others think it also may fail with imap, so be more detailed next time

UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

Sami

Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now

so this is my answer to topic

"Best way to move mail from one server to another"

after all there is no "best way", there are a few ways choose the one which fits best to your needs goals and starting points ( kind of servers etc )

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Quoting Robert Schetterer <rs@sys4.de>:

Am 04.09.2018 um 17:18 schrieb Sami Ketola:

...

...

On 4 Sep 2018, at 18.00, Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 16:52 schrieb Sami Ketola:

...

...

On 4 Sep 2018, at 17.47, Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 09:41 schrieb Sami Ketola:

...

imapsync always loses data

never saw this, be carefull by anounce such myths

It is a fact. Imapsync works over IMAP only and IMAP protocol
does not even support transferring all data. At most at least UID
numbering will be lost and end users need to invalidate their
local caches.

but using "looses data" might others think it also may fail with imap, so be more detailed next time

UID number is also data that is saved on the IMAP backend. If that
is lost then it's "lost data".

Sami

Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now

so this is my answer to topic

"Best way to move mail from one server to another"

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.

Quoting Sami Ketola <sami.ketola@dovecot.fi>:

...

On 4 Sep 2018, at 18.45, Rick Romero <rick@havokmon.com> wrote:

  https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.

No it does not. This is different thing. This is about managing
duplicates on multiple syncs. Imapsync seems to keep track of UID
-> MAIL mapping for server A and UID -> MAIL mapping for server B so
that on multiple runs it does not sync the same messages again.
However it does not use the same UID -> MAIL for server B as is
originally on server A. And that is the problem.

That's not what you said, you said there was a client sync issue. 
That's definitely an issue (especially if you run Mac Mail).

Ignoring duplicates on multiples runs is a non-issue, even with the
2011 version.  Recording UIDs is not necessary for that.

Rick

Sami is correct. imapsync loses data.

Source -> imapsync -> destination : the mailboxes are different to an IMAP client Source -> dsync -> destination (running Dovecot w/doveadm): the mailboxes are the same

You may not care that your users w/100,000 message mailboxes that have been cached on their client now have to invalidate that entire cache (and most likely re-download all of those messages).

So there is zero argument that, due to IMAP metadata loss, this is something that doesn't happen if dsync (or rsync) is used.

michael

On September 4, 2018 at 9:45 AM Rick Romero <rick@havokmon.com> wrote:

Quoting Robert Schetterer <rs@sys4.de mailto:rs@sys4.de >:

    > > 

...

    Am 04.09.2018 um 17:18 schrieb Sami Ketola:

        > > > 

...

            > > > > 

...

            On 4 Sep 2018, at 18.00, Robert Schetterer <rs@sys4.de mailto:rs@sys4.de > wrote:

            Am 04.09.2018 um 16:52 schrieb Sami Ketola:

                > > > > > 

...

                    > > > > > > 

...

                    On 4 Sep 2018, at 17.47, Robert Schetterer <rs@sys4.de mailto:rs@sys4.de > wrote:

                    Am 04.09.2018 um 09:41 schrieb Sami Ketola:

                        > > > > > > > 

> imapsync always loses data > > > > > > > > never saw this, be carefull by anounce such myths

                > > > > > It is a fact. Imapsync works over IMAP only and IMAP protocol does not even support transferring all data. At most at least UID numbering will be lost and end users need to invalidate their local caches.

            > > > > but using "looses data" might others think it also may fail with imap,

            so be more detailed next time

        > > > UID number is also data that is saved on the IMAP backend. If that is lost then it's "lost data".

        Sami

    > > Sorry i migrated terrabytes of mail with imapsync and never had a

    problem, it works as designed, also with maildir rsync did a good job,
    what never worked as it should was dsync ,cause of bugs ,that may
    changed now

    so this is my answer to topic

    "Best way to move mail from one server to another"
     

> 

https://imapsync.lamiral.info/FAQ.d/FAQ.Duplicates.txt

Seems to use UIDs so that 'data' isn't lost.

On 9/4/18 12:27 PM, Sami Ketola wrote:

...

On 4 Sep 2018, at 18.38, Robert Schetterer <rs@sys4.de> wrote:

Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now

I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.

Sami

why does not rsync work?

Ruben

-- So many immigrant groups have swept through our town that Brooklyn, like Atlantis, reaches mythological proportions in the mind of the world - RI Safir 1998 http://www.mrbrklyn.com DRM is THEFT - We are the STAKEHOLDERS - RI Safir 2002

http://www.nylxs.com - Leadership Development in Free Software http://www.brooklyn-living.com

Being so tracked is for FARM ANIMALS and and extermination camps, but incompatible with living as a free human being. -RI Safir 2013

Am 04.09.2018 um 18:27 schrieb Sami Ketola:

...

On 4 Sep 2018, at 18.38, Robert Schetterer <rs@sys4.de> wrote:

Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now

I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.

Sami

I think you should know the dove buglist, i used dovecot since the beginning and believe me dsync had a lot of problems, as i stated i hope this had changed now. I didnt used/tested it again the last years cause i had no big migration job to do, we ever tried to use it that days cause youre right we would have prefer not broke pop3 stuff at migration. But it never worked, so imapsync was the best way to go, also having courier, cyrus migrations with format/namespace change etc.

Imapsync works as designed and migration might not be ideal at some points but it has no real stoppers if migration is well prepared and organized.

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

Am 05.09.2018 um 07:37 schrieb Aki Tuomi:

...

On 04 September 2018 at 23:20 Robert Schetterer <rs@sys4.de> wrote:

Am 04.09.2018 um 18:27 schrieb Sami Ketola:

...

...

On 4 Sep 2018, at 18.38, Robert Schetterer <rs@sys4.de> wrote:

Sorry i migrated terrabytes of mail with imapsync and never had a problem, it works as designed, also with maildir rsync did a good job, what never worked as it should was dsync ,cause of bugs ,that may changed now

I would like to very much hear about the problems you had with dsync as we have successfully migrated tens of petabytes of mails with it successfully while preserving the UID numbers. Imapsync would have been totally out of question as it is absolutely crucial in those migrations to keep the UID -> MAIL pairs matching legacy server. And usually also POP3 UIDLs. If not keeping the data the servers would have literally melted under load when switching over then of thousands of users forcing them to redownload headers of even mail bodies for millions of mails.

Sami

I think you should know the dove buglist, i used dovecot since the beginning and believe me dsync had a lot of problems, as i stated i hope this had changed now. I didnt used/tested it again the last years cause i had no big migration job to do, we ever tried to use it that days cause youre right we would have prefer not broke pop3 stuff at migration. But it never worked, so imapsync was the best way to go, also having courier, cyrus migrations with format/namespace change etc.

Imapsync works as designed and migration might not be ideal at some points but it has no real stoppers if migration is well prepared and organized.

Best Regards MfG Robert Schetterer

There has been progress with dsync in the past years, and it can do courier and cyrus migrations with format/namespace changes. "Few years ago" is a long time.

we will give a retry end of this year and report

Aki

...

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG, 80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein

-- [*] sys4 AG

https://sys4.de, +49 (89) 30 90 46 64 Schleißheimer Straße 26/MG,80333 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief Aufsichtsratsvorsitzender: Florian Kirstein

Finally have some time to review list emails...

On Tue Sep 04 2018 03:41:50 GMT-0400 (Eastern Standard Time), Sami Ketola <sami.ketola@dovecot.fi> wrote:

imapsync always loses data.

Hi Sami,

Can you expand on this?

I used ImapSync to migrate from Dovecot to Office365 a couple of years ago, and didn't notice any issues with it at all.

Communigate Pro has a nice Migration/MoveIMAPMail utility. As Communigate is free to use for up to five accounts, I think it is legal to install it and use the MoveIMAPMail utility.

I have used it once to migrate mail from Communigate to dovecot.

-- Victor Sudakov, VAS4-RIPE, VAS47-RIPN 2:5005/49@fidonet http://vas.tomsk.ru/

I used (had to) imapsync two weeks ago to move Mails from dovecote 2.1 to 2.2, as doveadm failed (different versions issue).

Worked as expected. But all mails 60GB got resynced (summed up to approx. 250GB due to several accounts being used across multiple clients).

Tried both - without and with uid option. With latter one I did not have any duplicate mails being skipped; resync however seemed to be unavoidable.

-M

Am 26.09.2018 um 16:18 schrieb Tanstaafl <tanstaafl@libertytrek.org>:

Never mind, should have waited and read the entire thread...

On Wed Sep 26 2018 09:52:26 GMT-0400 (Eastern Standard Time), Tanstaafl <tanstaafl@libertytrek.org> wrote:

...

Finally have some time to review list emails...

On Tue Sep 04 2018 03:41:50 GMT-0400 (Eastern Standard Time), Sami Ketola <sami.ketola@dovecot.fi> wrote:

...

imapsync always loses data.

Hi Sami,

Can you expand on this?

I used ImapSync to migrate from Dovecot to Office365 a couple of years ago, and didn't notice any issues with it at all.