Hi
I've recently tried to use the Dovecadm backup to backup the emails.. with the following syntax
doveadm -Dv backup -u userx maildir:/mnt/nfs-backup/userx
Sounds to be OK with few emails... Some of them got a lot of emails and one f them got an error and stop !
dsync(userx): Debug: brain S: Import Trash: Import change
type=expunge GUID=1725fa475d774ee19cb98dfb6737b4f1 UID=24891
hdr_hash= result=GUIDs match
dsync(userx): Debug: brain S: Import Trash: Import change
type=expunge GUID=916ed110b4b1522868be6194f1ae36ff UID=24892
hdr_hash= result=GUIDs match
dsync(userx): Debug: brain S: Import Trash: Import change
type=expunge GUID=eb8d75c530a7b02fc26b494d9006c91b UID=24893
hdr_hash= result=GUIDs match
dsync(userx): Debug: brain S: Import Trash: Import change
type=expunge GUID=aee9155875c34861fd6500f1f2f51a26 UID=24894
hdr_hash= result=GUIDs match
dsync(userx): Debug: brain S: Import Trash: Last common UID=24894.
Delayed expunges=
dsync(userx): Debug: brain S: Import Trash: Saved UIDs:
dsync(userx): Debug: brain S: Import Trash: Finish update:
min_next_uid=24895 min_first_recent_uid=24895
min_highest_modseq=35344 min_highest_pvt_modseq=0
dsync(userx): Debug:
/mnt/nfs-backup/userx/.Trash/dovecot.index.cache: Compressed,
file_seq changed 1645803588 -> 1645803589, size=32, max_uid=24894
dsync(userx): Warning: Mailbox changes caused a desync. You may want
to run dsync again: Remote lost mailbox GUID
7e05c335174bf1608f0a02004eac7fb4 (maybe it was just deleted?)
dsync(userx): Debug: auth-master: conn
unix:/run/dovecot/auth-userdb: Disconnected: Connection closed (fd=10)I empty the trash... exactly the same problem...
Any idea why this ??
Thanks and regards,
Well no ..I thought that dsync was for synchro " realtime for 2 different places ?
Having no 2 machines in parallel ( Just a single machine ) , I thought that a backup at regular interval would be enough ?
So, a simple backup should be done by dsync finally ?
Do you recommend finally NOT to do a backup ( Doveadm backup ) but a replication process ? ( https://wiki.dovecot.org/Replication ) ?
On 2/28/22 06:24, Ben Burk wrote:
I'm not sure what you are attempting to do here. It looks like you just ran a doveadm backup and the process completed for userx with a warning that the remote system (your nfs mount) lost a particular mailbox (possibly your indexes changed or a mailbox was deleted). From the logs you pasted it appears the process completed normally.
I personally do not use dovecot's backup or replication processes. If I needed to I would use its replication process to sync active data between multiple systems, but I have no need for this as of yet. Personally I chose to create offsite backups using rsync a long time ago, as rebuilding a mailbox (reindexing) is very simple.
Try running doveadm mailbox status -u userx guid '*' as the mailbox administrator and see if you can find that GUID, 7e05c335174bf1608f0a02004eac7fb4. Also, see if the backup you've written to nfs has the GUID.
On 2/27/22 23:33, Stephane Magnier wrote:
I've seen in a previous post, that the fact to do an RSYNC might break the index.. So, I've heard that this is not recommended. that's the reason why I decided to find a way to do a "clean" backup and be able to come back online if needed.
So, do you use an RSYNC and in case of restoring the mailbox, do you do a simple
*doveadm* *index* *-u* *USERx* *INBOX *And that's it ? works fine ?**2) I will try the Backup , or Sync.. locally.. Effectively, I don't know where the problem comes from..I have effectively an NFS Mount for the mailboxes and a VM for the Email server and that could be another another point of failure :-*( *Thanks for sharing.. I am a bit in a rush... I realized that my backup maybe not correct.. and I prefer not to discover it while running into trouble..**
On 2/28/22 19:03, Ben Burk wrote:
"Stephane" == Stephane Magnier <steph.mag220@netcourrier.com> writes:
Sorry, I deleted your most recent email post before I could reply. But why don't you just do 'imapsync' instead from your production dovecot box to some other backup system? Otherwise I'd probably work to setup dovecot's own replication but only have it go one way.
For example, I've got a VPS out in the cloud for my email, and I should probably back it up to my home system using replication, but it would be strictly primary->secondary. I wouldn't be trying to run two primaries replicating between each other.
Imapsync would be an improvement over rsync because it works within dovecot, so you'd get a more consistent view, but maybe not quite as upto date. But how important is your email if you worry about losing 20 minutes worth of it? If it's that critical, then you should be investing in a more robust setup.
John
Stephane> I've seen in a previous post, that the fact to do an RSYNC Stephane> might break the index.. So, I've heard that this is not Stephane> recommended. that's the reason why I decided to find a way Stephane> to do a "clean" backup and be able to come back online if Stephane> needed.
Stephane> So, do you use an RSYNC and in case of restoring the mailbox, do you do a simple
Stephane> doveadm index -u USERx INBOX
Stephane> And that's it ? works fine ?
Stephane> 2) I will try the Backup , or Sync.. locally.. Effectively, I don't know where the problem comes from..I have effectively an NFS Mount Stephane> for the mailboxes and a VM for the Email server and that could be another another point of failure :-(
Stephane> Thanks for sharing.. I am a bit in a rush... I realized that my backup maybe not correct.. and I prefer not to discover it while running into trouble..
Stephane> On 2/28/22 19:03, Ben Burk wrote:
Stephane> I'm not sure what you are attempting to do here. It looks like you just ran a doveadm backup Stephane> and the process completed for userx with a warning that the remote system (your nfs mount) Stephane> lost a particular mailbox (possibly your indexes changed or a mailbox was deleted). From the Stephane> logs you pasted it appears the process completed normally.
Stephane> I personally do not use dovecot's backup or replication processes. If I needed to I would use Stephane> its replication process to sync active data between multiple systems, but I have no need for Stephane> this as of yet. Personally I chose to create offsite backups using rsync a long time ago, as Stephane> rebuilding a mailbox (reindexing) is very simple.
Stephane> Try running doveadm mailbox status -u userx guid '*' as the mailbox Stephane> administrator and see if you can find that GUID, 7e05c335174bf1608f0a02004eac7fb4. Also, see Stephane> if the backup you've written to nfs has the GUID.
Stephane> On 2/27/22 23:33, Stephane Magnier wrote:
Stephane> Well no ..I thought that dsync was for synchro " realtime for 2 different places ? Stephane> Having no 2 machines in parallel ( Just a single machine ) , I thought that a backup at Stephane> regular interval would be enough ? Stephane> So, a simple backup should be done by dsync finally ? Stephane> Do you recommend finally NOT to do a backup ( Doveadm backup ) but a replication process ? Stephane> ( https://wiki.dovecot.org/Replication ) ? Stephane> On 2/28/22 06:24, Ben Burk wrote:
Stephane> Did you try running dsync? Stephane> On 2/27/22 23:15, Stephane Magnier wrote:
Stephane> HI, Stephane> Any idea ? Any clue ? Stephane> On 2/25/22 21:50, Stephane Magnier wrote:
Stephane> Hi Stephane> I've recently tried to use the Dovecadm backup to backup the emails.. with Stephane> the following syntax Stephane> doveadm -Dv backup -u userx maildir:/mnt/nfs-backup/userx Stephane> Sounds to be OK with few emails... Some of them got a lot of emails and one f Stephane> them got an error and stop ! Stephane> dsync(userx): Debug: brain S: Import Trash: Import change type=expunge Stephane> GUID=1725fa475d774ee19cb98dfb6737b4f1 UID=24891 hdr_hash= result=GUIDs Stephane> match Stephane> dsync(userx): Debug: brain S: Import Trash: Import change type=expunge Stephane> GUID=916ed110b4b1522868be6194f1ae36ff UID=24892 hdr_hash= result=GUIDs Stephane> match Stephane> dsync(userx): Debug: brain S: Import Trash: Import change type=expunge Stephane> GUID=eb8d75c530a7b02fc26b494d9006c91b UID=24893 hdr_hash= result=GUIDs Stephane> match Stephane> dsync(userx): Debug: brain S: Import Trash: Import change type=expunge Stephane> GUID=aee9155875c34861fd6500f1f2f51a26 UID=24894 hdr_hash= result=GUIDs Stephane> match Stephane> dsync(userx): Debug: brain S: Import Trash: Last common UID=24894. Delayed Stephane> expunges= Stephane> dsync(userx): Debug: brain S: Import Trash: Saved UIDs: Stephane> dsync(userx): Debug: brain S: Import Trash: Finish update: min_next_uid= Stephane> 24895 min_first_recent_uid=24895 min_highest_modseq=35344 Stephane> min_highest_pvt_modseq=0 Stephane> dsync(userx): Debug: /mnt/nfs-backup/userx/.Trash/dovecot.index.cache: Stephane> Compressed, file_seq changed 1645803588 -> 1645803589, size=32, max_uid= Stephane> 24894 Stephane> dsync(userx): Warning: Mailbox changes caused a desync. You may want to Stephane> run dsync again: Remote lost mailbox GUID 7e05c335174bf1608f0a02004eac7fb4 Stephane> (maybe it was just deleted?) Stephane> dsync(userx): Debug: auth-master: conn unix:/run/dovecot/auth-userdb: Stephane> Disconnected: Connection closed (fd=10)
Stephane> I empty the trash... exactly the same problem... Stephane> Any idea why this ?? Stephane> Thanks and regards,
Hi John,
Thanks for sharing. Effectively, I prefer to check out and test it before running into troubles.. and discovered that I have no backup up and running :-)
OK thanks for the tips.. re-indexation..."doveadm index -u USERx INBOX"
Running : [root@mbox1 xenia]# doveadm mailbox status -u USERx guid '*' Trash guid=f6e62d01c1847b6196f400004eac7fb4 Sent guid=a82520269daa79610a8a00004eac7fb4 Drafts guid=223789209daa79610a8a00004eac7fb4 Junk guid=2033a72b9caa7961038a00004eac7fb4 INBOX guid=2133a72b9caa7961038a00004eac7fb4 [root@mbox1 xenia]#
Sounds to be the mess ? :-) Different guid.. possibly
- non, I din't try DSYNC.. to be honest I don't which way to go :-)
OK.I will see this :
doveadm -Dv backup -u userx maildir:/mnt/nfs-backup/userx That might be the solution
with this syntax, I've seen the SSH version.. ? ( doveadm -Dv backup -u userx ssh root@1.1.1.1:/fold1/fold2 ) I don't know if this will work
In fact, I don't need a real time synchronization system. A simple backup would be enough... but I wish to be able to retrieve a system ASAP, which is down..
However, I am quite interested in knowing and studying a way to get a " real time " synchronization system; where you have 2 emails servers in parallel... but for now, a simple backup is enough
Thanks?? I will test some of your examples.
On 3/27/22 22:09, John Stoffel wrote:
I would not recommend using imapsync to do backups as it loses data. There is no way for imapsync to retain all data as for example IMAP protocol does not allow client to set email UID numbers. Backup made with imapsync is always partial copy of the original and cannotbe restored identically.
Sami
"Sami" == Sami Ketola <sami@ketola.io> writes:
Sami> I would not recommend using imapsync to do backups as it loses Sami> data. There is no way for imapsync to retain all data as for Sami> example IMAP protocol does not allow client to set email UID Sami> numbers. Backup made with imapsync is always partial copy of the Sami> original and cannotbe restored identically.
That's true, it's a bit brute force and not ideal. But if you can't setup properly doveadm replication, I would think that imapsync is better than rsync, unless you can pause dovecot/postfix, snapshot the filesystem, unpause dovecot/postfix then rsync the snapshot to your remote backup server.
But I also think that if you're that terrified about losing email, getting dovecot's replication working even just one way would be an improvement.
But the OP doesn't really state his requirements for getting back online. If it's a personal site (like mine) then I can take my time, it's not *that* critical. If it's a customer setup, then why aren't you running replication already, with dovecot director and shared resilient storage behind it?
participants (5)
-
Ben Burk
-
John Stoffel
-
Reuben Farrelly
-
Sami Ketola
-
Stephane Magnier