[Dovecot] Questions regarding dbox migration
It has been my wish to move to dbox for a long time hoping to reduce the number of writes which is really killing us.
Now I wonder what may be the best way of doing so. I'm considering some sort of intermediate migration where the existing Maildir users are changed to single-dbox and then later upgraded to 2.0 and changed to multi-dbox when it becomes stable. But is this a reasonable task to perform on a production system or at all? The alternative is to wait for 2.0 to become completely stable and then go all the way at once. I would however much prefer to take it one step at a time and I think it would be safer to start out with single-dbox as it appears less complex.
Now the big question is whether multi-dbox and single-dbox are compatible formats. If a Maildir->dbox migration is made on a system running dovecot v. 1.1, would it then be trivial later changing to multi-dbox after upgrading to 2.0 or is a completely new migration then needed? Would this scenario be much different if the system is upgraded to version 1.2 before the change to single-dbox?
Kind regards, Mikkel
On Oct 14, 2009, at 7:03 AM, Mikkel wrote:
I'd wait for v2.0 if possible. v1.2's dbox has a couple of problems:
- antispam plugin doesn't work
- copying messages can't currently be done by hard linking, so
copying uses more disk I/O (this could be fixed somewhat easily
though, just haven't had time) - flags are backed up to individual dbox files "once in a while", but
probably practically pretty often so that also increases disk I/O usage
v2.0 already solved 3 for single+multi-dbox, 2 for multi-dbox and I'm
not sure about 1.
Now the big question is whether multi-dbox and single-dbox are
compatible formats.
Kind of, but not practically.
Migrating from single-dbox to multi-dbox isn't any easier than maildir
-> multi-dbox.
I'm trying to get v2.0.0 out pretty quickly though. v2.0.beta1 should
hopefully be out in less than a month. The main problem with it is
actually how to make it enough backwards compatible that everyone
won't start hating me.
On 10/14/2009, Timo Sirainen (tss@iki.fi) wrote:
I'm trying to get v2.0.0 out pretty quickly though. v2.0.beta1 should hopefully be out in less than a month.
Great news on how fast 2.0 is coming along! I finally got the go ahead to convert my biggest client, but he wants to hold off (political reasons) for a few months, so I think I'll just plan on starting to test 2.0 soon, and making that the target for his installation.
The main problem with it is actually how to make it enough backwards compatible that everyone won't start hating me.
I honestly don't see that ever happening Timo - and please, don't go down the same road Microsoft did with Windows.
Not that you need any help, but I would suggest to just focus on making the conversion tools (mbox/maildir > (m)dbox, old config > new config, etc) rock solid, and document any incompatibilities really well.
--
Best regards,
Charles
At 12:18 PM 10/14/2009, Charles Marcus wrote:
It'd be nice to think you were kidding, but I'll be you're not.
I'd guess that there's no two "the same" configs of dovecot out there. I can't see any way to test for all of them. Heck, documentation for any particular environment/config, while abundant, is never quite exactly right because of that. (and for that matter I never could find one set of docs that covered everything I needed for a complete setup - I hate to merge dozens of docs on implementation). Dovecot is just a component of a larger mail infrastructure that's part of the zillions of choices in open source.
And that's going to suck. Mail is probably the most critical business app out there. Upgrading with incompatibilities is going to be a nightmare for me, and that's even considering that I have a parallel pre-production environment.
But it comes with using open source software (and quality code at that), so I live with it.
Rick
Timo Sirainen skrev:
From your comments it appears like dbox and mdbox are quite different in many ways. Is mdbox going to replace dbox completely or are you expecting to keep both formats? My point is: what's going to be the difference between dbox and mbox with mdbox_rotate_size set small enough to allow only one mail per file?
That is great news. I'm looking very much forward to make the shift to dbox/mdbox. A new mail storage format is really, really, really interesting from my point of view. CPU and RAM are both very cheap by now but disk I/O remains as expensive as always and really is the only limiting factor these days and is what's driving up the costs of running any larger mail system. The performance gains you mentioned previously this year would probably cut the total datacenter costs in half and that is quite an accomplishment.
How stable do you think dbox and mdbox are at the moment?
Regards, Mikkel
On Wed, 2009-10-14 at 21:28 +0200, Mikkel wrote:
The main difference is that mdbox needs to lock files when saving messages. That's not especially nice with NFS. Single-dbox currently locks index files, but it can be made entirely lockless eventually. There are also some other differences like:
- in mdbox all messages exist in storage/ directory, while in dbox messages exist in separate mailbox directories
- mdbox has a separate storage/dovecot.map.index that needs to be updated
- in mdbox message can be copied by doing a couple of small appends to index files, while with dbox the file needs to be hard linked (and currently even that's not done)
How stable do you think dbox and mdbox are at the moment?
mdbox passed 16 hours of imaptest stress testing with zero errors. I'm thinking about starting to use it at some point for my own mails. dbox in v1.2 should also be stable, but in v2.0 last I tried imaptest could break it, I should fix it someday soon.
Timo Sirainen skrev:
So basically you prefer mdbox but are maintaining dbox because of its almost lockless design which is better for NFS users?
Do you consider it to be viable having two different dbox formats or are you planning to keep only one of them in a long term perspective?
Regards, Mikkel
On Wed, 2009-10-14 at 23:04 +0200, Mikkel wrote:
I'm planning on keeping both of them. And it's not necessarily only because of NFS users. Multi-dbox was done mainly because filesystems suck (mailbox gets fragmented all around the disk). Maybe if filesystems in future suck less, single-dbox will be better. Or perhaps SSDs make the fragmentation problem mostly irrelevant.
And note that there are no real world statistics on how much faster multi-dbox is compared to single-dbox (or maildir). Maybe the difference isn't all that big after all. Or maybe it's a lot bigger than I thought. I've no idea.
Timo Sirainen skrev:
You are talking about directories being fragmented right? In case of mdbox wouldn't you have the very same problem since larger files may be fragmented all over the disk just like many small files in a directory might?
I think the impact on imap operations and mail delivery probably would very little due to bigger files.
But pop3 users just download everything once in a while and should benefit tremendously from just having to read one file sequentially as opposed to read many small files.
So I can definitely see the point in mdbox but I better stay away from it, using NFS... :/
Regards, Mikkel
On Wed, 2009-10-14 at 23:59 +0200, Mikkel wrote:
Files in a directory written to different parts of disk, yes.
I guess this depends on filesystem. But the files would typically be about 2 MB of size. I think filesystems usually copy more data around to avoid fragmentation.
In any case if there are expunged messages, files containing them would be recreated (nightly or something). That'll unfragment the files.
And finally one thing I've also been thinking about has been that perhaps new mails could be created into separate individual files. A nightly run would then gather them together into a larger file.
So I can definitely see the point in mdbox but I better stay away from it, using NFS... :/
What kind of index related errors have you seen in logs? Dovecot can handle most index corruptions without losing much (if any) data. Everything related to dovecot.index.cache can at least be ignored.
Timo Sirainen skrev:
It would be nice if this recreation interval (nightly, weekly, monthly?) was made tunable. Some users would have mailboxes a several hundred megabytes and having to recreate thousands of these every night because of a single mail getting expunged a day could result in a huge performance hit.
I think this could be a great idea in some setups and a pretty bad one in others. In my setup for instance incoming emails account for about half the disk write activity and though activity is somewhat lower during the night there still is a lot of activity around the clock.
In the proposed design all the incoming emails would have to be written to the disk twice (if I got it right?) and at a time when there would still be a relatively high activity (because there always is).
So if this should increase performance overall there would have to be a somewhat large initial gain in order to justify the double writing. Also most emails are either read shortly after arriving or not at all so the primary access to the mails would happen while the emails are still located in single files.
My point is that there could be many reasons why such a design might actually lead to poorer performance so this should probably be tested extensively before being implemented.
But this could be a really nice solution if implemented in such a way the incoming mails are stored in single files on a separate configurable device (which would optimally be a flash device) and then moved to the actual storage at night.
The errors I get are like these two:
dovecot: Oct 01 15:13:05 Error: POP3(account@domain): Transaction log file /local/account_homedir/Maildir/dovecot.index.log: marked corrupted
dovecot: Oct 01 15:13:57 Error: IMAP(another_account@domain): Corrupted transaction log file /local/another_homedir/Maildir/dovecot.index.log seq 229: duplicate transaction log sequence (229) (sync_offset=32860)
Regards, Mikkel
On Thu, 2009-10-15 at 10:55 +0200, Mikkel wrote:
It's actually left up to sysadmin to run "doveadm purge -a" from cronjob.
It doesn't matter what the user's full mailbox size is. It matters how many of these max. ~2 MB dbox files have expunged messages. Typically users would expunge only new mails, so typically there would be only a single file that needs to be recreated.
This alone isn't an error, just says that error happened earlier.
Hmm. This means that Dovecot thought that dovecot.index.log file was rotated, but when it reopened it it actually had the same sequence number in the header. What NFS server are you using? One possible reason for this is if file's inode number suddenly changes.
Timo Sirainen skrev:
That way it seems pretty clever. But if the number of messages in each mdbox file (we are talking about multi-dox and not single-dbox right?) is configurable via mdbox_rotate_size (ranging from 1 to infinity I guess) then how can you assume that each file is no more than ~2 MB?
I'm using a NAS appliance (Sun S7410) that is basically just a modified Solaris 10 NFS server on x86 hardware (AMD). The system has had some stability issues and therefore it's difficult to tell whether that specific error happened during normal operation or if it's just a special case.
The client also runs Solaris 10 but this is the Sparc version.
Solaris uses NFSv4 as default (and I haven't changed this).
Anyway do you think that dovecot would be able to recover from that kind of error without loss of information if dbox/mdbox where used instead of Maildir?
Regards, Mikkel
On Fri, 2009-10-16 at 00:24 +0200, Mikkel wrote:
The default is 2 MB, and whoever sets it larger deserves what they get (for better or worse).
Well, it marks it corrupted so it loses some state. But I guess I could change the code to not mark it corrupted but instead just fail. Then it wouldn't lose anything.
Timo Sirainen <tss@iki.fi> wrote:
[...]
Do you play to support both ways migration tools? dbox <-> maildir(s)
-- [pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu "Unlike most net.puritans, however, I feel that what OTHER consenting computers do in the privacy of their own phone connections is their own business." -- John Woods, jfw@eddie.mit.edu
On Wed, 2009-10-14 at 21:39 +0200, Andrzej Adam Filip wrote:
convert plugin can already do that. In v2.0 I'm planning on getting rid of convert plugin and relying on dsync program. It can do two-way sync between any mailbox formats and between different machines if wanted (e.g. via ssh).
On Oct 14, 2009, at 7:03 AM, Mikkel wrote:
It has been my wish to move to dbox for a long time hoping to reduce
the number of writes which is really killing us.
BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should
reduce disk i/o, and I'm really interested in hearing how much.
On 10/14/2009, Timo Sirainen (tss@iki.fi) wrote:
BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how much.
What are the downsides? Also, I'm guessing maybe there are certain conditions where you definitely don't want to do it?
--
Best regards,
Charles
On Wed, 2009-10-14 at 12:18 -0400, Charles Marcus wrote:
The only reason you wouldn't want to set it is if cur/ directory is modified outside Dovecot. Or Dovecot is used without index files (or e.g. with NFS and local index files). The point being: Dovecot trusts index files to be up-to-date with cur/ directory's contents.
Although even then it still checks the cur/'s mtime to see if it had changed. The only problem is if Dovecot changes it at the same time as non-Dovecot. Then it doesn't know that there was a change. Unless the filesystem supports nano/microsecond mtimes, that'll make it very unlikely for Dovecot to miss this.
On 10/14/2009 12:24 PM, Timo Sirainen wrote:
BTW. Have you tried maildir_very_dirty_syncs=yes setting? That should reduce disk i/o, and I'm really interested in hearing how much.
What are the downsides? Also, I'm guessing maybe there are certain conditions where you definitely don't want to do it?
Perfect explanation, thanks. :)
Timo Sirainen skrev:
I don't think I've tried that one. Earlier on I experimented with fsync_disable=yes (which made a huge difference by the way) but that was before I started using mail_nfs_storage=yes and mail_nfs_index=yes
I would like to try using maildir_very_dirty_syncs=yes but is it advisable in combination with NFS?
Regards, Mikkel
On Wed, 2009-10-14 at 21:14 +0200, Mikkel wrote:
It should be fine with NFS if indexes are also on NFS. Although I just fixed a bug related to it: http://hg.dovecot.org/dovecot-1.2/rev/7956cc1086e1
Indexes on NFS are problematic now though if multiple servers can access the mailbox at the same time. mail_nfs_index=yes is supposed to help with that, but it's not perfect either. Long term solution would be for Dovecots in different machines to talk to each others directly instead of through NFS.
Timo Sirainen skrev:
The system is currently running dovecot version 1.1.19. Would you consider it safe to try it on that version as well?
Is worse now than previously?
I have been running at production setup with two servers accessing the same Maildir data from NFS without any problems for quite a while now. Load is spread randomly between the two servers so I can only assume that by coincidence they sometimes try to access the same mailbox. This has functioned quote well with many versions of the 1.1.x dovecot releases so unless some new issues have been introduced I don't think I should fear anything in that regard :-)
Regards, Mikkel
On Wed, 2009-10-14 at 23:18 +0200, Mikkel wrote:
Yes. v1.2.6 + these two patches should make the performance better:
http://hg.dovecot.org/dovecot-1.2/rev/ebdba086e3b1 http://hg.dovecot.org/dovecot-1.2/rev/7956cc1086e1
With dbox index corruption becomes a worse problem than with maildir, because index is the only location where message flags are kept. v2.0 creates dovecot.index.backup files every once in a while though.
And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors.
Timo Sirainen wrote:
And you've actually been looking at Dovecot's error log? Good if it doesn't break, most people seem to complain about random errors.
Well, it does complain once in a while but it has never resulted in data being lost in any way. But I guess your point is that this might happen with dbox under the same circumstances. A very good reason to wait for 2.0 I guess...
Regards, Mikkel
On Wed, 2009-10-14 at 23:52 +0200, Mikkel wrote:
That's the theory anyway. :)
How often are they created anyway?
Whenever dovecot.index file would normally get recreated, the old one is first link()ed to dovecot.index.backup. So that depends on various things. You could compare timestamp differences between dovecot.index and dovecot.index.log in your current system. Perhaps I should add some extra code that guarantees that .backup gets created at least once .. a day? week? .. (assuming there have been changes)
Timo Sirainen wrote:
I would like it to happen as often as possible unless you expect a lot of I/O to happen on this account. In that case perhaps it could be made a configurable option so the system administrator could decide for himself which risk/performance profile would fit in the specific situation?
Regards, Mikkel
participants (5)
-
Andrzej Adam Filip
-
Charles Marcus
-
dovecot＠corwyn.net
-
Mikkel
-
Timo Sirainen