On Thu, Jul 30, 2009 at 07:04:01PM +0200, Thomas Hummel wrote:

On Tue, Jul 28, 2009 at 07:30:27PM +0200, Thomas Hummel wrote:

...

I took for granted that it was that the client would download all messages again since it might be confused by some UID changes.

Is it correct ? Timo ?

...

But I don't really see why (on the IMAP protocol level) and I don't know how to reproduce it.

I'm asking this because I'm investigating if CONTROL may be on a local filesystem (for performance reason) instead of an NFS filesystem :

I'm at a point where postfix(+amavisd+clamav) and dovecot on the same machine is too much load. Let's say I've got 2 identical servers and I want to load balance postfix and dovecot between the two. One simple solution would be to :

. run postfix on one server (and having a not started ready to run dovecot on it) . run dovecot on another (and having a not started ready to run postfix on it)

In such a setup, I would want to quickly be able (if the server runing dovecot crashed) to run dovecot on the other. If CONTROL is local, that may mean starting from crash regarding dovecot-uidlist...

Another setup would be to run dovecot and postfix on both servers and to use DNS round-robin, counting on the fact that once a client has done a gethostbyname() or similar, it should have no reaon to change of server (maybe the fact that clients like TB open many connections would cause a problem though).

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Thu, Jul 30, 2009 at 01:33:27PM -0400, Timo Sirainen wrote:

I think it's a bad idea. Besides the UID problems causing all kinds of trouble with clients (some clients ignore UIDVALIDITY completely (Apple Mail, at least used to)

Woah! That's indeed stupid from them...

and the result is that opening a message from list opens a completely different mail)

That I don't get since, even if UIDALIDITY change, and even if new messages arrive in the mailbox between dovecot-uidlist erasement and recreation, since INTERNAL DATE is not supposed to change, same messages will get same UIDs won't they ? At least, that's what happens in my tests.

control files also contain dovecot-keywords which provides the maildir filename letter <-> keyword mapping.

Yes, I know about keywords, I didn't mentionned it though.

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 11:30:30AM -0400, Timo Sirainen wrote:

For example if you have mails with
UIDs 1, 2, 5 and 10 and you delete dovecot-uidlist and dovecot.index*,
Dovecot gives them UIDs 1, 2, 3, 4.

Ok, I get it.

But the only way I can think of a client can detect that UIDs are not the same for the same messages is by detecting that UIDVALIDITY has changed. So if only dovecot-uidlist gets deleted but index (which store UIDVALIDITY) survives, what will happen in the client ? Will he fetch the wrong data ?

Thanks.

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:

i.e. Dovecot gets also the next_uid from index file, and starts assigning UIDs beginning from it.

Clear. I probably deleted indexes as well in my test and didn't notice a new UIDVALIDITY.

But what about the client's cache ? Does it knows anything about UIDs ?

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 12:49:04PM -0400, Timo Sirainen wrote:

Most clients cache mails based on UIDVALIDITY and UID, so if UIDVALIDITY changes, it re-downloads all mails. If just UIDs change, then it's handled exactly the same way as if all messages just got expunged and added back, so again it'll re-download them.

You mean, one single UID differ between the client's cache and what the server sends and the whole mailbox is fetched again ?

So UIDVALIDITY is just basically the "instance number" of a mailbox, right ?

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:00:10PM -0400, Timo Sirainen wrote:

Then again, client doesn't know if UID changes. It handles those situations exactly the same way as if message was expunged and a new message was added.

So on what criteria does a client fetch a message ? Does it just ask anything above nextUID ?

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:18:17PM -0400, Timo Sirainen wrote:

Depends on client.

Ok but I don't understand why you say that a client, though caching UID and UIDVALIDITY, makes no difference between a new message and some UID change on the server ?

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:22:37PM -0400, Timo Sirainen wrote:

Because client thinks UID = message. If a message's UID changes, the client has no idea that it's still the same message.

Yes but what does and UID has changes from the client pint of view then ? You mean something like, in its cache, the client has a mapping like

slot 1 <-> UID u slot 2 <-> UID v ...

and he sees

slot 1 <-> UID z ?

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:34:59PM -0400, Timo Sirainen wrote:

Thanks for your explanations and patience Timo!

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, 2009-07-31 at 18:48 +0200, Thomas Hummel wrote:

On Fri, Jul 31, 2009 at 12:26:04PM -0400, Timo Sirainen wrote:

So, to come back at my initial question :

Is it correct to say that what's wrong in dovecot-uidlist lost is the potential lost of UIDVALIDITY and nextUID (if indexes are deleted as well) ?

Right.

I mean dovecot-uidlist lost alone has not much consequences as long as you still got indexes ?

Clients will still re-download all messages, although they're less likely to break completely. But I still don't understand how you could lose dovecot-uidlist but not indexes.

On Fri, 2009-07-31 at 18:59 +0200, Thomas Hummel wrote:

On Fri, Jul 31, 2009 at 12:50:11PM -0400, Timo Sirainen wrote:

...

But I still don't understand how you could lose dovecot-uidlist but not indexes.

In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?

But aren't both of them in local disk then?

On Fri, 2009-07-31 at 19:04 +0200, Thomas Hummel wrote:

On Fri, Jul 31, 2009 at 01:00:38PM -0400, Timo Sirainen wrote:

...

...

In a setup like mine where INDEXES and CONTROL point to different places for instance. Is there something wrong with such a solution ?

But aren't both of them in local disk then?

No. At the moment, the Maildirs and the control files are on NFS, indexes are on local disk. That's ok since I've got only one server for now.

Since I plan to split dovecot and postfix on 2 different servers but want to be able to use anyone of them back to run both postfix and dovecot if one of them crash, I plan to put indexes and control files on NFS.

OK. It just sounded like you were thinking about putting index files to NFS and control files to local disk. So there's really no normal situation where only control files could get lost.

On Fri, Jul 31, 2009 at 07:04:32PM +0200, Thomas Hummel wrote:

Since I plan to split dovecot and postfix on 2 different servers but want to be able to use anyone of them back to run both postfix and dovecot if one of them crash, I plan to put indexes and control files on NFS.

Another solution would be to DNS round-robin on 2 servers runing dovecot but I think I've read on the dovecot wiki that it wasn't a good idea. Instead, I understood that I could use the IMAP proxy to send the same user on the same server.

However, I wonder if it's not overkill since once the client has resolved the name, the TCP connexion has no reason to change ? But some clients open more than one connexion for one user. That can be a problem I guess...

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:15:07PM -0400, Timo Sirainen wrote:

Right, it'll easily cause all kinds of caching related problems even with the mail_nfs_*=yes settings (which also slow things down).

Ok.

And some users use multiple clients.

That's right.

BTW. If you're going to run Postfix on both servers and both deliver mails to any users, you're still going to have the same caching problems if you're using Dovecot's deliver.

No, I'm going to run postfix on one, dovecot on the other (and I don't use deliver for now - I know I should ;-)).

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Fri, Jul 31, 2009 at 01:20:17PM -0400, Timo Sirainen wrote:

Well, that's pretty much the same thing. I meant that if Dovecots on both servers touch the same mailbox at the same time, there are caching problems. Regardless of what part of Dovecot touches it.

Of course, sorry, I'm not used (yet) to think "deliver".

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau

On Thu, Jul 30, 2009 at 01:21:40PM -0400, Timo Sirainen wrote:

Just dovecot-uidlist, or also dovecot.index*?

Just dovecot-uidlist.

If you delete both, UIDVALIDITY is changed. If you delete only dovecot-uidlist, it'll probably preserve UIDVALIDITY and just give new UIDs to messages (because next_uid is also stored in index file).

I don't get that : do indexes store UIDVALIDITY ? nextUID isn't enough to guaranty mailbox instance uniqueness, right ?

And, same as in my previous answer : why would they get new UIDs ? As they will be "sorted" (maybe because of INTERNAL DATE) in the same order ? When I erase dovecot-uidlist manually, it gets recreated with the same UIDs (but I didn't delete indexes though)...

Well, Dovecot reads the uidlist file lazily. If the index files still exist and you don't try to open any mail or there are no new mails, then uidlist's existence isn't checked and UIDs won't change.

Same as above : I cannot get them to change in a test situation where I delete dovecot-uidlist, send a new mail and check mail again.

Thanks

-- Thomas Hummel | Institut Pasteur <hummel@pasteur.fr> | Pôle informatique - systèmes et réseau