[Dovecot] Plugin issue with update from 2.0.19 to 2.1.17

Thu Oct 17 17:05:18 EEST 2013

Hi Charles.

I don't think that is the same issue...

I tried to set the maildir_broken_filename_sizes=yes and have tryed the Timo fix patch (attached here: http://www.dovecot.org/list/dovecot/2012-March/064731.html) but the index continues to recalculate the size.

This is after the Timo fix patch:

Oct 16 20:50:13 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Cached message size smaller than expected (367 < 529)
Oct 16 20:50:13 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381967409.M54518P30563.teste,S=367,W=378:2,S to /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381967409.M54518P30563.teste,S=529:2,S
Oct 16 20:50:13 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Corrupted index cache file /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 8

And this is after the maildir_broken_filename_sizes=yes conf:

Oct 17 12:21:04 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Cached message size smaller than expected (368 < 530)
Oct 17 12:21:04 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381969983.M452882P30597.teste,S=530:2,S to /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381969983.M452882P30597.teste,S=530:2,S
Oct 17 12:21:04 teste dovecot: imap(reinaldo at exemplo.com.br): Error: Corrupted index cache file /storage/teste/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 11

So... I noticed that after the maildir_broken_filename_sizes=yes i still got the error but there no changes on the S flag of the filename.

Is the dovecot.index.cache used for text searching too? Because if it is i think that i really need some way to make my plugin informs the correct size to dovecot, as i think that works on dovecot 2.0.19.

PS: I had to change some calls on my plugin in order to comply with some changes on dovecot 2.1.17.  That is the list of the calls that had been changed... Could be that I had  make another choice for values of the functions?

Before (2.0.19):
-        return zmail->super.get_stream(_mail, hdr_size, body_size, stream_r);
After (2.1.17):
+        return zmail->super.get_stream(_mail, 1, hdr_size, body_size, stream_r);

Before (2.0.19):
-    if (zmail->super.get_stream(_mail, NULL, NULL, &input) < 0){
After (2.1.17):
+    if (zmail->super.get_stream(_mail, 1, NULL, NULL, &input) < 0){

Before (2.0.19):
-            full_input[1] = i_stream_create_fd(fd1, 0, TRUE);
After (2.1.17):
+            full_input[1] = i_stream_create_fd(fd1, (size_t)-1, TRUE);

Before (2.0.19):
-            if (zmail->super.get_stream(ctx->dest_mail, NULL, NULL, &input) >=  0){
After (2.1.17):
+            if (zmail->super.get_stream(ctx->dest_mail, 1, NULL, NULL, &input) >=  0){

Before (2.0.19):
-    return o_stream_create(&emexis_stream->ostream);
After (2.1.17):
+    return o_stream_create(&emexis_stream->ostream, NULL);

And after define:
    mail->v.get_stream = myplugin_permail_get_stream;

i have this function declaration on 2.0.19:
static int myplugin_permail_get_stream(struct mail *_mail,
        struct message_size *hdr_size,
        struct message_size *body_size,
        struct istream **stream_r)
{

and this on 2.1.17:
static int emexis_permail_get_stream(struct mail *_mail,
        bool get_body ATTR_UNUSED,
        struct message_size *hdr_size,
        struct message_size *body_size,
        struct istream **stream_r)
{

> CC: dovecot at dovecot.org
> From: spork at bway.net
> Subject: Re: [Dovecot] Plugin issue with update from 2.0.19 to 2.1.17
> Date: Tue, 15 Oct 2013 17:15:05 -0400
> To: ramatukuma at hotmail.com
> 
> 
> On Oct 15, 2013, at 4:33 PM, Reinaldo Matukuma wrote:
> 
> > Hello. Probably only Timo can help-me with this.
> > 
> > I have a self-made plugin based on the zlib plugin that i use to cryptograph the messages at inbox.
> > 
> > As a side-effect of the cryptography, my plugin changes the size of the message, but until 2.0.19 this works well with dovecot index and the W/S flags.
> > 
> > But now, i'm going to upgrade to 2.1.17 and now i have these messages on log at my test ambiance:
> > 
> > Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Cached message size smaller than expected (367 < 529)
> > Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Maildir filename has wrong S value, renamed the file from /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=367,W=378:2,S to /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/cur/1381879158.M634385P5208.test,S=529:2,S
> > Oct 15 20:19:25 test dovecot: imap(reinaldo at exemplo.com.br): Error: Corrupted index cache file /storage/test/messages/exemplo.com.br/reinaldo/Maildir/.Sent/dovecot.index.cache: Broken physical size for mail UID 6
> 
> I think there's something else up, this looks quite similar to what I reported the other day:
> 
> http://dovecot.org/list/dovecot/2013-October/092917.html
> 
> Do you get further messages after the "broken physical size" indicating that dovecot is then trying to open the file based on the original filename?
> 
> Regardless, setting this should turn off the file size/name corrections, perhaps that will help:
> 
> maildir_broken_filename_sizes=yes
> 
> Charles
> 
> > 
> > I understood with these messages that the dovecot is arguing to get the real size of the file now, overriding the return of size from my crypto plugin. But i don't understood if this is a consequence from the changes to correct the index issue (the dovecot.index.cache issue).
> > 
> > So... I want know if the correct way to fix this would be change my plugin to return the real size of the file (that will be larger than the effective message that the plugin returne after the de-cryptography) or if i need use some new  function to the plugin replace the expected size based on the real size of the message, not of the file.
> > 
> > Thanks
> > 
> > Reinaldo
> > 		 	   		  
> 