[Dovecot] Overloading IMAP Server

Harford, Ken

21 Jan 2013 21 Jan '13

8:09 p.m.

Hi All, I am having an issue with dovecot IMAP reading and writing to one specific mailbox. I have a check email delivery script that sends out a email and then logs into dovecot to check if the mailbox that the message has been delivered to. However, the servers that send these messages send them in bunches depending on how many postfix spools it is checking. So I could get 60 messages all being sent to 1 mailbox and then having to read that same mailbox to check if the message actually got delivered.

I am using postfix as the MTA and dovecot is actually delivering the message to the mailbox

I am getting a lot of errors such as the following:

Could not connect to IMAP server imap failed: IMAP RECEIVE CRITICAL - Could not select INBOX

Any ideas as to what I can tweak in dovecot to allow these messages to be verified.

Thanks for any information or help you could provide!!

Ken

Show replies by date

Steffen Kaiser

22 Jan 22 Jan

10:04 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 21 Jan 2013, Harford, Ken wrote:

...

       I am having an issue with dovecot IMAP reading and writing to 
one specific mailbox. I have a check email delivery script that sends

What's the purpose of the verification:

The prompt delivery?
The delivery at all?
The correct delivery?

...

out a email and then logs into dovecot to check if the mailbox that the message has been delivered to. However, the servers that send these messages send them in bunches depending on how many postfix spools it is checking. So I could get 60 messages all being sent to 1 mailbox and then having to read that same mailbox to check if the message actually got delivered.

I am getting a lot of errors such as the following:

Could not connect to IMAP server imap failed: IMAP RECEIVE CRITICAL - Could not select INBOX

Is this an error logged by Dovecot?

...

Any ideas as to what I can tweak in dovecot to allow these messages to be verified.

a) Wait a bit before logging into the IMAP server and check the messages sequentionally, not in parallel. E.g. your script could synchronize itself with other parallel instances by a shared lock.

b) Use mail_log and a script on the mail server monitoring Dovecot logs and register delivered message ids and/or recipient. Then figure out a good way to probe, if one pariticular message has been seen.

-or- make note of message id/recipient on the sender side, somehow push (ssh or something) that information on regular basis to the mail server, where a script verifies that data.

c) To verify the correct delivery: use some sort of checksumming.

Kind regards,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUP5WEF3r2wJMiz2NAQIatAgAjwZR1VGnyURuncLRNeeY0dBaqWzAUcNK g4YoiXB6NLxP6Xr/QO6Pw+6Q3sECJ0CJdwImOz56FVFNuiOygUMQ1dCdZ+Tpy9Wu xeAdobrkOo2uzgRnUh7+WLeZ8k32gZZztCAQA90uTvB3MBox+7J/P8/mApp+8/tr JKyhUy8RJIYXH+yrw/ZkbjmOSEEkf/wTPnBQcMKw5/4odha4lu+yEKx766mm0YCe 3USL7eVqN0Q9FwUH+iCjnRJAdKRETP4CnfQpjVp7eiJX7HD0iNwhucDwf037Ev2N PG173M40pMllRyesjiV8gwOu8mAHu6V3QUiFJkYAKFOotUVO6xMrNQ== =SeaP -----END PGP SIGNATURE-----

Robert Schetterer

10:20 a.m.

Am 22.01.2013 10:04, schrieb Steffen Kaiser:

...

Hi , i think you like to do some monitoring ?

you may do it like this

if you have i.e xymon client on the mailserver and a central xymon monitoring server , write a server mailscript which sends some check mail i.e all 5 minutes from the monitoring server to the mailserver

then on the mailserver write some simple mailcheck monitoring client script if the mail arrived i.e in .new of the maildir mailbox i.e all 5 mins , go yellow if it failed for two times, go red if this fails for three times simply delete the test mail after check, use a special check mailbox which is not accessable from outbound ( postfix access table etc )

if you want additional imap/pop3 check with the test mail the check script could use i.e some imap/pop3 console mail reader perhaps mutt or pine etc for reading and deleting the test mail etc, you may integrate checksumming at some stage

i wouldnt deal with logs for this kind of monitoring, or you may do it addtional

i think there are nagios scripts allready ,which might work this way

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich

Robert Schetterer

10:27 a.m.

Am 22.01.2013 10:20, schrieb Robert Schetterer:

...

Am 22.01.2013 10:04, schrieb Steffen Kaiser:

...
On Mon, 21 Jan 2013, Harford, Ken wrote:

...
       I am having an issue with dovecot IMAP reading and writing
to one specific mailbox. I have a check email delivery script that sends
What's the purpose of the verification:

The prompt delivery?

The delivery at all?

The correct delivery?

...
out a email and then logs into dovecot to check if the mailbox that the message has been delivered to. However, the servers that send these messages send them in bunches depending on how many postfix spools it is checking. So I could get 60 messages all being sent to 1 mailbox and then having to read that same mailbox to check if the message actually got delivered.

...
I am getting a lot of errors such as the following:

...
Could not connect to IMAP server imap failed: IMAP RECEIVE CRITICAL - Could not select INBOX

Is this an error logged by Dovecot?

...
Any ideas as to what I can tweak in dovecot to allow these messages to be verified.

a) Wait a bit before logging into the IMAP server and check the messages sequentionally, not in parallel. E.g. your script could synchronize itself with other parallel instances by a shared lock.

b) Use mail_log and a script on the mail server monitoring Dovecot logs and register delivered message ids and/or recipient. Then figure out a good way to probe, if one pariticular message has been seen. -or- make note of message id/recipient on the sender side, somehow push (ssh or something) that information on regular basis to the mail server, where a script verifies that data.

c) To verify the correct delivery: use some sort of checksumming.

Kind regards,

-- Steffen Kaiser
Hi , i think you like to do some monitoring ?

you may do it like this

if you have i.e xymon client on the mailserver and a central xymon monitoring server , write a server mailscript which sends some check mail i.e all 5 minutes from the monitoring server to the mailserver

then on the mailserver write some simple mailcheck monitoring client script if the mail arrived i.e in .new of the maildir mailbox i.e all 5 mins , go yellow if it failed for two times, go red if this fails for three times simply delete the test mail after check, use a special check mailbox which is not accessable from outbound ( postfix access table etc )

if you want additional imap/pop3 check with the test mail the check script could use i.e some imap/pop3 console mail reader perhaps mutt or pine etc for reading and deleting the test mail etc, you may integrate checksumming at some stage

i wouldnt deal with logs for this kind of monitoring, or you may do it addtional

i think there are nagios scripts allready ,which might work this way

Best Regards MfG Robert Schetterer

however all monitoring server have simple imap and pop3 checks at default so the will alarm you when there get no login slot at the mailserver

real time log monitoring is more complex, and mostly event related which might different from server to server

Best Regards MfG Robert Schetterer

-- [*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich

Harford, Ken

4:01 p.m.

The correct delivery.

The error message is generated by the script. This is a set of perl scripts from Nagios

check_email_delivery check_imap_receive check_smtp_send

I am going to see if I can delay the IMAP portion of the main script,

Thanks for the suggestion Steffen

Ken

On 1/22/13 4:04 AM, "Steffen Kaiser" <skdovecot@smail.inf.fh-brs.de> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 21 Jan 2013, Harford, Ken wrote:

...
       I am having an issue with dovecot IMAP reading and writing
to one specific mailbox. I have a check email delivery script that sends
What's the purpose of the verification:

The prompt delivery?

The delivery at all?

The correct delivery?

...
out a email and then logs into dovecot to check if the mailbox that the message has been delivered to. However, the servers that send these messages send them in bunches depending on how many postfix spools it is checking. So I could get 60 messages all being sent to 1 mailbox and then having to read that same mailbox to check if the message actually got delivered.

I am getting a lot of errors such as the following:

Could not connect to IMAP server imap failed: IMAP RECEIVE CRITICAL - Could not select INBOX

Is this an error logged by Dovecot?

...
Any ideas as to what I can tweak in dovecot to allow these messages to be verified.

a) Wait a bit before logging into the IMAP server and check the messages sequentionally, not in parallel. E.g. your script could synchronize itself with other parallel instances by a shared lock.

b) Use mail_log and a script on the mail server monitoring Dovecot logs and register delivered message ids and/or recipient. Then figure out a good way to probe, if one pariticular message has been seen.

-or- make note of message id/recipient on the sender side, somehow push (ssh or something) that information on regular basis to the mail server, where a script verifies that data.

c) To verify the correct delivery: use some sort of checksumming.

Kind regards,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUP5WEF3r2wJMiz2NAQIatAgAjwZR1VGnyURuncLRNeeY0dBaqWzAUcNK g4YoiXB6NLxP6Xr/QO6Pw+6Q3sECJ0CJdwImOz56FVFNuiOygUMQ1dCdZ+Tpy9Wu xeAdobrkOo2uzgRnUh7+WLeZ8k32gZZztCAQA90uTvB3MBox+7J/P8/mApp+8/tr JKyhUy8RJIYXH+yrw/ZkbjmOSEEkf/wTPnBQcMKw5/4odha4lu+yEKx766mm0YCe 3USL7eVqN0Q9FwUH+iCjnRJAdKRETP4CnfQpjVp7eiJX7HD0iNwhucDwf037Ev2N PG173M40pMllRyesjiV8gwOu8mAHu6V3QUiFJkYAKFOotUVO6xMrNQ== =SeaP -----END PGP SIGNATURE-----

Harford, Ken

4 Feb 4 Feb

4:45 p.m.

Okay, I set a wait period of 30 seconds and I am still having issues.

I am sending about 50 messages from 6 servers (300 messages) to the same IMAP mailbox at the same time and dovecot doesn't seem to be able handle that kind of message count.

I am using dovecot delivery instead of postfix delivery

Any other ideas?

Thanks

Ken

On 1/22/13 4:04 AM, "Steffen Kaiser" <skdovecot@smail.inf.fh-brs.de> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Mon, 21 Jan 2013, Harford, Ken wrote:

...
       I am having an issue with dovecot IMAP reading and writing
to one specific mailbox. I have a check email delivery script that sends
What's the purpose of the verification:

The prompt delivery?

The delivery at all?

The correct delivery?

...
out a email and then logs into dovecot to check if the mailbox that the message has been delivered to. However, the servers that send these messages send them in bunches depending on how many postfix spools it is checking. So I could get 60 messages all being sent to 1 mailbox and then having to read that same mailbox to check if the message actually got delivered.

I am getting a lot of errors such as the following:

Could not connect to IMAP server imap failed: IMAP RECEIVE CRITICAL - Could not select INBOX

Is this an error logged by Dovecot?

...
Any ideas as to what I can tweak in dovecot to allow these messages to be verified.

a) Wait a bit before logging into the IMAP server and check the messages sequentionally, not in parallel. E.g. your script could synchronize itself with other parallel instances by a shared lock.

b) Use mail_log and a script on the mail server monitoring Dovecot logs and register delivered message ids and/or recipient. Then figure out a good way to probe, if one pariticular message has been seen.

-or- make note of message id/recipient on the sender side, somehow push (ssh or something) that information on regular basis to the mail server, where a script verifies that data.

c) To verify the correct delivery: use some sort of checksumming.

Kind regards,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)

iQEVAwUBUP5WEF3r2wJMiz2NAQIatAgAjwZR1VGnyURuncLRNeeY0dBaqWzAUcNK g4YoiXB6NLxP6Xr/QO6Pw+6Q3sECJ0CJdwImOz56FVFNuiOygUMQ1dCdZ+Tpy9Wu xeAdobrkOo2uzgRnUh7+WLeZ8k32gZZztCAQA90uTvB3MBox+7J/P8/mApp+8/tr JKyhUy8RJIYXH+yrw/ZkbjmOSEEkf/wTPnBQcMKw5/4odha4lu+yEKx766mm0YCe 3USL7eVqN0Q9FwUH+iCjnRJAdKRETP4CnfQpjVp7eiJX7HD0iNwhucDwf037Ev2N PG173M40pMllRyesjiV8gwOu8mAHu6V3QUiFJkYAKFOotUVO6xMrNQ== =SeaP -----END PGP SIGNATURE-----

4493

Age (days ago)

4507

Last active (days ago)

List overview

5 comments

3 participants

participants (3)

Harford, Ken
Robert Schetterer
Steffen Kaiser