[Dovecot] Very High Load on Dovecot 2 and Errors in mail.err.

Urban Loesch bind at enas.net
Wed May 16 09:59:04 EEST 2012


Hi,

I have a DELL PE R610 (32GB RAM 2x Six Core CPU and about 1,4 TB RAID 10)
running with 20.000 Mailaccounts behind 2 Dovecot IMAP/POP3 Proxies on a Debian Lenny.

The Server was running about 1 year without any problems. 15Min Load was between 0,5 and max 8.
No high IOWAIT. CPU Idletime about 98%.

But since yesterday morning the Systemload on the Server has been increased over 500. I Think this is
very high. The strange thing: there was no IOWAIT and the CPU Idle time was allways the same on about 98%.

The total amount of IMAP Sessions is about 300 - 600.

Current vmstat and iostat:

#  vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
  1  0      0 27040576 635460 3495600    0    0    14    15    2   31  0  0 99  0
  0  0      0 27040320 635468 3496064    0    0   804   455 1383 1281  0  0 98  1
  0  0      0 27047016 634964 3489312    0    0   216   156 1841 1292  1  0 98  1
  0  0      0 27047140 635028 3489012    0    0   240   619 1629 1658  0  0 96  3
  0  0      0 27047264 635120 3489172    0    0    92     0 1069  881  0  0 100  0
  0  0      0 27047388 635120 3489256    0    0     0    46 1404 1265  0  0 100  0
  0  0      0 27047512 635136 3489312    0    0   128   471 1539 1354  0  0 99  1
  0  0      0 27047388 635156 3489384    0    0    12   360 1108  952  0  0 99  0
  0  0      0 27047516 635160 3489408    0    0   104    12  893  677  0  0 99  0
^C

#  iostat -k
Linux 3.0.28-vs2.3.2.3-rol-em64t (mailstore4) 	16.05.2012 	_x86_64_	(24 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0,08    0,00    0,09    0,34    0,00   99,49

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              67,35       636,28      1080,48   31337690   53215361
dm-0             73,53       591,55       893,93   29134837   44027496
dm-1             15,06        42,40       181,79    2088493    8953661
drbd1            51,21       316,06       277,94   15566325   13689004
drbd0             9,22        23,50        80,38    1157641    3958796

Current Load:
08:55:44 up 14:04,  3 users,  load average: 19,64, 14,47, 10,49
  Load is increasing.


The only strange thing I can see is this:
# ps -ostat,pid,time,wchan='WCHAN-xxxxxxxxxxxxxxxxxxxx',cmd ax  |grep D
STAT   PID     TIME WCHAN-xxxxxxxxxxxxxxxxxxxx CMD
D    18713 00:00:00 synchronize_srcu           dovecot/imap
D    18736 00:00:00 synchronize_srcu           dovecot/imap
D    18775 00:00:05 synchronize_srcu           dovecot/imap
D    20330 00:00:00 synchronize_srcu           dovecot/imap
D    20357 00:00:00 synchronize_srcu           dovecot/imap
D    20422 00:00:00 synchronize_srcu           dovecot/imap
D    20687 00:00:00 synchronize_srcu           dovecot/imap
S+   20913 00:00:00 pipe_wait                  grep D

There are many imap processes in D State. Amount is increasing.
I think they are delayed and are wating for some event.
I have no idea on which event they are waiting.

And many of this in "mail.err" Log:
May 16 08:52:11 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 17468 is ignoring idle SIGINT
May 16 08:52:24 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20307 is ignoring idle SIGINT
May 16 08:52:25 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20318 is ignoring idle SIGINT
May 16 08:52:26 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18964 is ignoring idle SIGINT
May 16 08:52:28 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 19244 is ignoring idle SIGINT
May 16 08:54:22 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 21177 is ignoring idle SIGINT
May 16 08:54:41 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20647 is ignoring idle SIGINT
May 16 08:55:10 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18836 is ignoring idle SIGINT
May 16 08:55:17 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 18857 is ignoring idle SIGINT
May 16 08:55:19 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 21176 is ignoring idle SIGINT
May 16 08:55:24 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20688 is ignoring idle SIGINT
May 16 08:55:25 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20973 is ignoring idle SIGINT
May 16 08:56:44 dcot-rolmail-1.rolmail.net dovecot: master: Error: service(imap): Process 20326 is ignoring idle SIGINT


Have you any Idea ho I can troubleshoot this problem?



Some more technical information about the system:
# uname -a
Linux mailstore4 3.0.28-vs2.3.2.3-rol-em64t #1 SMP Thu May 3 09:31:08 CEST 2012 x86_64 GNU/Linux

Dovecot Packages on Backend: installed:
#  dpkg -l |grep dovecot
ii  debian-dovecot-auto-keyring     2010.01.30                   GnuPG archive keys of the Automatic Dovecot Debian repository
ii  dovecot-common                  2:2.0.13-0~auto+54           secure mail server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-imapd                   2:2.0.13-0~auto+54           secure IMAP server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-lmtpd                   2:2.0.13-0~auto+54           secure LMTP server for Dovecot
ii  dovecot-managesieved            2:2.0.13-0~auto+54           secure ManageSieve server for Dovecot
ii  dovecot-mysql                   2:2.0.13-0~auto+54           MySQL support for Dovecot
ii  dovecot-pop3d                   2:2.0.13-0~auto+54           secure POP3 server that supports mbox, maildir, dbox and mdbox mailboxes
ii  dovecot-sieve                   2:2.0.13-0~auto+54           sieve filters support for Dovecot

Dovecot Packages on both Proxies installed:
ii  debian-dovecot-auto-keyring     2010.01.30               GnuPG archive keys of the Automatic Dovecot
ii  dovecot-common                  2:2.0.17-0~auto+4        Transitional package for dovecot
ii  dovecot-core                    2:2.0.17-0~auto+4        secure mail server that supports mbox, maild
ii  dovecot-gssapi                  2:2.0.17-0~auto+4        GSSAPI authentication support for Dovecot
ii  dovecot-imapd                   2:2.0.17-0~auto+4        secure IMAP server that supports mbox, maild
ii  dovecot-managesieved            2:2.0.17-0~auto+4        secure ManageSieve server for Dovecot
ii  dovecot-mysql                   2:2.0.17-0~auto+4        MySQL support for Dovecot
ii  dovecot-pgsql                   2:2.0.17-0~auto+4        PostgreSQL support for Dovecot
ii  dovecot-pop3d                   2:2.0.17-0~auto+4        secure POP3 server that supports mbox, maild
ii  dovecot-sieve                   2:2.0.17-0~auto+4        sieve filters support for Dovecot
ii  dovecot-sqlite                  2:2.0.17-0~auto+4        SQLite support for Dovecot


# doveconf -n
# 2.0.13 (02d97fb66047): /etc/dovecot/dovecot.conf
# OS: Linux 3.0.28-vs2.3.2.3-rol-em64t x86_64 Debian 6.0.2 ext4
auth_cache_negative_ttl = 0
auth_cache_size = 40 M
auth_cache_ttl = 12 hours
auth_mechanisms = plain login
auth_username_format = %Lu
auth_verbose = yes
deliver_log_format = msgid=%m: %$ %p %w
disable_plaintext_auth = no
login_trusted_networks = 195.254.252.207 195.254.252.208 2a02:2e8:0:2:0:143:0:1 2a02:2e8:0:2:0:143:0:2
mail_gid = mailstore
mail_location = mdbox:/home/vmail/%d/%n:INDEX=/home/dovecotindex/%d/%n
mail_plugins = " quota zlib"
mail_uid = mailstore
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
copy include variables body enotify environment mailbox date ihave imapflags notify
mdbox_rotate_size = 5 M
passdb {
   args = /etc/dovecot/dovecot-sql-account.conf
   driver = sql
}
plugin {
   mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
   mail_log_fields = uid box msgid size from
   mail_log_group_events = no
   quota = dict:Storage used::file:%h/dovecot-quota
   sieve = ~/.dovecot.sieve
   sieve_dir = ~/sieve
   sieve_extensions = +notify +imapflags
   sieve_max_redirects = 10
   zlib_save = gz
   zlib_save_level = 5
}
protocols = imap pop3 lmtp sieve
service auth-worker {
   process_min_avail = 12
   service_count = 0
   vsz_limit = 512 M
}
service imap-login {
   inet_listener imap {
     port = 143
   }
   process_min_avail = 12
   service_count = 0
   vsz_limit = 256 M
}
service imap {
   process_min_avail = 12
   service_count = 0
   vsz_limit = 512 M
}
service lmtp {
   inet_listener lmtp {
     address = *
     port = 24
   }
   unix_listener /var/spool/postfix/private/dovecot-lmtp {
     group = postfix
     mode = 0666
     user = postfix
   }
   vsz_limit = 512 M
}
service pop3-login {
   inet_listener pop3 {
     port = 110
   }
   process_min_avail = 12
   service_count = 0
   vsz_limit = 256 M
}
service pop3 {
   service_count = 0
   vsz_limit = 256 M
}
ssl = no
ssl_cert = </etc/ssl/certs/dovecot.pem
ssl_key = </etc/ssl/private/dovecot.pem
userdb {
   args = /etc/dovecot/dovecot-sql-account.conf
   driver = sql
}
protocol lmtp {
   mail_plugins = " quota zlib sieve zlib"
}
protocol imap {
   imap_client_workarounds = tb-extra-mailbox-sep
   mail_max_userip_connections = 40
   mail_plugins = " quota zlib imap_quota imap_zlib"
}
protocol pop3 {
   mail_plugins = " quota zlib"
   pop3_logout_format = bytes_sent=%o top=%t/%p, retr=%r/%b, del=%d/%m, size=%s uidl_hash=%u
}

Thanks and regards
Urban Loesch


More information about the dovecot mailing list