replicator: Panic: data stack: Out of memory when allocating 268435496 bytes
Gerben Wierda
gerben.wierda at rna.nl
Fri Jan 6 22:20:32 UTC 2023
How problematic is it to have
default_vsz_limit = 0
in dovecot.conf? macOS+MacPorts had this as a requirement even.
Gerben
> On 6 Jan 2023, at 16:49, Paul Kudla <paul at scom.ca> wrote:
>
>
> i ran into this as well
>
> here is the full config for mine with replication
>
> # cat dovecot.conf
> # 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
> # OS: FreeBSD 12.1-RELEASE amd64
> # Hostname: mail18.scom.ca
>
> auth_debug = no
> auth_debug_passwords = no
>
> default_process_limit = 16384
>
> mail_debug = no
>
> #lock_method = dotlock
> #mail_max_lock_timeout = 300s
>
> #mbox_read_locks = dotlock
> #mbox_write_locks = dotlock
>
> mmap_disable = yes
> dotlock_use_excl = no
> mail_fsync = always
> mail_nfs_storage = no
> mail_nfs_index = no
>
> auth_mechanisms = plain login
> auth_verbose = yes
> base_dir = /data/dovecot/run/
> debug_log_path = syslog
> disable_plaintext_auth = no
> dsync_features = empty-header-workaround
>
> info_log_path = syslog
> login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
>
>
> mail_location = maildir:~/
>
> mail_plugins = " virtual notify replication fts fts_lucene "
> mail_prefetch_count = 20
>
> protocols = imap pop3 lmtp sieve
>
>
> protocol lmtp {
> mail_plugins = $mail_plugins sieve
> postmaster_address =
> }
>
> service lmtp {
> process_limit=1000
> vsz_limit = 512m
> client_limit=1
> unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
> group = postfix
> mode = 0600
> user = postfix
> }
> }
>
> protocol lda {
> mail_plugins = $mail_plugins sieve
> }
>
> service lda {
> process_limit=1000
> vsz_limit = 512m
> }
>
> service imap {
> process_limit=4096
> vsz_limit = 2g
> client_limit=1
> }
>
> service pop3 {
> process_limit=1000
> vsz_limit = 512m
> client_limit=1
> }
>
> namespace inbox {
> inbox = yes
> location =
> mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
> }
> mailbox Sent {
> auto = subscribe
> special_use = \Sent
> }
> mailbox Trash {
> auto = subscribe
> special_use = \Trash
> }
> prefix =
> separator = /
> }
>
> passdb {
> args = /usr/local/etc/dovecot/dovecot-pgsql.conf
> driver = sql
> }
>
> doveadm_port = 12345
> doveadm_password = secretxxxx
>
> service doveadm {
> process_limit = 0
> process_min_avail = 0
> idle_kill = 0
> client_limit = 1
> user = vmail
> inet_listener {
> port = 12345
> }
> }
>
> service config {
> unix_listener config {
> user = vmail
> }
> }
>
> dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
> #dsync_remote_cmd = doveadm sync -d -u%u
>
> replication_dsync_parameters = -d -N -l 300 -U
>
> plugin {
> mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
> mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
> push_notification_driver = dlog
>
> sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
> #sieve = ~/.dovecot.sieve
> sieve_duplicate_default_period = 1h
> sieve_duplicate_max_period = 1h
> sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
> sieve_global_dir = /usr/local/etc/dovecot/sieve
> sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve
>
>
> mail_replica = tcp:10.221.0.19:12345
> #mail_replica = remote:vmail at 10.221.0.19
> #replication_sync_timeout = 2
>
> fts = lucene
> fts_lucene = whitespace_chars=@.
> fts_autoindex = yes
> fts_languages = en
> }
>
> #sieve_extensions = vnd.dovecot.duplicate
>
> #sieve_plugins = vnd.dovecot.duplicate
>
> service anvil {
> process_limit = 1
> client_limit=5000
> vsz_limit = 512m
> unix_listener anvil {
> group = vmail
> mode = 0666
> }
> }
>
> service indexer-worker {
> vsz_limit = 2g
> }
>
>
>
> service auth {
> process_limit = 1
> client_limit=5000
> vsz_limit = 1g
>
> unix_listener auth-userdb {
> mode = 0660
> user = vmail
> group = vmail
> }
> unix_listener /var/spool/postfix/private/auth {
> mode = 0666
> }
>
> }
>
> service stats {
> process_limit = 1000
> vsz_limit = 1g
> unix_listener stats-reader {
> group = vmail
> mode = 0666
> }
> unix_listener stats-writer {
> group = vmail
> mode = 0666
> }
> }
> userdb {
> args = /usr/local/etc/dovecot/dovecot-pgsql.conf
> driver = sql
>
> }
>
> protocol imap {
> mail_max_userip_connections = 50
> mail_plugins = $mail_plugins notify replication
> }
>
> protocol pop3 {
> mail_max_userip_connections = 50
> mail_plugins = $mail_plugins notify replication
> }
>
> protocol imaps {
> mail_max_userip_connections = 25
> mail_plugins = $mail_plugins notify replication
> }
>
> protocol pop3s {
> mail_max_userip_connections = 25
> mail_plugins = $mail_plugins notify replication
> }
>
>
> service managesieve-login {
> process_limit = 1000
> vsz_limit = 1g
> inet_listener sieve {
> port = 4190
> }
> }
>
> verbose_proctitle = yes
>
> replication_max_conns = 100
>
> replication_full_sync_interval = 1d
>
> service replicator {
> client_limit = 0
> drop_priv_before_exec = no
> idle_kill = 4294967295s
> process_limit = 1
> process_min_avail = 0
> service_count = 0
> vsz_limit = 8g
> unix_listener replicator-doveadm {
> mode = 0600
> user = vmail
> }
> vsz_limit = 8192M
> }
>
>
> service aggregator {
> process_limit = 1000
> #vsz_limit = 1g
> fifo_listener replication-notify-fifo {
> user = vmail
> group = vmail
> mode = 0666
> }
>
> }
>
> service pop3-login {
> process_limit = 1000
> client_limit = 100
> vsz_limit = 512m
> }
>
>
> service imap-urlauth-login {
> process_limit = 1000
> client_limit = 1000
> vsz_limit = 1g
> }
>
>
> service imap-login {
> process_limit=1000
> client_limit = 1000
> vsz_limit = 1g
> }
>
>
> protocol sieve {
> managesieve_implementation_string = Dovecot Pigeonhole
> managesieve_max_line_length = 65536
> }
>
>
>
>
> #Addition ssl config
> !include sni.conf
>
> with sni cert support (examples)
>
> # cat sni.conf
> #sni.conf
> ssl = yes
> verbose_ssl = yes
> ssl_dh =</usr/local/etc/dovecot/dh-4096.pem
> ssl_prefer_server_ciphers = yes
> #ssl_min_protocol = TLSv1.2
>
> #Default *.scom.ca
> ssl_key =</usr/local/etc/dovecot/scom.pem
> ssl_cert =</usr/local/etc/dovecot/scom.pem
> ssl_ca =</usr/local/etc/dovecot/scom.pem
>
> local_name .scom.ca {
> ssl_key =</usr/local/etc/dovecot/scom.pem
> ssl_cert =</usr/local/etc/dovecot/scom.pem
> ssl_ca =</usr/local/etc/dovecot/scom.pem
>
> }
>
> local_name mail.clancyca.com {
> ssl_key =</usr/local/etc/dovecot/cert/mail.clancyca.com
> ssl_cert =</usr/local/etc/dovecot/cert/mail.clancyca.com
> ssl_ca =</usr/local/etc/dovecot/cert/mail.clancyca.com
> }
>
> local_name mail.paulkudla.net {
> ssl_key =</usr/local/etc/dovecot/cert/mail.paulkudla.net
> ssl_cert =</usr/local/etc/dovecot/cert/mail.paulkudla.net
> ssl_ca =</usr/local/etc/dovecot/cert/mail.paulkudla.net
> }
>
> local_name mail.ekst.ca {
> ssl_key =</usr/local/etc/dovecot/cert/mail.ekst.ca
> ssl_cert =</usr/local/etc/dovecot/cert/mail.ekst.ca
> ssl_ca =</usr/local/etc/dovecot/cert/mail.ekst.ca
> }
>
> local_name mail.hamletdevelopments.ca {
> ssl_key =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
> ssl_cert =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
> ssl_ca =</usr/local/etc/dovecot/cert/mail.hamletdevelopments.ca
> }
>
> pg sql support supporting replication
>
> # cat dovecot-pgsql.conf
> driver = pgsql
> connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
> default_pass_scheme = PLAIN
>
> password_query = SELECT username as user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
>
> user_query = SELECT home, uid, gid FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
>
> #iterate_query = SELECT user, password FROM email_users WHERE username = '%u' and password <> 'alias' and status = True and destination = '%u'
>
> iterate_query = SELECT "username" as user, domain FROM email_users WHERE status = True and alias_flag = False
>
>
>
>
>
> Happy Friday !!!
> Thanks - paul
>
> Paul Kudla
>
>
> Scom.ca Internet Services <http://www.scom.ca>
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
>
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> Email paul at scom.ca
>
> On 1/6/2023 5:32 AM, Gerben Wierda wrote:
>>> On 6 Jan 2023, at 08:53, Aki Tuomi <aki.tuomi at open-xchange.com <mailto:aki.tuomi at open-xchange.com>> wrote:
>>>
>>>
>>>
>>> On January 6, 2023 3:56:39 AM GMT+02:00, Gerben Wierda <gerben.wierda at rna.nl <mailto:gerben.wierda at rna.nl>> wrote:
>>>> One step further in my quest to create a replacement mail server.
>>>>
>>>> I now have my old mail server (2.3.19.1, macOS + MacPorts) and my new (2.3.20, Alpine Linux, Docker, apk package). When I turn on replication it works, but, after a while I see:
>>>>
>>>> Jan 06 00:50:31 replicator: Panic: data stack: Out of memory when allocating 268435496 bytes
>>>> Jan 06 00:50:32 replicator: Fatal: master: service(replicator): child 133 killed with signal 6 (core dumped)
>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Sync failure:
>>>> Jan 06 00:50:32 lmtp(pid 195 user sysbh): Warning: replication(sysbh): Remote sent invalid input: -
>>>>
>>>> I've removed synchronous operation for now (found a message on the net suggesting that) but is this known and what does it mean?
>>>>
>>>> Gerben Wierda (LinkedIn <https://www.linkedin.com/in/gerbenwierda <https://www.linkedin.com/in/gerbenwierda>>)
>>>> R&A IT Strategy <https://ea.rna.nl/ <https://ea.rna.nl/>> (main site)
>>>> Book: Chess and the Art of Enterprise Architecture <https://ea.rna.nl/the-book/ <https://ea.rna.nl/the-book/>>
>>>> Book: Mastering ArchiMate <https://ea.rna.nl/the-book-edition-iii/ <https://ea.rna.nl/the-book-edition-iii/>>
>>>>
>>>
>>> Dovecot default memory limit is 256M. You should probably set
>>>
>>> service replicator {
>>> vsz_limit = 2G
>>> }
>>>
>>> because replicator might have to use more memory, especially for larger indexes.
>>>
>>> Aki
>> That is a good tip as well.
>> I had followed this bit of experience from someone else: https://marc.info/?l=dovecot&m=164438199727640 <https://marc.info/?l=dovecot&m=164438199727640>, haven't seen any err message since. But that might be because they are in sync now and both sides are aware. Can I trigger full replication again so I can test?
>> Gerben
>> --
>> This message has been scanned for viruses and
>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
>> believed to be clean.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://dovecot.org/pipermail/dovecot/attachments/20230106/cba5fa7e/attachment-0001.htm>
More information about the dovecot
mailing list