Hello,
I've recently converted from using procmail as an lda to using lmtp+sieve (and it's overall great).
My setup is currently doing post-queue spamassassin to pickup user rules (and to be substantially similar to how it worked before). It mostly works, but I get some mail forwarded from another host that also runs spamassassin, so the mail comes with X-Spam headers already set.
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
My before.sieve looks like this:
require ["vnd.dovecot.filter","editheader"];
rule:[spamfilter]
if size :under 10485760
{
deleteheader :matches "X-Spam-Flag" "*";
deleteheader :matches "X-Spam-Status" "*";
deleteheader :matches "X-Spam-Level" "*";
deleteheader :matches "X-Spam-Checker-Version" "*";
filter "spamc" ["-s","10485760","-U","/var/run/spamd.sock"];
}
Trace file says:
## Started executing script 'before'6: size :under test
6: comparing message size 4197
6: with lower limit 10485760
6: jump if result is false
6: not jumping
11: deleteheader command
11: deleting matching occurrences of header X-Spam-Flag' 11: header X-Spam-Flag' not found
12: deleteheader command
12: deleting matching occurrences of header X-Spam-Status' 12: starting :matches' match with i;ascii-casemap' comparator: 12: matching value No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_..
.'
12: with key *' => 1 12: deleting header with value No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALI
D_AU,DKIM_VALID_EF,FREEMAIL_FROM,HTML_MESSAGE, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham au
tolearn_force=no version=3.4.2'
12: finishing match with result: matched
13: deleteheader command
13: deleting matching occurrences of header X-Spam-Level' 13: header X-Spam-Level' not found
14: deleteheader command
14: deleting matching occurrences of header X-Spam-Checker-Version' 14: starting :matches' match with i;ascii-casemap' comparator: 14: matching value SpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org'
14: with key *' => 1 14: deleting header with value SpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org'
14: finishing match with result: matched
16: filter action
16: execute program `spamc'
16: executed program successfully
16: changed message
## Finished executing script 'before'
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
So, two questions:
- What am I missing?
- Is there a way to match any X-Spam-* header? (when I tried deleteheader :matches "X-Spam*" it didn't match anything).
Thanks.
-- Daniel J. Luke
Hi!
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.
My addheader case was solved by adding an explicit keep or a fileinto.
I hope that will also help you, Vincent
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.
I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
Daniel J. Luke
On Jun 11, 2021, at 9:54 AM, Daniel J. Luke <dluke@geeklair.net> wrote:
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
While that didn't work, adding a keep to the end of the user rules works - and even better, I don't need the deleteheaders (as it preserves the headers as set by the spamc filter).
This behavior is very surprising to me (I wouldn't expect rules ending with an explicit keep to be any different from rules ending with an implicit one), but at least it's working now - and hopefully being in the list archive will save someone else some time in the future.
-- Daniel J. Luke
On 11 Jun 2021, at 07:54, Daniel J. Luke <dluke@geeklair.net> wrote:
On Jun 11, 2021, at 3:51 AM, Vincent Brillault <vincent.brillault@cern.ch> wrote:
It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
I had a similar issue recently (with addheader, not deleteheader) and I was pointed to the RFC: https://datatracker.ietf.org/doc/html/rfc5293#section-5
The deleteheader action does not affect Sieve's implicit keep.```I haven't gone over the RFCs with a fine toothed comb, but having implicit and explicit keep with different behaviors is surprising.
Agreed. And I think we all hit this.
My addheader case was solved by adding an explicit keep or a fileinto.
I added a keep; to then end of my before.sieve and it doesn't change the behavior (with or without the deleteheaders I always see the other hosts X-Spam headers in the delivered mail).
Are you sure the message you are seeing is being processed by your before.sieve? Os there any evidence at all in the mail that it was hit sieve?
For example, one of my sieve_before
cat /usr/lib/dovecot/sieve/filespam.sieve
require ["fileinto", "imap4flags"];
if header :contains "X-spam-flag" "YES" { fileinto :flags "\Seen" "Junk"; }
I know that sieve fires because messages marked as spam are tied into the user's Junk mail.
You know that the sieve_before scripts are run AS THE USER, not as dovecot or root, right? I mention this because it was something that caught me out.
-- Growing up leads to growing old, and then to dying/And dying to me don't sound like all that much fun.
On 11/06/2021 04:25, Daniel J. Luke wrote:
Hello,
I've recently converted from using procmail as an lda to using lmtp+sieve (and it's overall great).
My setup is currently doing post-queue spamassassin to pickup user rules (and to be substantially similar to how it worked before). It mostly works, but I get some mail forwarded from another host that also runs spamassassin, so the mail comes with X-Spam headers already set.
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
I cannot reproduce this with master. Keep and implicit keep act the same. My filter program successfully changes the message and the deleteheader commands properly drop the indicated headers.
What version is this? What is your configuration (output from dovecot -n)
Regards,
Stephan.
My before.sieve looks like this:
require ["vnd.dovecot.filter","editheader"];
rule:[spamfilter]
if size :under 10485760 { deleteheader :matches "X-Spam-Flag" "*"; deleteheader :matches "X-Spam-Status" "*"; deleteheader :matches "X-Spam-Level" "*"; deleteheader :matches "X-Spam-Checker-Version" "*"; filter "spamc" ["-s","10485760","-U","/var/run/spamd.sock"]; }
Trace file says:
## Started executing script 'before' 6: size :under test 6: comparing message size 4197 6: with lower limit 10485760 6: jump if result is false 6: not jumping11: deleteheader command 11: deleting matching occurrences of header
X-Spam-Flag' 11: headerX-Spam-Flag' not found 12: deleteheader command 12: deleting matching occurrences of headerX-Spam-Status' 12: starting:matches' match withi;ascii-casemap' comparator: 12: matching valueNo, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_.. .' 12: with key*' => 1 12: deleting header with valueNo, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALI D_AU,DKIM_VALID_EF,FREEMAIL_FROM,HTML_MESSAGE, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham au tolearn_force=no version=3.4.2' 12: finishing match with result: matched 13: deleteheader command 13: deleting matching occurrences of headerX-Spam-Level' 13: headerX-Spam-Level' not found 14: deleteheader command 14: deleting matching occurrences of headerX-Spam-Checker-Version' 14: starting:matches' match withi;ascii-casemap' comparator: 14: matching valueSpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org' 14: with key*' => 1 14: deleting header with valueSpamAssassin 3.4.2 (2018-09-13) on braeburn.macports.org' 14: finishing match with result: matched 16: filter action 16: execute program `spamc' 16: executed program successfully 16: changed message ## Finished executing script 'before'It then falls through my personal sieve filter and hits the implicit keep at the end and ends up in my inbox - with the original headers.
So, two questions:
- What am I missing?
- Is there a way to match any X-Spam-* header? (when I tried deleteheader :matches "X-Spam*" it didn't match anything).
Thanks.
On Jun 12, 2021, at 5:16 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
I can see that my sieve_before script successfully sends the message off to spamc/spamd - however the message ends up in my mailbox with only the pre-existing X-Spam headers (I had expected them to be replaced or to see duplicates). I thought I'd try to just strip them from incoming mail so I used deleteheader, I got a trace saying it matches and deletes the headers, then sends the mail off to spamc/spamd - but the message that ends up in my mailbox has the headers that were presumably deleted.
I cannot reproduce this with master. Keep and implicit keep act the same. My filter program successfully changes the message and the deleteheader commands properly drop the indicated headers.
What version is this? What is your configuration (output from
dovecot -n)
My before script sends to spamc or does deleteheader (as noted) but user script does:
rule:[mailman lists with mailboxes]
if allof( header :matches "List-Id" "*<*.*", mailboxexists "${2}" ) { fileinto "${2}"; stop; }
rule:[discard duplicates]
if duplicate { discard; }
keep;
I don't see header problems in mail that ends up ina fileinto'd mailbox, just my INBOX - I suspect it's interaction with the 'duplicate' implementation (which the RFC says operates on the original un-modified message).
dovecot -n output below -
2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
Pigeonhole version 0.5.14 (1b5c82b2)
OS: Darwin 20.5.0 x86_64
Hostname: vroomfondel.geeklair.net
auth_username_format = %Ln auth_verbose = yes default_internal_group = mail default_internal_user = _dovecot default_login_user = _dovenull first_valid_gid = 500 first_valid_uid = 501 last_valid_gid = 599 last_valid_uid = 599 login_greeting = geeklair.net mail ready. mail_location = mbox:~/Mail/:INBOX=~/.mbox:INDEX=~/.dovecot-indexes mail_plugins = fts fts_lucene zlib managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date index ihave duplicate mime foreverypart extracttext namespace { inbox = yes location = mailbox Drafts { auto = no special_use = \Drafts } mailbox Sent { auto = subscribe special_use = \Sent } mailbox Trash { auto = no special_use = \Trash } mailbox spam { auto = create special_use = \Junk } prefix = separator = / } namespace archive { inbox = no list = no location = mbox:~/MailArchive/ prefix = "#Archive/" separator = / } passdb { args = dovecot driver = pam } plugin { fts = lucene fts_autoindex = yes fts_autoindex_exclude = "#Archive/*" fts_lucene = whitespace_chars=@. recipient_delimiter = + sieve = file:~/.sieve;active=~/.dovecot.sieve sieve_before = /usr/local/etc/dovecot/sieve/before.sieve sieve_default = /usr/local/etc/dovecot/sieve/default.sieve sieve_default_name = file_spam sieve_filter_bin_dir = /usr/local/lib/dovecot/sieve-filter sieve_filter_exec_timeout = 720s sieve_global_extensions = +vnd.dovecot.filter +editheader sieve_plugins = sieve_extprograms } protocols = imap pop3 lmtp sieve service auth { unix_listener /usr/local/var/spool/postfix/private/auth { group = _postfix mode = 0660 user = _postfix } user = root } service imap-login { inet_listener imaps { port = 993 ssl = yes } process_min_avail = 1 service_count = 0 } service imap { vsz_limit = 512 M } service lmtp { unix_listener /usr/local/var/spool/postfix/private/dovecot-lmtp { group = postfix mode = 0600 user = postfix } } service managesieve-login { inet_listener sieve { port = 4190 } process_min_avail = 0 service_count = 0 vsz_limit = 64 M } service managesieve { process_limit = 12 } service pop3-login { inet_listener pop3 { port = 0 } inet_listener pop3s { port = 995 ssl = yes } process_min_avail = 1 service_count = 0 } service pop3 { vsz_limit = 512 M } ssl_cert = </opt/local/etc/letsencrypt/live/geeklair.net/fullchain.pem ssl_cipher_list = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ssl_dh = # hidden, use -P to show it ssl_key = # hidden, use -P to show it ssl_prefer_server_ciphers = yes syslog_facility = local1 userdb { driver = passwd } verbose_proctitle = yes version_ignore = yes protocol imap { mail_max_userip_connections = 32 mail_plugins = fts fts_lucene zlib imap_zlib } protocol pop3 { pop3_uidl_format = %08Xu%08Xv } protocol lda { mail_plugins = fts fts_lucene zlib sieve postmaster_address = postmaster@geeklair.net } protocol lmtp { mail_plugins = fts fts_lucene zlib sieve postmaster_address = postmaster@geeklair.net }
-- Daniel J. Luke
participants (4)
-
＠lbutlr
-
Daniel J. Luke
-
Stephan Bosch
-
Vincent Brillault