dovecot-2.1-pigeonhole: lib-sieve: vnd.dovecot.duplicate extensi...
pigeonhole at rename-it.nl
pigeonhole at rename-it.nl
Tue Oct 16 22:33:25 EEST 2012
details: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/12a4e11ecd4c
changeset: 1660:12a4e11ecd4c
user: Stephan Bosch <stephan at rename-it.nl>
date: Tue Oct 16 21:33:15 2012 +0200
description:
lib-sieve: vnd.dovecot.duplicate extension: Added new features to the duplicate test.
It is now possible to track duplicates based on arbitrary headers or even arbitrary string values using the new :header and :value arguments respectively.
The experation time can be configured using the new :seconds argument.
This change is backwards compatible as long as the name argument wasn't used. This is now a :handle <handle> argument.
diffstat:
doc/rfc/spec-bosch-sieve-duplicate.txt | 284 +++++++--
doc/rfc/xml/reference.IMAP4FLAGS.xml | 15 +
doc/rfc/xml/reference.MAILBOX.xml | 15 +
doc/rfc/xml/reference.VACATION.xml | 17 +
doc/rfc/xml/spec-bosch-sieve-duplicate.xml | 164 ++++-
src/lib-sieve/plugins/vnd.dovecot/duplicate/ext-duplicate-common.c | 137 +++-
src/lib-sieve/plugins/vnd.dovecot/duplicate/ext-duplicate-common.h | 11 +-
src/lib-sieve/plugins/vnd.dovecot/duplicate/ext-duplicate.c | 3 +-
src/lib-sieve/plugins/vnd.dovecot/duplicate/tst-duplicate.c | 273 ++++++++-
tests/extensions/vnd.dovecot/duplicate/errors.svtest | 2 +-
tests/extensions/vnd.dovecot/duplicate/errors/syntax.sieve | 5 +-
tests/extensions/vnd.dovecot/duplicate/execute.svtest | 10 +-
12 files changed, 747 insertions(+), 189 deletions(-)
diffs (truncated from 1305 to 300 lines):
diff -r 2b5ff3818a9f -r 12a4e11ecd4c doc/rfc/spec-bosch-sieve-duplicate.txt
--- a/doc/rfc/spec-bosch-sieve-duplicate.txt Sat Oct 13 10:30:57 2012 +0200
+++ b/doc/rfc/spec-bosch-sieve-duplicate.txt Tue Oct 16 21:33:15 2012 +0200
@@ -2,7 +2,7 @@
Pigeonhole Project S. Bosch
- February 25, 2012
+ October 16, 2012
Sieve Email Filtering: Detecting Duplicate Deliveries
@@ -10,23 +10,65 @@
Abstract
This document defines a new vendor-defined test command "duplicate"
- for the "Sieve" email filtering language that tests whether an e-mail
- message is a duplicate, i.e. whether it was seen before by the
- delivery agent. Users can use this new test to remove duplicate
- deliveries commonly caused by mailing list subscriptions or mail
- account aliases.
+ for the "Sieve" email filtering language. It can be used to test
+ whether a particular string value is a duplicate, i.e. whether it was
+ seen before by the delivery agent that is executing the Sieve script.
+ The main application for this new test is detecting duplicate message
+ deliveries commonly caused by mailing list subscriptions or
+ redirected mail addresses.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Bosch [Page 1]
+
+ Sieve: Detecting Duplicate Deliveries October 2012
Table of Contents
- 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2
- 2. Conventions Used in This Document . . . . . . . . . . . . . . . 2
- 3. Test "duplicate" . . . . . . . . . . . . . . . . . . . . . . . 2
- 4. Sieve Capability Strings . . . . . . . . . . . . . . . . . . . 3
- 5. Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
- 6. Security Considerations . . . . . . . . . . . . . . . . . . . . 3
- 7. Normative References . . . . . . . . . . . . . . . . . . . . . 3
- Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 3
+ 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3
+ 2. Conventions Used in This Document . . . . . . . . . . . . . . . 3
+ 3. Test "duplicate" . . . . . . . . . . . . . . . . . . . . . . . 4
+ 4. Sieve Capability Strings . . . . . . . . . . . . . . . . . . . 5
+ 5. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
+ 6. Security Considerations . . . . . . . . . . . . . . . . . . . . 6
+ 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 6
+ 7.1. Normative References . . . . . . . . . . . . . . . . . . . 6
+ 7.2. Informative References . . . . . . . . . . . . . . . . . . 6
+ Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 7
@@ -52,26 +94,52 @@
-Bosch [Page 1]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Bosch [Page 2]
- Sieve: Detecting Duplicate Deliveries February 2012
+ Sieve: Detecting Duplicate Deliveries October 2012
1. Introduction
This is an extension to the Sieve filtering language defined by RFC
- 5228 [SIEVE]. It adds a test to determine whether a message was seen
- before by the delivery agent based on the Message-ID header.
+ 5228 [SIEVE]. It adds a test to determine whether a certain string
+ value was seen before by the delivery agent in an earlier execution
+ of the Sieve script. This can be used to detect and handle duplicate
+ message deliveries.
Duplicate deliveries are a common side-effect of being subscribed to
- a mailing list. If a member of the list decides to reply to both the
- user and the mailing list itself, the user will get a copy of the
- message directly and through mailing list. In another scenario, the
- user has several aliases for his mail account and one of his contacts
- sends the message to multiple addresses that eventually map to the
- same account. Using the vnd.dovecot.duplicate extension, users have
- the means to detect such duplicates and deal with these
- appropriately, e.g. by discarding them.
+ a mailing list. For example, if a member of the list decides to
+ reply to both the user and the mailing list itself, the user will get
+ a copy of the message directly and through mailing list. Also, if
+ someone cross-posts over several mailing lists to which the user is
+ subscribed, the user will receive a copy from each of those lists.
+ In another scenario, the user has several redirected mail addresses
+ all pointing to his main mail account. If one of the user's contacts
+ sends the message to more than one of those addresses, the user will
+ receive more than a single copy. Using the "vnd.dovecot.duplicate"
+ extension, users have the means to detect and handle such duplicates,
+ e.g. by discarding them or putting them in a special folder.
+
+ Duplicate messages are normally detected using the Message-ID header
+ field, which is required to be unique for each message. However, the
+ "duplicate" test is flexible enough to use different (weaker)
+ criteria for defining what makes a message a duplicate, for example
+ based on the subject line. Also, other applications of this new test
+ command are possible, as long as the tracked value is a string.
This extension is specific to the Pigeonhole Sieve implementation for
the Dovecot Secure IMAP server. It will therefore most likely not be
@@ -89,31 +157,79 @@
arguments syntax.
+
+
+
+
+
+
+
+Bosch [Page 3]
+
+ Sieve: Detecting Duplicate Deliveries October 2012
+
+
3. Test "duplicate"
- Usage: "duplicate" [<name: string>]
+ Usage: "duplicate" [":seconds" <timeout: number>]
+ [":header" <header-name: string> /
+ ":value" <value: string>]
+ [":handle" <handle: string>]
- The "duplicate" test keeps track of which Message-ID values were seen
- before by this test in an earlier delivery operation. It evaluates
- to "true" when the Message-ID header of the current message was seen
- before. If it is not known, the test evaluates to "false" and the
- Message-ID is added to a persistent internal tracking list.
- Implementations SHOULD limit the number of messages that are tracked
- and SHOULD let Message-ID entries expire after some short period of
- time.
+ The "duplicate" test keeps track of which values were seen before by
+ this test in an earlier execution of this Sieve script. In its basic
+ form, the tested value is the content of the Message-ID header of the
+ message. This way, this test can be used to detect duplicate
+ deliveries of the same message. It can also detect duplicate
+ deliveries based on other message header fields if requested and it
+ can even use a user-provided string value, e.g. as composed from text
+ extracted from the message using the "variables" [VARIABLES]
+ extension.
- Using the "name" argument, the duplicate test can be employed for
- multiple independent purposes. Only when the Message-ID was seen
- before in an earlier script execution by a duplicate test with the
+ The "duplicate" test evaluates to "true" when the provided value was
+ seen before. If the value is not known, the test evaluates to
+ "false" and the value is added to an internal value tracking list.
+ Implementations SHOULD limit the number of values (and thereby
+ messages) that are tracked. Also, implementations SHOULD let entries
+ in the value tracking list expire after a short period of time.
+ The user can explicitly control the length this expiration time by
+ means of the ":seconds" argument. If the ":seconds" argument is
+ omitted, an appropriate default MUST be used. Sites SHOULD impose a
+ maximum limit on the expiration time. If that limit is exceeded, the
+ maximum value MUST silently be substituted; exceeding the limit MUST
+ NOT produce an error.
+ By default the tracked value is the content of the message's
+ Message-ID header field. For more advanced purposes, the content of
+ another header can be chosen for tracking by specifying the ":header"
+ argument. The tracked string value can also be specified explicitly
+ using the ":value" argument. The ":header" and ":value" arguments
+ are mutually exclusive and specifying both for a single "duplicate"
+ test command MUST trigger an error at compile time. If the value is
+ extracted from a header, i.e. when the ":value" argument is not used,
+ leading and trailing whitespace (see Section 2.2 of RFC 5228 [SIEVE])
+ MUST first be trimmed from the value before executing the test.
-Bosch [Page 2]
+ Using the ":handle" argument, the duplicate test can be employed for
+ multiple independent purposes. Only when the tracked value was seen
+ before in an earlier script execution by a "duplicate" test with the
+ same ":handle" argument, it is recognized as a duplicate.
+
+ NOTE: The necessary mechanism to track duplicate messages is very
+
+
+
+Bosch [Page 4]
- Sieve: Detecting Duplicate Deliveries February 2012
+ Sieve: Detecting Duplicate Deliveries October 2012
- same "name" argument, it is recognized as a duplicate.
+ similar to the mechanism that is needed for tracking duplicate
+ responses for the "vacation" [VACATION] action. One way to implement
+ the necessary mechanism for the "duplicate" test is therefore to
+ store a hash of the tracked value and, if provided, the ":handle"
+ argument.
4. Sieve Capability Strings
@@ -122,29 +238,71 @@
advertise the capability string "vnd.dovecot.duplicate".
-5. Example
+5. Examples
- In this example, duplicate deliveries are stored in a special folder
- contained in the user's Trash folder. If the folder does not exist,
- it is created. This way, the user has a chance to recover messages
- when necessary. Messages that are not recognized as duplicates are
- stored in the user's inbox as normal.
+ In the following basic example, message duplicates are detected by
+ tracking the Message-ID header. Duplicate deliveries are stored in a
+ special folder contained in the user's Trash folder. If the folder
+ does not exist, it is created automatically using the "mailbox"
+ [MAILBOX] extension. This way, the user has a chance to recover
+ messages when necessary. Messages that are not recognized as
+ duplicates are stored in the user's inbox as normal.
require ["vnd.dovecot.duplicate", "fileinto", "mailbox"];
if duplicate {
- fileinto :create "Trash/Duplicate";
+ fileinto :create "Trash/Duplicate";
}
+ The next example shows a more complex use of the "duplicate" test.
+ The user gets network alerts from a set of remote automated
+ monitoring systems. Multiple notifications can be received about the
+ same event from different monitoring systems. The Message-ID of
+ these messages is different, because these are all distinct messages
+ from different senders. To avoid being notified multiple times about
+ the same event the user writes the following script:
+
+ require ["vnd.dovecot.duplicate", "variables", "imap4flags",
+ "fileinto"];
+
+ if header :matches "subject" "ALERT: *" {
+ if duplicate :seconds 60 :value "${1}" {
+ setflag "\\seen";
+ }
+ fileinto "Alerts";
+ }
+
+ The subjects of the notification message are structured with a
+
+
+
More information about the dovecot-cvs
mailing list