[Dovecot] Antispam plugin custom behavior?

older
[Dovecot] quota-rewrite patch &...

Kyle Wheeler

11 Jun 2008 11 Jun '08

6:25 p.m.

Hello,

I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:

 1. If you have a false-negative, put the spam into the Spam.Report
 folder
 2. If you have a false-positive (which has all kinds of ugly
 spamassassin protective markup in it), put the message into the
 Spam.NotSpam folder

Currently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.

I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.

I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).

Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?

~Kyle

The optimist thinks this is the best of all possible worlds. The pessimist fears it is true. -- J. Robert Oppenheimer

Attachments:

attachment.sig (application/pgp-signature — 204 bytes)

Show replies by date

Juan Asensio Sánchez

11 Jun 11 Jun

6:51 p.m.

The anispam plugin does exactly what you need, and you could forget the cron script. If you use SpamAssassin, you could add a rule to Sieve to move the Spam messages when they arrives to the Spam folder. If a user moves a message from Spam folder to any other folder, then the message is considered a false possitive (when this move is detected you could run sa-learn inmediatly, without the cron script); the other way, when a user moves a message INTO the spam folder you could run again the sa-learn script, but with different parameters.

I am testing antispam plugin with dovecot 1.1rcX, but it looks there is a conflict with quota plugin. Anyway you can view the documentation i have done, but it is in spanish.

http://wiki.nutum.es/linux/samba/samba_ldap_mds/instalacion_y_configuracion_...

Regards.

2008/6/11 Kyle Wheeler <kyle-dovecot@memoryhole.net>:

...

Kyle Wheeler

7:24 p.m.

On Wednesday, June 11 at 05:51 PM, quoth Juan Asensio Sánchez:

...

Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior.

Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved.

~Kyle

The borrower is the slave of the lender. -- Proverbs 22:7

Hugo Monteiro

7:33 p.m.

Kyle Wheeler wrote:

...

No different behaviour for the end user. Your user could continue to drag&drop messages in/out of the designated Spam folder. The only different, which is not visible to the end user, is that the retrain of false positives is activated by pulling the messages out of the Spam folder, rather than having to specifically put it in a Ham folder. That said, they can continue to use the Ham folder as a placebo. ;)

Out of curiosity, why would you need to alter the message when moving it around?

Hugo Monteiro.

-- ci.fct.unl.pt:~# cat .signature

Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307

Centro de Informática Faculdade de Ciências e Tecnologia da Universidade Nova de Lisboa Quinta da Torre 2829-516 Caparica Portugal Telefone: +351 212948596 Fax: +351 212948548 www.ci.fct.unl.pt apoio@fct.unl.pt

ci.fct.unl.pt:~# _

Kyle Wheeler

7:47 p.m.

On Wednesday, June 11 at 05:33 PM, quoth Hugo Monteiro:

...

...
Well, for one thing, this is different behavior than what my users are used to, and I'd rather not have to re-explain how things work and deal with confusion about the difference in behavior.

Plus, unless I misunderstand the antispam plugin (quite possible), it doesn't *alter* the message when you remove it from the Spam folder --- because if it did, that could confuse IMAP clients that expect messages not to change when moved.

~Kyle

No different behaviour for the end user. Your user could continue to drag&drop messages in/out of the designated Spam folder.

You misunderstood my original email. Users have two folders: one for reporting spam that wasn't identified as spam (the "Spam.Report" folder), one for rescuing ham that was misidentified as spam (the "Spam.NotSpam" folder).

Thus, there is no "implicit" behavior - putting a message in one of those folders explicitly tells the system "learn this message" or "unlearn this message".

...

The only different, which is not visible to the end user, is that the retrain of false positives is activated by pulling the messages out of the Spam folder, rather than having to specifically put it in a Ham folder.

How is that not visible to the end-user?

...

That said, they can continue to use the Ham folder as a placebo. ;)

Heh. I see. But the user expects messages in the Ham folder to disappear and be automatically redelivered to their inbox.

...

Out of curiosity, why would you need to alter the message when moving it around?

Because my system uses SpamAssassin's "report_safe" feature (see http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html#it...). When a message is identified as spam, instead of modifying the original message, the user receives a report message with the original message attached as a text/plain attachment. This makes reading the original rather difficult (on purpose). Thus, one of the benefits of moving the message to the NotSpam folder is that the message is restored to its original form and redelivered.

~Kyle

Unthinking respect for authority is the greatest enemy of truth. -- Albert Einstein

Hugo Monteiro

7:01 p.m.

Kyle Wheeler wrote:

...

Hello,

I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
1. If you have a false-negative, put the spam into the Spam.Report
folder
2. If you have a false-positive (which has all kinds of ugly
spamassassin protective markup in it), put the message into the
Spam.NotSpam folder
Currently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.

I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.

I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).

Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?

~Kyle

Hello Kyle,

Have you tried the plugin using the mailtrain backend? Basically it will forward the message, as attachment, to spam/notspam addresses that you define. That includes the use of a %u variable expansion, if you choose to use retrain addresses like like spam-username@domain.com or something. I've been pretty happy with with it and it scales a lot better than piping the message into a retrain command, since the mail system itself will handle the load in a more intelligent way.

Regards,

Hugo Monteiro.

-- ci.fct.unl.pt:~# cat .signature

Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307

ci.fct.unl.pt:~# _

Kyle Wheeler

7:35 p.m.

On Wednesday, June 11 at 05:01 PM, quoth Hugo Monteiro:

...

Have you tried the plugin using the mailtrain backend?

The antispam plugin? No, I haven't... mostly because it looks like no matter which backend I use, I'd have to alter the user-visible interface to my training system (which I don't really want to do), and it still doesn't handle the "altered message" problem.

...

Hmmm, load is something I hadn't thought about... (the system I'm working with at the moment has plenty of capacity to spare). That's a good point.

However, one of the goals here is to make it so that if a user identifies a message that has been mistakenly tagged as spam (and "sanitized" by SpamAssassin, e.g. via the "report_safe" setting), they can get the message corrected (and back to its original form) immediately. As it is, they have to put it into the NotSpam folder and wait a couple minutes for the message to reappear in the INBOX (because the cron job only runs every so often).

~Kyle

No, I don't know that Atheists should be considered as citizens, nor should they be considered patriots. This is one nation under God. -- George H. W. Bush, August 27, 1987

Hugo Monteiro

7:41 p.m.

Kyle Wheeler wrote:

...

No need to reply to the other message regarding the "why". ;)

Has for what you're saying .. maybe it could be possible to extend the plugin to remove/add headers.. Though i have no idea on the impact that would have on IMAP clients...

Regards,

Hugo Monteiro.

-- ci.fct.unl.pt:~# cat .signature

Hugo Monteiro Email : hugo.monteiro@fct.unl.pt Telefone : +351 212948300 Ext.15307

ci.fct.unl.pt:~# _

Daniel L. Miller

9:06 p.m.

Kyle Wheeler wrote:

...

Couple possibilities:

Have an SMTP server dedicated to spam re-processing. This dedicated function SMTP server could use your existing scripts to strip the unwanted headers and then pass it back to your standard server. This would be trivial with Postfix, either using domain mapping or dedicated IP/port listeners.
If you're not using an SMTP server with Postfix's flexibility, use something like smtpprox to add it.

Bottom line is use a tool that works within the IMAP server (the anti-spam plugin) to detect mail changes and then pass the message on to a separate process for treatment.

Daniel

Andre Rodier

8:34 p.m.

On Wed, 2008-06-11 at 10:25 -0500, Kyle Wheeler wrote:

...

Hello,

I currently have a setup on my system with what I call "magic folders" to enable spam filter training. Here's how it works:
 1. If you have a false-negative, put the spam into the Spam.Report
 folder
 2. If you have a false-positive (which has all kinds of ugly
 spamassassin protective markup in it), put the message into the
 Spam.NotSpam folder
Currently what happens is that a cron job comes along every five minutes and processes the messages in those folders. In the case of the NotSpam folder, it strips the message of the spamassassin markup,
retrains the bayesian net, and redelivers the message (e.g. via deliver). In the case of the Report folder, the message is used to train the bayesian net (among other things) and then deleted.

I'd love to be able to trigger these actions when the mail is moved, rather than have a cron job inspecting the mailboxes.

I looked into the antispam plugin (http://johannes.sipsolutions.net/Projects/dovecot-antispam), which seems nice but doesn't appear sufficiently generic for my needs. What would really work is if I could get it set up such that putting a message into either of those directories is turned into piping the message to a script of my choosing (a different one for each folder).

Does anyone know a good way of getting my own custom behavior in here, or is my cronjob setup probably the best way?

~Kyle

Hello Kyle,

As a temporary solution, and if your linux box as iNotify support, I suggest you use incron. incron is an "inotify cron" system. It works like the regular cron but is driven by filesystem events instead of time events.

Andre Rodier r e d 2 - Service Driven Development 34-35 Eastcastle Street, London W1W 8DW www.red2.co.uk | andre.rodier@red2.co.uk

(+44) 0870 471 8492 direct (+44) 0751 124 4961 mobile

Kyle Wheeler

12 Jun 12 Jun

6:09 p.m.

On Wednesday, June 11 at 06:34 PM, quoth Andre Rodier:

...

Interesting idea... thanks!

~Kyle

It's amazing how much "mature wisdom" resembles being too tired. -- Robert A. Heinlein

Johannes Berg

12:43 a.m.

Kyle,

...

 1. If you have a false-negative, put the spam into the Spam.Report
 folder
 2. If you have a false-positive (which has all kinds of ugly
 spamassassin protective markup in it), put the message into the
 Spam.NotSpam folder

...

No, the plugin doesn't support that. You _could_ do that by loading the plugin twice (the plugin supports compiling as a different plugin name) and configuring the first as such:

declare "Spam.NotSpam" as "spam" folder and make the spam->notspam move a no-op, the notspam->spam the reporting and the second
declare "spam.report" as "spam" folder and make the spam->notspam move a no-op, the notspam->spam do the reporting

But that's quite complicated.

johannes

Johannes Berg

12:51 a.m.

On Wed, 2008-06-11 at 23:43 +0200, Johannes Berg wrote:

...

Kyle,

"Obviously", I qualify as a spammer because I wrote the antispam plugin. Or something like that.

...

johannes

Kyle Wheeler

6:13 p.m.

On Wednesday, June 11 at 11:51 PM, quoth Johannes Berg:

...

Heh, sorry about that - I have my server set to reject non-list-related messages to my kyle-dovecot address... which was a good idea until I started posting to the list again.

~Kyle

When you pray, do not be like the hypocrites, who love to stand and pray in the synagogues and on street corners so that others may see them. Amen, I say to you, they have received their reward. -- Matthew 6:5

Steffen Kaiser

12:08 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Wed, 11 Jun 2008, Kyle Wheeler wrote:

...

those directories is turned into piping the message to a script of my choosing (a different one for each folder).

There is a pipe plugin, which README exactly fits this sentence, search the archives of this list.

Bye,

Steffen Kaiser -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFIUOeBVJMDrex4hCIRAuGRAKCoeDuCI2c595H0ubW8BKTT4NS0EwCePkK3 fDuFvdTMtQEo7S4/BERSEn8= =YfAK -----END PGP SIGNATURE-----

6171

Age (days ago)

6172

Last active (days ago)

List overview

14 comments

7 participants

participants (7)

Andre Rodier
Daniel L. Miller
Hugo Monteiro
Johannes Berg
Juan Asensio Sánchez
Kyle Wheeler
Steffen Kaiser