[Dovecot] Released Pigeonhole v0.4.0 for Dovecot v2.2.1.
Hello Dovecot users,
I finally release the first version of Pigeonhole v0.4 for Dovecot v2.2. The main reason for the delay was that some unexpected (design) problems occurred with the new doveadm-sieve plugin, which allows synchronizing sieve scripts using doveadm sync.
One important change is that I incorporated the extprograms plugin into the main Pigeonhole release. With this plugin it is possible to execute administrator-controlled external programs (shell scripts) from the Sieve interpreter, e.g. for special message delivery, filtering and string manipulation.
The list of new features is not that impressive. I've been working on IMAP URLAUTH, IMAP CATENATE and HTTP support for Dovecot. Currently, I'm building an SMTP submission proxy server. After all that I plan to spend more time on Sieve development. One of the most important issues on my list is implementing support for using alternative types of storage (e.g. a database) for Sieve scripts, rather than only a filesystem directory as it is now.
Changelog v0.4.0:
- Added doveadm-sieve plugin that provides the possibility to sync Sieve scripts using doveadm sync along with the user's mailboxes.
- Added the Sieve extprograms plugin to the main Pigeonhole package. It is still a plugin, but it is now included so that a separate compile is no longer necessary and distributors are likely to include it. The extprograms plugin provides Sieve language extensions that allows executing (administrator-controlled) external programs for message delivery, message filtering and string manipulation. Refer to doc/plugins/sieve_extprograms.txt for more information.
- Added debug message showing Pigeonhole version at initialization. Makes it very clear that the plugin is properly loaded.
- Finished implementation of the Sieve include extension. It should now fully conform to RFC 6609. The main addition is the new :optional tag which makes the include command ignore missing included scripts without an error.
- Finished implementation of the Sieve environment extension as much as possible. Environment items "location", "phase" and "domain" now also return a usable value.
The release is available as follows:
http://www.rename-it.nl/dovecot/2.2/dovecot-2.2-pigeonhole-0.4.0.tar.gz http://www.rename-it.nl/dovecot/2.2/dovecot-2.2-pigeonhole-0.4.0.tar.gz.sig
Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for more information. Have fun testing this new release and don't hesitate to notify me when there are any problems.
Regards,
-- Stephan Bosch stephan@rename-it.nl
On 2013-05-09 10:35 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Hello Dovecot users,
I finally release the first version of Pigeonhole v0.4 for Dovecot v2.2.
Yay! Thanks so much Stephan.
One question though...
Currently, I'm building an SMTP submission proxy server.
Can you elaborate on this? Does this mean for example, that we could use dovecot as our submission server and auto-save-to-sent, avoiding the overhead of the 'Copy to Sent' behavior we are currently forced to use where a message is first uploaded when it is sent, then again when it is saved to the sent folder?
This would be awesome, as we deal with a lot of large attachments, and when people are working from home, it can take many many seconds (even a minute or so for very large attachments depending on their internet connection speed) to send, and then it has to do it all over again to save to sent.
Thanks again!
--
Best regards,
Charles
On 5/9/2013 6:05 PM, Charles Marcus wrote:
On 2013-05-09 10:35 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Currently, I'm building an SMTP submission proxy server.
Can you elaborate on this?
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA. But the main goal for this project is to implement an SMTP submission server with full support for the LEMONADE profile (https://tools.ietf.org/html/rfc4550). It acts as a proxy server, so it doesn't queue anything; once the client sees a success reply for the message submission, it is already accepted in the actual MTA queue.
For authentication it uses the normal Dovecot login strategy. This means that after authentication, it can run with the user's privileges and access the user's mail storage directly. However, I also plan to provide support for running it as a completely unprivileged service.
Does this mean for example, that we could use dovecot as our submission server ...
Yes.
... and auto-save-to-sent, avoiding the overhead of the 'Copy to Sent' behavior we are currently forced to use where a message is first uploaded when it is sent, then again when it is saved to the sent folder?
Depends a bit on what you have in mind. The LEMONADE profile has a forward-without-download scheme for this, using the SMTP BURL extension (https://tools.ietf.org/html/rfc4468) and IMAP CATENATE (https://tools.ietf.org/html/rfc4469) and URLAUTH (https://tools.ietf.org/html/rfc4467). Using CATENATE, the client can combine existing message parts with new text to compose a new message. Using SMTP BURL and IMAP URLAUTH, the SMTP server can access that message directly from the IMAP server without the need for the client to download it first.
Some more direct approach is also possible, e.g. let the submission server store the message in the Sent folder implicitly (as Google apparently does). This has a few problems though, mainly that the mail client will have to be configured correctly not to store an additional copy of its own. Unfortunately, there is no standardized method of signalling this from server to client. Google probably filters out the duplicates, we don't really know. Also, which folder does the user use as Sent folder? We could use the IMAP SPECIAL-USE (https://www.ietf.org/rfc/rfc6154.txt) extension to find out. Anyway, adding support for implicitly storing sent messages in the \Sent folder should be easy enough, but it is not a fool-proof solution. Timo was discussing this a while back on the SMTP mailinglist, but people there weren't too enthusiastic about standardizing a feature like this so far.
This would be awesome, as we deal with a lot of large attachments, and when people are working from home, it can take many many seconds (even a minute or so for very large attachments depending on their internet connection speed) to send, and then it has to do it all over again to save to sent.
The LEMONADE profile is rather elaborate and not many clients or servers support it yet. I'm hoping that by providing a chicken, more eggs will follow soon.
To provide some sort of solution for the short term, I guess I'll just add an optional auto-save-to-sent feature. When the submission service has direct access to the user's mail storage, that is trivial to implement. However, if the submission service is unprivileged, that will be a little more difficult. Probably, in that case I'll make it use a special support service to perform the actual delivery to the sent folder. Any suggestions are welcome.
Regards,
Stephan.
On 10.5.2013, at 0.23, Stephan Bosch <stephan@rename-it.nl> wrote:
Anyway, adding support for implicitly storing sent messages in the \Sent folder should be easy enough, but it is not a fool-proof solution. Timo was discussing this a while back on the SMTP mailinglist, but people there weren't too enthusiastic about standardizing a feature like this so far.
I was thinking about continuing that after we have implemented something. It seems that the "best" idea really was the one I had in the beginning and Alexey also suggested:
250 2.0.0 [localcopy] Message accepted
Very IMAP-like instead of SMTP-like, but .. well, it's a very IMAP-specific feature in any case (until IMAP replacement protocol comes some day).
On 5/9/2013 11:35 PM, Timo Sirainen wrote:
On 10.5.2013, at 0.23, Stephan Bosch <stephan@rename-it.nl> wrote:
Anyway, adding support for implicitly storing sent messages in the \Sent folder should be easy enough, but it is not a fool-proof solution. Timo was discussing this a while back on the SMTP mailinglist, but people there weren't too enthusiastic about standardizing a feature like this so far. I was thinking about continuing that after we have implemented something.
Yeah, good idea.
My patch queue (hg mq) is here:
http://hg.rename-it.nl/dovecot-2.2-patches/
The toplevel submission.patch also updates the Dovecot TODO file, so you can look there for what remains to be done. The auto-save-to-sent feature is not mentioned there yet.
It seems that the "best" idea really was the one I had in the beginning and Alexey also suggested:
250 2.0.0 [localcopy] Message accepted
Very IMAP-like instead of SMTP-like, but .. well, it's a very IMAP-specific feature in any case (until IMAP replacement protocol comes some day).
ManageSieve uses this textual response-code style as well. :)
Some SMTP replies already have a somewhat structured text part, e.g. a domain name as the first word, or an optional mailbox specification enclosed in '<' and '>'. So, I don't think it would be completely strange to put it there.
Regards,
Stephan.
Hey Stephan,
On 05/09/2013 11:23 PM, Stephan Bosch wrote:
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA. But the main goal for this project is to implement an SMTP submission server with full support for the LEMONADE profile (https://tools.ietf.org/html/rfc4550). It acts as a proxy server, so it doesn't queue anything; once the client sees a success reply for the message submission, it is already accepted in the actual MTA queue.
I have one remark and one question:
Remark: Don't forget XCLIENT / XFORWARD support to help the "real" MTA understand who it's really talking to.
Question: Will the new SMTP submission code somehow solve the robustness issues with sieve doing SMTP submission? We talked about it last November. Subject was "[Dovecot] Sieve puts incoming message into inbox on any problem with submission_host".
Regards
Christian
On 5/10/2013 12:12 PM, Christian Rohmann wrote:
Hey Stephan,
On 05/09/2013 11:23 PM, Stephan Bosch wrote:
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA. But the main goal for this project is to implement an SMTP submission server with full support for the LEMONADE profile (https://tools.ietf.org/html/rfc4550). It acts as a proxy server, so it doesn't queue anything; once the client sees a success reply for the message submission, it is already accepted in the actual MTA queue.
I have one remark and one question:
Remark: Don't forget XCLIENT / XFORWARD support to help the "real" MTA understand who it's really talking to.
XCLIENT is already implemented. But, afaik, this is only supported by Postfix. I also noticed a problem with XCLIENT LOGIN=<user>. Even when that is specified, Postfix doesn't allow relaying for a client authenticated through Dovecot submission. I am still not sure what I am messing up there (I did configure smtp_recipient_restrictions correctly I believe).
What is XFORWARD good for? It looks very similar, but focused on dealing with mail filter intermediaries. I don't think this applies here.
Question: Will the new SMTP submission code somehow solve the robustness issues with sieve doing SMTP submission? We talked about it last November. Subject was "[Dovecot] Sieve puts incoming message into inbox on any problem with submission_host".
Probably. I'll keep that in mind when implementing the new SMTP client in lib-smtp. It will also require some changes in the LDA/LMTP handling of temporary delivery failures. This is also a good occasion to finish Sieve ereject support.
Regards,
Stephan.
- Stephan Bosch <stephan@rename-it.nl>:
On 5/10/2013 12:12 PM, Christian Rohmann wrote:
Hey Stephan,
On 05/09/2013 11:23 PM, Stephan Bosch wrote:
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA. But the main goal for this project is to implement an SMTP submission server with full support for the LEMONADE profile (https://tools.ietf.org/html/rfc4550). It acts as a proxy server, so it doesn't queue anything; once the client sees a success reply for the message submission, it is already accepted in the actual MTA queue.
I have one remark and one question:
Remark: Don't forget XCLIENT / XFORWARD support to help the "real" MTA understand who it's really talking to.
XCLIENT is already implemented. But, afaik, this is only supported by Postfix. I also noticed a problem with XCLIENT LOGIN=<user>. Even when that is specified, Postfix doesn't allow relaying for a client authenticated through Dovecot submission. I am still not sure what I am messing up there (I did configure smtp_recipient_restrictions correctly I believe).
What is XFORWARD good for? It looks very similar, but focused on dealing with mail filter intermediaries. I don't think this applies here.
It forwards the META data for logging purposes and is useful to create consistent logging.
p@rick
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On 5/10/2013 2:20 PM, Patrick Ben Koetter wrote:
- Stephan Bosch <stephan@rename-it.nl>:
What is XFORWARD good for? It looks very similar, but focused on dealing with mail filter intermediaries. I don't think this applies here. It forwards the META data for logging purposes and is useful to create consistent logging.
I understood as much from:
http://www.postfix.org/XFORWARD_README.html
But I don't quite understand how this is different from XCLIENT, apart from the SOURCE and IDENT items perhaps.
Regards,
Stephan.
- Stephan Bosch <stephan@rename-it.nl>:
On 5/10/2013 2:20 PM, Patrick Ben Koetter wrote:
- Stephan Bosch <stephan@rename-it.nl>:
What is XFORWARD good for? It looks very similar, but focused on dealing with mail filter intermediaries. I don't think this applies here. It forwards the META data for logging purposes and is useful to create consistent logging.
I understood as much from:
http://www.postfix.org/XFORWARD_README.html
But I don't quite understand how this is different from XCLIENT, apart from the SOURCE and IDENT items perhaps.
XCLIENT impersonates a client and the SMTP server will act as if the XCLIENT was the real client, e.g. it will apply ACLs and other policies to the XCLIENT personality.
XFORWARD will not alter the SMTP server behaviour. The client and message data from XFORWARD will only be used for logging purposes.
p@rick
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
On 5/10/2013 3:02 PM, Patrick Ben Koetter wrote:
- Stephan Bosch <stephan@rename-it.nl>:
But I don't quite understand how this is different from XCLIENT, apart from the SOURCE and IDENT items perhaps. XCLIENT impersonates a client and the SMTP server will act as if the XCLIENT was the real client, e.g. it will apply ACLs and other policies to the XCLIENT personality.
XFORWARD will not alter the SMTP server behaviour. The client and message data from XFORWARD will only be used for logging purposes.
Ah.
One question: what should I do when the server allows both of these? Or is that impossible?
Regards,
Stephan.
- Stephan Bosch <stephan@rename-it.nl>:
On 5/10/2013 3:02 PM, Patrick Ben Koetter wrote:
- Stephan Bosch <stephan@rename-it.nl>:
But I don't quite understand how this is different from XCLIENT, apart from the SOURCE and IDENT items perhaps. XCLIENT impersonates a client and the SMTP server will act as if the XCLIENT was the real client, e.g. it will apply ACLs and other policies to the XCLIENT personality.
XFORWARD will not alter the SMTP server behaviour. The client and message data from XFORWARD will only be used for logging purposes.
Ah.
One question: what should I do when the server allows both of these? Or is that impossible?
It is possible to offer both capabilities and I think the goal defines if you should impersonate another client or merely forward client meta data.
p@rick
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Stephan, On Fri, 10 May 2013, Stephan Bosch wrote:
On 5/10/2013 12:12 PM, Christian Rohmann wrote:
Hey Stephan,
On 05/09/2013 11:23 PM, Stephan Bosch wrote:
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA. But the main goal for this project is to implement an SMTP submission server with full support for the LEMONADE profile (https://tools.ietf.org/html/rfc4550). It acts as a proxy server, so it doesn't queue anything; once the client sees a success reply for the message submission, it is already accepted in the actual MTA queue.
I have one remark and one question:
Remark: Don't forget XCLIENT / XFORWARD support to help the "real" MTA understand who it's really talking to.
XCLIENT is already implemented. But, afaik, this is only supported by Postfix. Exim has the "-bs" command line option. From spec:
-bs
This option causes Exim to accept one or more messages by reading SMTP
commands on the standard input, and producing SMTP replies on the standard
output. SMTP policy controls, as defined in ACLs (see chapter 42) are
applied. Some user agents use this interface as a way of passing
locally-generated messages to the MTA.
In this usage, if the caller of Exim is trusted, or untrusted_set_sender is
set, the senders of messages are taken from the SMTP MAIL commands.
Otherwise the content of these commands is ignored and the sender is set up
as the calling user. Unqualified addresses are automatically qualified
using qualify_domain and qualify_recipient, as appropriate, unless the -bnq
option is used.
The -bs option is also used to run Exim from inetd, as an alternative to
using a listening daemon. Exim can distinguish the two cases by checking
whether the standard input is a TCP/IP socket. When Exim is called from
inetd, the source of the mail is assumed to be remote, and the comments
above concerning senders and qualification do not apply. In this situation,
Exim behaves in exactly the same way as it does when receiving a message
via the listening daemon.Could you implement this interface to a backend server, too?
Thanks for your work, regards, Lutz
On 5/10/2013 2:43 PM, Lutz Preßler wrote:
Stephan, On Fri, 10 May 2013, Stephan Bosch wrote: Exim has the "-bs" command line option. From spec:
Could you implement this interface to a backend server, too?
As long as it talks SMTP, it shouldn't be that difficult to facilitate this. But, what exactly is the benefit of this over a normal TCP connection?
Regards,
Stephan.
Stephan Bosch wrote:
On 5/10/2013 2:43 PM, Lutz Preßler wrote:
Stephan, On Fri, 10 May 2013, Stephan Bosch wrote: Exim has the "-bs" command line option. From spec:
Could you implement this interface to a backend server, too?
As long as it talks SMTP, it shouldn't be that difficult to facilitate this. But, what exactly is the benefit of this over a normal TCP connection?
Might by useful with (x)inetd which opens the TCP listener itself and feeds SMTP commands into stdin of an executable.
Regards Daniel
On 2013-05-09 5:23 PM, Stephan Bosch <stephan@rename-it.nl> wrote:
On 5/9/2013 6:05 PM, Charles Marcus wrote:
On 2013-05-09 10:35 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Currently, I'm building an SMTP submission proxy server.
Can you elaborate on this?
It basically acts as a front-end to your normal MTA. First of all, it provides a convenient way to add SMTP AUTH support to any MTA.
Excellent, thanks Stephan.
Just to make sure I understand this correctly, basically, this means that if someone needs to provide SASL *client* capability on a postfix+dovecot system - ie, so that postfix can relay certain emails to certain destinations through an alternate relay server that requires SASL based SMTP AUTH - they would no longer need cyrus-sasl to accomplish this?
... and auto-save-to-sent, avoiding the overhead of the 'Copy to Sent' behavior we are currently forced to use where a message is first uploaded when it is sent, then again when it is saved to the sent folder?
Depends a bit on what you have in mind. The LEMONADE profile has a forward-without-download scheme for this, using the SMTP BURL extension (https://tools.ietf.org/html/rfc4468) and IMAP CATENATE (https://tools.ietf.org/html/rfc4469) and URLAUTH (https://tools.ietf.org/html/rfc4467). Using CATENATE, the client can combine existing message parts with new text to compose a new message. Using SMTP BURL and IMAP URLAUTH, the SMTP server can access that message directly from the IMAP server without the need for the client to download it first.
Some more direct approach is also possible, e.g. let the submission server store the message in the Sent folder implicitly (as Google apparently does). This has a few problems though, mainly that the mail client will have to be configured correctly not to store an additional copy of its own. Unfortunately, there is no standardized method of signalling this from server to client. Google probably filters out the duplicates, we don't really know. Also, which folder does the user use as Sent folder? We could use the IMAP SPECIAL-USE (https://www.ietf.org/rfc/rfc6154.txt) extension to find out. Anyway, adding support for implicitly storing sent messages in the \Sent folder should be easy enough, but it is not a fool-proof solution. Timo was discussing this a while back on the SMTP mailinglist, but people there weren't too enthusiastic about standardizing a feature like this so far.
Ok, I agree the main problem would be the possibility of duplicate messages, but I would think with the powerful filtering capabilities of sieve, it should be possible (not sure how easy though) to hard code a filter to watch for and filter/remove/delete any duplicate that the MUA uploads.
The LEMONADE profile is rather elaborate and not many clients or servers support it yet. I'm hoping that by providing a chicken, more eggs will follow soon.
I like that dovecot is willing to take a chance on being first to support these kinds of enhanced services, but I will say, it is very important that any support for said enhancements be rock-solid.
To provide some sort of solution for the short term, I guess I'll just add an optional auto-save-to-sent feature.
Sounds great to me, but...
In my opinion, because of the ubiquitous nature of MUAs saving messages to a sent folder, having a reliable and low-impact method for automatically filtering/removing/deleting these duplicates out should be a requirement before this feature is considered ready. It will be a big and immediate problem for any installation that chooses to enable this feature, as virtually all MUAs will be configured to save sent messages to a/the sent folder. It will also be an ongoing problem for all installations (existing and new alike), as users add their accounts to new computers, phones, tablets and other devices/MUAs, totally ignoring the instructions from their providers that they no longer need to enable this feature.
In fact... after thinking about this some more, I wonder...
Would there be some reasonably reliable way to detect when an MUA is uploading/saving messages to the Sent folder, and if so, could the LEMONADE protocol be leveraged to create/send a 'notification' email to that user based on some kind of system template (hard coded? customizable?), informing them that there is no need to do this, and even including a link to a dovecot wiki page explaining how to disable the 'Save copy to Sent folder' feature in common MUAs?
Then it would be up to individual SysAdmins to keep the wiki updated with sections for any clients they become aware of that aren't already on the page.
Maybe future enhancements could even (try to) detect the MUA client (is this possible to do reliably?), and a direct link to the section of the wiki for that specific client could be provided?
Another thing that I know that google is really good at is automatically filtering (I guess they're deleting?) any and all duplicate emails. I have noticed this when copying a message store from one IMAP server to a gmail account. I had cases where the number of messages in certain folders wasn't the same, and upon investigation, noticed that the original/source in fact had some duplicate messages in certain folders.
So, maybe you could 'kill two birds with one stone' so to speak. and whatever is done to address the duplicate Sent messages could also be leveraged to address duplicate messages in general? Although I guess it is not the same problem, so maybe not...
When the submission service has direct access to the user's mail storage, that is trivial to implement. However, if the submission service is unprivileged, that will be a little more difficult.
Are you talking about the difference between dovecot accessing mails with one system user, vs accessing mails with the individual users userID?
Probably, in that case I'll make it use a special support service to perform the actual delivery to the sent folder. Any suggestions are welcome.
Thanks Stephan,
--
Best regards,
Charles
On 5/10/2013 4:02 PM, Charles Marcus wrote:
On 2013-05-09 5:23 PM, Stephan Bosch <stephan@rename-it.nl> wrote:
First of all, it provides a convenient way to add SMTP AUTH support to any MTA.
Just to make sure I understand this correctly, basically, this means that if someone needs to provide SASL *client* capability on a postfix+dovecot system - ie, so that postfix can relay certain emails to certain destinations through an alternate relay server that requires SASL based SMTP AUTH - they would no longer need cyrus-sasl to accomplish this?
Ehhh.. no :) It implements the server-side SMTP AUTH, so that your MTA doesn't have to any more. So the client will authenticate to Dovecot rather than to the regular MTA/MSA. But, again, this is a rather trivial matter and not the main reason for building this proxy.
The LEMONADE profile is rather elaborate and not many clients or servers support it yet. I'm hoping that by providing a chicken, more eggs will follow soon.
I like that dovecot is willing to take a chance on being first to support these kinds of enhanced services, but I will say, it is very important that any support for said enhancements be rock-solid.
What do you mean exactly?
To provide some sort of solution for the short term, I guess I'll just add an optional auto-save-to-sent feature.
Sounds great to me, but...
In my opinion, because of the ubiquitous nature of MUAs saving messages to a sent folder, having a reliable and low-impact method for automatically filtering/removing/deleting these duplicates out should be a requirement before this feature is considered ready. It will be a big and immediate problem for any installation that chooses to enable this feature, as virtually all MUAs will be configured to save sent messages to a/the sent folder. It will also be an ongoing problem for all installations (existing and new alike), as users add their accounts to new computers, phones, tablets and other devices/MUAs, totally ignoring the instructions from their providers that they no longer need to enable this feature.
Yes, I agree.
In fact... after thinking about this some more, I wonder...
Would there be some reasonably reliable way to detect when an MUA is uploading/saving messages to the Sent folder,
Hmm, not sure. Do MUAs normally generate the Message-ID header, or is that created by the server? That could be one way to detect the duplicates in the Sent folder.
and if so, could the LEMONADE protocol be leveraged to create/send a 'notification' email to that user based on some kind of system template (hard coded? customizable?), informing them that there is no need to do this, and even including a link to a dovecot wiki page explaining how to disable the 'Save copy to Sent folder' feature in common MUAs?
Then it would be up to individual SysAdmins to keep the wiki updated with sections for any clients they become aware of that aren't already on the page.
Maybe future enhancements could even (try to) detect the MUA client (is this possible to do reliably?), and a direct link to the section of the wiki for that specific client could be provided?
Relying on user action doesn't sound like a very appealing solution to me. :)
Another thing that I know that google is really good at is automatically filtering (I guess they're deleting?) any and all duplicate emails. I have noticed this when copying a message store from one IMAP server to a gmail account. I had cases where the number of messages in certain folders wasn't the same, and upon investigation, noticed that the original/source in fact had some duplicate messages in certain folders.
That is entirely possible.
So, maybe you could 'kill two birds with one stone' so to speak. and whatever is done to address the duplicate Sent messages could also be leveraged to address duplicate messages in general? Although I guess it is not the same problem, so maybe not...
You mean something like this?
http://hg.rename-it.nl/dovecot-2.2-pigeonhole/raw-file/tip/doc/rfc/spec-bosc...
When the submission service has direct access to the user's mail storage, that is trivial to implement. However, if the submission service is unprivileged, that will be a little more difficult.
Are you talking about the difference between dovecot accessing mails with one system user, vs accessing mails with the individual users userID?
No, I'd like to be able to run SMTP submission without any direct filesystem access privileges, with e.g. one submission process handing submissions for many clients/users at the same time. For accessing the URLAUTHs there is already a support service in current Dovecot. Something similar could be devised for storing messages to Sent folders in that case.
Regards,
Stephan.
On 2013-05-10 10:37 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
On 5/10/2013 4:02 PM, Charles Marcus wrote:
On 2013-05-09 5:23 PM, Stephan Bosch <stephan@rename-it.nl> wrote:
First of all, it provides a convenient way to add SMTP AUTH support to any MTA.
Just to make sure I understand this correctly, basically, this means that if someone needs to provide SASL *client* capability on a postfix+dovecot system - ie, so that postfix can relay certain emails to certain destinations through an alternate relay server that requires SASL based SMTP AUTH - they would no longer need cyrus-sasl to accomplish this?
Ehhh.. no :) It implements the server-side SMTP AUTH, so that your MTA doesn't have to any more. So the client will authenticate to Dovecot rather than to the regular MTA/MSA. But, again, this is a rather trivial matter and not the main reason for building this proxy.
Ok... so, will this make it easier to add client side sasl support to dovecots dovecot-sasl implementation to eliminate the need for postfix+dovecot systems to continue to rely on cyrus-sasl for MTA client side sasl support?
The LEMONADE profile is rather elaborate and not many clients or servers support it yet. I'm hoping that by providing a chicken, more eggs will follow soon.
I like that dovecot is willing to take a chance on being first to support these kinds of enhanced services, but I will say, it is very important that any support for said enhancements be rock-solid.
What do you mean exactly?
Sorry - was referring mainly to my later comments about how to implement the Save-To-Sent folder stuff...
Would there be some reasonably reliable way to detect when an MUA is uploading/saving messages to the Sent folder,
Hmm, not sure. Do MUAs normally generate the Message-ID header, or is that created by the server? That could be one way to detect the duplicates in the Sent folder.
Sorry, I have no idea... but...
Maybe this feature could simply require the use of the dovecot submission server, so all you'd have to do is figure out how to best let the submission server handle it. Maybe have it add a custom ID header that is later removed? Or maybe even not removed?
and if so, could the LEMONADE protocol be leveraged to create/send a 'notification' email to that user based on some kind of system template (hard coded? customizable?), informing them that there is no need to do this, and even including a link to a dovecot wiki page explaining how to disable the 'Save copy to Sent folder' feature in common MUAs?
Then it would be up to individual SysAdmins to keep the wiki updated with sections for any clients they become aware of that aren't already on the page.
Maybe future enhancements could even (try to) detect the MUA client (is this possible to do reliably?), and a direct link to the section of the wiki for that specific client could be provided?
Relying on user action doesn't sound like a very appealing solution to me. :)
Nor me, but the fact is, since MUAs are configured by end users, and there is no way dovecot can change an MUAs account settings (to disable Save-To-Sent), what choice do we have?
That is why I suggested some way to automatically inform users about this.
Another (maybe better) option would be the SysAdmin could define a specific email address to handle these notifications, and it would be on them to get their users' MUAs configured correctly.
I'd still like to see the option to inform users directly though - again, if this is even possible.
Another thing that I know that google is really good at is automatically filtering (I guess they're deleting?) any and all duplicate emails. I have noticed this when copying a message store from one IMAP server to a gmail account. I had cases where the number of messages in certain folders wasn't the same, and upon investigation, noticed that the original/source in fact had some duplicate messages in certain folders.
That is entirely possible.
So, maybe you could 'kill two birds with one stone' so to speak. and whatever is done to address the duplicate Sent messages could also be leveraged to address duplicate messages in general? Although I guess it is not the same problem, so maybe not...
You mean something like this?
http://hg.rename-it.nl/dovecot-2.2-pigeonhole/raw-file/tip/doc/rfc/spec-bosc...
Lol! I see you're way ahead of me... ;)
Thanks again Stephan.
--
Best regards,
Charles
Am 10.05.2013 17:17, schrieb Charles Marcus:
On 2013-05-10 10:37 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Ehhh.. no :) It implements the server-side SMTP AUTH, so that your MTA doesn't have to any more. So the client will authenticate to Dovecot rather than to the regular MTA/MSA. But, again, this is a rather trivial matter and not the main reason for building this proxy.
Ok... so, will this make it easier to add client side sasl support to dovecots dovecot-sasl implementation to eliminate the need for postfix+dovecot systems to continue to rely on cyrus-sasl for MTA client side sasl support?
[root@srv-rhsoft:~]$ postconf -n | grep dovecot smtpd_sasl_type = dovecot
dovecot.conf: service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660 user = postfix group = postfix } }
and any dovecot user works the same way and with the same auth-mechs with postfix - in use here since 2009
any in this case means rally any like also below to get rid of problems with legacy client-configs of a old server which supported % instead of @, now both works equal as username
auth_username_translation = %@AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz
Searching my mailboxes for something and noticed this in my spam bucket...
On 2013-05-10 11:35 AM, Reindl Harald <h.reindl@thelounge.net> wrote:
Am 10.05.2013 17:17, schrieb Charles Marcus:
On 2013-05-10 10:37 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Ehhh.. no :) It implements the server-side SMTP AUTH, so that your MTA doesn't have to any more. So the client will authenticate to Dovecot rather than to the regular MTA/MSA. But, again, this is a rather trivial matter and not the main reason for building this proxy.
Ok... so, will this make it easier to add client side sasl support to dovecots dovecot-sasl implementation to eliminate the need for postfix+dovecot systems to continue to rely on cyrus-sasl for MTA client side sasl support?
[root@srv-rhsoft:~]$ postconf -n | grep dovecot smtpd_sasl_type = dovecot
dovecot.conf: service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660 user = postfix group = postfix } }
and any dovecot user works the same way and with the same auth-mechs with postfix - in use here since 2009
What does that have to do with providing MTA CLIENT-side SASL_AUTH support?
Do you not know the difference between smtpD_sasl and smtP_sasl?
Server-side = smtpd
Client-side = smtp
Currently the only way postfix can perform CLIENT-side sasl_auth is with cyrus-sasl, because dovecot_sasl doesn't support it, and never has.
any in this case means rally any like also below to get rid of problems with legacy client-configs of a old server which supported % instead of @, now both works equal as username
auth_username_translation = %@AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz
No clue what you're talking about here...
Am 11.05.2013 16:38, schrieb Charles Marcus:
What does that have to do with providing MTA CLIENT-side SASL_AUTH support?
nothing and it was explained you several times here and on the postfix-list that it is not supported after looking in the archives and so i have no clue why you do not understand that this is NOT useful because postfix's SASL client implementation is using a mapping which has usually NOTHING to do with your local users
why would you use your local users to authenticate against foreign servers? so why are you hijack every random thread with the same topic?
Do you not know the difference between smtpD_sasl and smtP_sasl? Server-side = smtpd Client-side = smtp
i understand much more than only this
auth_username_translation = %@AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz No clue what you're talking about here...
well, this maybe because you understand generally not how authentication works and what makes sense and what is clueless
Stephan Bosch wrote:
On 2013-05-09 10:35 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Currently, I'm building an SMTP submission proxy server. To provide some sort of solution for the short term, I guess I'll just add an optional auto-save-to-sent feature. When the submission service has direct access to the user's mail storage, that is trivial to implement. However, if the submission service is unprivileged, that will be a little more difficult. Probably, in that case I'll make it use a special support service to perform the actual delivery to the sent folder. Any suggestions are welcome.
Please don't forget to implement the "submission director proxy", we absolutely need this in order to be able to work with NFS+mdbox and prevent corrupted indexes.
We cannot just store something directly into the local "Sent" Folder, we need to go through the director to the appropriate NFS client host.
Regards Daniel
On 5/14/2013 11:52 PM, Daniel Parthey wrote:
Stephan Bosch wrote:
On 2013-05-09 10:35 AM, Stephan Bosch <stephan@rename-it.nl> wrote:
Currently, I'm building an SMTP submission proxy server. To provide some sort of solution for the short term, I guess I'll just add an optional auto-save-to-sent feature. When the submission service has direct access to the user's mail storage, that is trivial to implement. However, if the submission service is unprivileged, that will be a little more difficult. Probably, in that case I'll make it use a special support service to perform the actual delivery to the sent folder. Any suggestions are welcome. Please don't forget to implement the "submission director proxy", we absolutely need this in order to be able to work with NFS+mdbox and prevent corrupted indexes.
We cannot just store something directly into the local "Sent" Folder, we need to go through the director to the appropriate NFS client host.
Yes, that is already on the TODO list.
Regards,
Stephan.
On 5/9/2013 11:05 AM, Charles Marcus wrote:
This would be awesome, as we deal with a lot of large attachments, and when people are working from home, it can take many many seconds (even a minute or so for very large attachments depending on their internet connection speed) to send, and then it has to do it all over again to save to sent.
Charles have you looked into Thunderbird Filelink?
https://support.mozillamessaging.com/en-US/kb/filelink-large-attachments
You can use a 3rd party service or your own WebDAV server. Keeps large attachments out of mailbox storage, doesn't save them to Sent Items, moves the file over the wire only once.
-- Stan
On 2013-05-10 4:41 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
On 5/9/2013 11:05 AM, Charles Marcus wrote:
This would be awesome, as we deal with a lot of large attachments, and when people are working from home, it can take many many seconds (even a minute or so for very large attachments depending on their internet connection speed) to send, and then it has to do it all over again to save to sent.
Charles have you looked into Thunderbird Filelink?
https://support.mozillamessaging.com/en-US/kb/filelink-large-attachments
You can use a 3rd party service or your own WebDAV server. Keeps large attachments out of mailbox storage, doesn't save them to Sent Items, moves the file over the wire only once.
Hi Stan,
Thanks for the idea, and yes, I'm aware of filelink. While the idea is nice, it wouldn't fulfill our needs. Our data must be kept private, and while there is the WebDAV extension, its functionality is very basic (files with the same name are overwritten silently, no support for expiring links, etc).
But anyway, I don't much like the idea of totally separating attachments from emails, it just feels off to me.
Thanks,
Charles
Am 10.05.2013 12:42, schrieb Charles Marcus:
On 2013-05-10 4:41 AM, Stan Hoeppner <stan@hardwarefreak.com> wrote:
On 5/9/2013 11:05 AM, Charles Marcus wrote:
This would be awesome, as we deal with a lot of large attachments, and when people are working from home, it can take many many seconds (even a minute or so for very large attachments depending on their internet connection speed) to send, and then it has to do it all over again to save to sent.
Charles have you looked into Thunderbird Filelink?
https://support.mozillamessaging.com/en-US/kb/filelink-large-attachments
You can use a 3rd party service or your own WebDAV server. Keeps large attachments out of mailbox storage, doesn't save them to Sent Items, moves the file over the wire only once.
Hi Stan,
Thanks for the idea, and yes, I'm aware of filelink. While the idea is nice, it wouldn't fulfill our needs. Our data must be kept private, and while there is the WebDAV extension, its functionality is very basic (files with the same name are overwritten silently, no support for expiring links, etc).
But anyway, I don't much like the idea of totally separating attachments from emails, it just feels off to me.
Thanks,
Charles
just a little off topic but may for interest to sombody
https://github.com/fincluster/owncloud_mail_attachments
Best Regards MfG Robert Schetterer
-- [*] sys4 AG
http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Stephan Bosch wrote:
One of the most important issues on my list is implementing support for using alternative types of storage (e.g. a database) for Sieve scripts, rather than only a filesystem directory as it is now.
What about using Dovecot's lib-fs to abstract the storage layer?
Regards Daniel
On 5/14/2013 11:12 PM, Daniel Parthey wrote:
Stephan Bosch wrote:
One of the most important issues on my list is implementing support for using alternative types of storage (e.g. a database) for Sieve scripts, rather than only a filesystem directory as it is now. What about using Dovecot's lib-fs to abstract the storage layer?
Yes, that is one of the possibilities I have in mind.
Regards,
Stephan.
participants (10)
-
Charles Marcus
-
Christian Rohmann
-
Daniel Parthey
-
Lutz Preßler
-
Patrick Ben Koetter
-
Reindl Harald
-
Robert Schetterer
-
Stan Hoeppner
-
Stephan Bosch
-
Timo Sirainen