[Dovecot] Three oddities
FBSD 4.8, Dovecot-0.99.10 1) I have mail accounts for people who use Macs and Windows, so their mail is stored on a Unix box where they don't have accounts. When their mail comes in, it is stored in /var/mail/<accountname>. When I was setting this up, I used my own Unix account for testing, and that works fine. But for the other mail accounts, which don't correspond to Unix accounts, I want to give ownership of the /var/mail/* files to another account that doesn't have my privileges. However, no one can access their mail unless I own their files. As long as I own all the /var/mail/* files, everything works fine. But if I shift ownership to another account, users can log on to Dovecot but attempts to access an IMAP folder return an error.
All files and directories in /var/mail/* are in the mail group. I tried adding dovecot to that group, but that didn't help.
Since everyone can log on to Dovecot, but can't access their mail unless I own their files, I'm assuming that there's some simple ownership setting that I'm overlooking. But I haven't a clue what it is. Can someone tell me what I need to change?
The Macs have Eudora 4.2 installed, and the Win boxes have Eudora 5.1. The Macs can delete mail, put it in the trash, and empty the trash. The Win clients can mark mail deleted, but they can't remove it. The mail stays in the IMAP folder until the same user accesses their mail from a Mac and deletes it. If a user drags a message to another folder, a copy is created in the new folder but the old copy remains in the original folder. I'm not having this problem with Mutt, so it the problem seems to be specific to the Windows verson of Eudora. Is there a fix?
We're pulling our e-mail off our ISP's POP server and loading it into Dovecot to allow people to work with their mail from different machines. I'm using getmail to make the transfer. Is anyone else doing POP to IMAP, and how are they doing it? Any suggestions?
TIA, Bob Hall
Bob Hall explained:
finally a question I can answer. :-)
if you're not going to have direct SMTP delivery, you are doing it the only way you really can. One refinement would be to collect all e-mail for all addresses into a single pop3 box and use multidrop type functionality to retrieve and disburse the mail appropriately.
in theory it would be possible to retrieve using pop3 and then push using imap but that is too ugly to even contemplate.
---eric
On Mon, Aug 18, 2003 at 09:07:57PM -0400, Eric S. Johansson wrote:
If I understand you correctly, that's exactly what I'm doing. And here I thought my setup was a thing of beauty and a joy forever.
I did look at the descriptions of multidrop setups in the fetchmail and getmail documentation, but decided, since eash user has his own POP account, to keep it that way in IMAP. Taking accounts that were seperate, blending them, and then seperating them again seemed more complicated than keeping them seperate. Am I wrong?
Bob Hall
On Tue, 2003-08-19 at 04:07, Eric S. Johansson wrote:
in theory it would be possible to retrieve using pop3 and then push using imap but that is too ugly to even contemplate.
Well, UW-IMAP supports that. I've actually thought about supporting as well :) It would work better to poll the pop3 box only when IMAP's INBOX is being accessed by IMAP client. Luckily this can be implemented as a plugin.
Timo Sirainen explained:
it's one of those "just because you can do it, doesn't mean you should" situations. One of the big problems is that you bypass a lot of normal mail system functionality like antivirus or antispam filters. In my antispam system (camram), I'm currently using procmail as a way of intercepting the mail stream and working on the messages. Depositing messages via an imap backdoor would totally bypass filtering.
Additionally, if you use a collect pop3 box and multidrop distribution, you reduce your security exposure because only one password known (i.e. that for the pop box) versus 1 for every user.
I will need to look into the plug-in capability real soon now because it might be a good way to implement a correction mechanism for messages that aren't caught by the filter.
---eric
On Tuesday, Aug 19, 2003, at 16:39 Europe/Helsinki, Eric S. Johansson wrote:
Nothing would prevent the same plugin from putting the messages through spam and virus checkers.
I meant that you'd still use multiple normal pop3 boxes. You could have per-user configuration for the pop3 box locations and passwords. Either using some ~/.dovecotrc file or ANNOTATEMORE extension once we support that.
Timo Sirainen explained:
Nothing would prevent the same plugin from putting the messages through spam and virus checkers.
quite correct and that occurred to me about 30 pico seconds after I hit the send button. at the same time, it's yet another thing that requires manual configuration. Yet another thing that can and probably will go wrong.
ah. sorry about misunderstanding. Seems like there's quite a bit more functionality in dovecot then the web site conveys. No blame. I understand the phenomenon quite well and am a victim of myself. The camram documentation is currently a year out of date and I want to get 0.2 out the door before I start updating it.
---eric
On Tue, 2003-08-19 at 03:18, Bob Hall wrote:
What userdb are you using? Dovecot gets the uid/gid for users from there. Whatever you do, don't use the "dovecot" user's uid there :) Rather create a new one. Hm. Maybe I should rename it to dovecot-login to make it more clear what it's supposed to used for..
Are there error messages in log file? /var/log/maillog probably. I can't think of any reason why it does that..
On Tue, Aug 19, 2003 at 04:23:00PM +0300, Timo Sirainen wrote:
The userdb is ldap. What are the uid and gid supposed to do? I have the default uid in dovecot-ldap.conf and the individual uidNumber and values in the LDAP registry set to the mail account (the one I want to own the mail files). The default gid and the individual gidNumbers are set to the mail group, which is the group assigned to all files in /var/mail. If the mail account owns the files, users can't access them. If I own the files, everthing works fine, but my uid and gid numbers are not listed anywhere that dovecot has access to. I know that imap-master checks uids and gids for validity, but aside from that I don't know what they're used for.
I thought that maybe the fact that I'm a wheel and the mail user is not had something to do with it, but adding the mail account to the wheel group had no effect. I tried adding dovecot and the mail user to the mail group, again with no effect. I still have to be the owner of the mail files.
I don't think that will make it clearer. Both dovecot.conf and "ps waux | grep dovecot" make it clear what the dovecot processes are used for. If you name it dovecot-login, I think it would be just as easy for someone to get confused and try to use the dovecot uid for themselves, in the belief that they need it to log in.
I think you're at a point where you need to think about how you are going to explain Dovecot to users. If relatively ignorant users like me can install it easily and securely, then Dovecot will have a reputation for being easy to use and secure. If we don't know how to install Dovecot easily and securely, then it will get a reputation for being complicated and insecure, regardless of the reality. An application's reputation, earned or unearned, tends to be established by the lowest tier of users. If I feel I understand how to install and set up Dovecot, then I will feel that my installation is secure. If I have doubts about what I've done, then I'm going to have doubts about my security. And I don't think you want to spend all your spare time explaining Dovecot to us foolish beginners.
No. Nothing in maillog, and nothing in dovecot.rawlog. That's something else I'm confused about: When exactly does Dovecot write to the raw logs? I've got a boatload of entries from 20030814, and nothing before or since.
Also, I get the following in the output from "ps waux | grep dovecot":
dovecot 1004 0.0 0.9 2208 536 ?? S 7:58AM 0:01.48 imap-login: imap-login [IP address] (imap-login)
The machine at [IP address] is physically shut down. Should there still be an imap-login process for it?
On Wed, 2003-08-20 at 00:11, Bob Hall wrote:
What userdb are you using? Dovecot gets the uid/gid for users from
The userdb is ldap. What are the uid and gid supposed to do?
imap process is run under them. Actually after reading your mail, it feels like you haven't even noticed their existence? :)
Sounds correct then. Did you verify with ps that the imap processes are also running under that user? Are you sure the full /var/mail/etc path is accessible to that user?
Well, that sounds weird. It's sounds like it's still running the imap processes under your uid?
They're just extra checks.
I thought that maybe the fact that I'm a wheel and the mail user is not had something to do with it,
No, and actually Dovecot drops the wheel group permissions if you happen to belong to it :)
I think you're at a point where you need to think about how you are going to explain Dovecot to users.
Yea .. I should write some simple to read installation manual.
But there's "Dovecot starting up" message anyway?
They're written to immediately. If you're not seeing them, there's two possibilities: dovecot.rawlog isn't located in user's home directory (or the home directory isn't given at all), or it doesn't have permissions to write there.
Hmm. Not really. Check with strace -p what it's doing?
On Wed, Aug 20, 2003 at 06:11:59AM +0300, Timo Sirainen wrote:
In the paragraph you quote below, I described setting the global and
individual uids and gids, so I must have been aware of their existence.
I thought they were being used to set permissions, but you explain
otherwise below.
The main dovecot process runs under root. If I understand correctly, it forks subprocesses (imap) that run under the user. These never show up in ps. Is there a way to capture a transitory process in ps? My Dovecot installation is currently hosed, so the imap processes may not be spawning.
Yea. How do I check that, if imap never shows up in ps?
Just validity checks? OK, that makes sense.
Yes.
I've got it in my home directory, and it's listed in homeDirectory in the ldap registry. I made dovecot.rawlog world writable. Nothing is showing up.
I'll try that as soon as I un-hose Dovecot.
Bob Hall
On Wed, 2003-08-20 at 08:19, Bob Hall wrote:
I meant the imap processes actually.
Try logging in manually?
telnet localhost imap2 x login user pass
Then you should have imap process.
I've got it in my home directory, and it's listed in homeDirectory in the ldap registry. I made dovecot.rawlog world writable. Nothing is showing up.
Oh and third thing. You had to configure dovecot with --with-rawlog. Actually that's probably the problem, otherwise it would print some permission denied errors in log file.
On Wed, Aug 20, 2003 at 08:51:32AM +0300, Timo Sirainen wrote:
Then you're right. They didn't show up in ps, so I wasn't aware of them, except knowing from the documentation that imap-master begat imap and most of the work is done in imap. So I'm assuming that imap-master and the main dovecot process (/usr/local/sbin/dovecot) are the same. I knew, in a general sort of way, that they had to be lurking somewhere, but out of sight and out of mind.
No, the problem was my stupidity, not my ignorance. I was using telnet localhost 143 which is the same, but I was grepping ps for dovecot, not imap. Now that I can see the imap processes, things are a lot clearer.
I did that back when I first started having problems, so raw logging has been enabled the whole time. So far, I've gotten records in dovecot.rawlog only with a test account where the mail user name matched the Unix account name and I was using passwd instead of LDAP. None of the actual mail user names match a Unix account name. All of the mail accounts run under a Unix account, and they all have the Unix account home directory in their LDAP homeDirectory records. I'm guessing that Dovecot isn't pulling this info from the LDAP registry, and therefore can't post inside dovecot.rawlog.
At this point, after grepping ps for imap, I've got a much better understanding of what's going on. Everthing makes sense, and everything works the way I want, except raw logging.
I think I've found a another bug. In dovecot-ldap.conf it says:
You can use same UID and GID for all user accounts if you really want to.
If the UID/GID is still found from LDAP reply, it overrides these values.
This is ambiguous. I took it to mean it (LDAP entry) overrides these (global) values What it actually means, or at least what actually happens, is it (global) overrides these (LDAP entry) values Furthermore, if user_global_uid isn't explicitly set, it defaults to 0. If I set the uid number not equal to 0 in the LDAP entry, but don't set user_global_uid, then login fails. From maillog: Aug 20 03:15:15 kongemord dovecot: Logins with UID 0 not permitted (user rjhjr) Aug 20 03:15:15 kongemord imap-login: Internal login failure: rjhjr [127.0.0.1]
The bug is this; since user_global_uid overrides LDAP entries, and since user_global_uid has a built-in default value of 0, you have to set user_global_uid, and consequently you can only use one uid setting if you use LDAP. That's exactly what I want to do, but I can think of situations where I might want to use different uid numbers for different mail accounts. I think it would be better to do things in the opposite way:
- LDAP entries override user_global_uid. This would allow multiple uid numbers, but also allow you to specify a default value for LDAP entries with no uid number.
- There's no default value for user_global_uid. If it isn't explicitly set, then it isn't set. That way, you don't have to have a default value.
- If the uid isn't set in either the LDAP entry or user_global_uid, then the user gets a login error saying that there's no uid.
Bob Hall
On Wednesday, Aug 20, 2003, at 11:29 Europe/Helsinki, Bob Hall wrote:
So I'm assuming that imap-master and the main dovecot process (/usr/local/sbin/dovecot) are the same.
yep.
With Linux you could check what home directory imap process really used by looking at /proc/pid/pwd symlink. I don't know if FreeBSD has anything similiar.
Oh? It's a bug then.
Yes, the error message could be better.
Timo, first of all I want to say how grateful I am for the time you spent helping to me.
On Wed, Aug 20, 2003 at 07:47:36PM +0300, Timo Sirainen wrote:
I'm clueless on this. What does /proc/pid/pwd symlink do? Can you give an example from the command line?
Thank god. I finally found something that *I* didn't screw up. :) As I said, it doesn't interfere with my present set up, but in other situation your LDAP users will want more flexibility.
Actually, that error message was fine. In combination with the documentation, it made it clear what the problem was. That's what error messages are supposed to do; point you to something that is covered in the documentation. My problems with interpreting other error messages had mostly to do with lack of documentation. I did a lot of googling while I was setting Dovecot up, and while I didn't get much helpful info, I did find comments along the lines of "interesting, but poorly documented". That "poorly documented" may kill your project. People aren't going to be attracted to your software if it has a reputation for being poorly documented and hard to configure. Error messages by themselves are no good. You have to think of error messages as a part of your overall documentation.
I want to second the people in other threads who suggested setting up some sort of collaborative documentation project. My reasons:
- The people who know best what documentation is needed are the people using the software. They also tend to have a better idea how to express things in ways that make sense to users.
- Doing an installation and setup is kind of exciting, and users tend to be a little hyped when it's over. I want to talk about what I've done, so I feel motivated to write about it. The developer is caught up in coding, and also has to talk to users, debug, and various administrative tasks. Writing documentation is just another task, so the developer doesn't have the motivation that some users have.
- Even though I feel motivated to write, I don't have the time to write a complete description of Dovecot/LDAP installation. (That's why they put a clock on the Leaning Tower of Piza; there's no point in having the inclination if you haven't got the time.) If the basic document already existed, I could just add the missing bits that were relevant to my experience, which would take a lot less time and be a lot more feasable for busy people.
- Because the work is spread among many people, collaborative documentation tends to happen faster.
- People who collaborate with your project help spread the word. They tend to spend time telling other people why they're putting in the effort, which means telling people what they like about your project.
Bob Hall
On Wednesday, Aug 20, 2003, at 22:24 Europe/Helsinki, Bob Hall wrote:
It would just show current directory of the process. Like:
lrwxrwxrwx 1 cras cras 0 2003-08-20 23:25 cwd -> /home/cras
Yes, but "no UID given" isn't really same as "UID 0". :)
One of the reasons I haven't yet really bothered to write much is because Dovecot is just now changing a lot. Configuration file syntax changes, namespaces were added, indexes work differently than before .. what else ..
I want to second the people in other threads who suggested setting up some sort of collaborative documentation project.
Well, I installed MoinMoin Wiki, but didn't yet look much into it. I guess I should at least remove most of the default pages. There's a lot of german text.
On Wed, Aug 20, 2003 at 11:34:05PM +0300, Timo Sirainen wrote:
OK, I'll google a bit and see if I can educate myself on this.
"no UID given" would be appropriate if Dovecot was ignoring the LDAP-supplied values and supplying no default UID. You know what Dovecot does better than I do, but it appears as if Dovecot was trying to use a UID; that of root. Since LDAP wasn't suppling UID 0, I knew the problem was with Dovecot, which led me to the config files. Had the message been "no UID given", I would have wasted time trying to figure out why Dovecot wasn't getting a UID.
Putting it differently, an error message should tell you, as briefly as possible, exactly what triggered it, without explaining it. Explanations belong in the documentation. The error message should be at the same level as the documentation; e.g if you don't document your procedure calls, the error message shouldn't contain the procedure call that triggered it. (Actually, it can, but that's for the programmers, not the users.) The user should be able to take words or phrases from the error message and use them to search the documentation. "UID 0" meets those criteria, but "no UID given" is an explanation.
No programming project is ever finished. The developers just get tired and drift away. If you wait for an obvious stopping point to write documentation, you'll wait for ever.
Looks good in lynx. I'll try Mozilla when I have time. Programming is fun, writing documentation is boring. Keep the fun/boring ratio as high as possible.
Bob
In regard to getting rawlog entries with passwd but not with LDAP:
On Wed, Aug 20, 2003 at 11:34:05PM +0300, Timo Sirainen wrote:
In FBSD, you get the files associated with a process with fstat -p <pid> For an imap process opened under LDAP authentication, the output is USER CMD PID FD MOUNT INUM MODE SZ|DV R/W postkont imap 5876 root / 2 drwxr-xr-x 1024 r postkont imap 5876 wd / 7961 drwx------ 512 r postkont imap 5876 text /usr 2301620 -r-xr-xr-x 326420 r postkont imap 5876 0* internet stream tcp c5f1efc0 postkont imap 5876 1* internet stream tcp c5f1efc0 postkont imap 5876 2 / 7957 crw-rw-rw- null r postkont imap 5876 3* local dgram c644bd20 <-> c6357f00
The working directory is marked by "wd" in the "FD" column. In this case, it's /. According to the man page, a "text" directory is an "executable text inode". I don't know what that means; I'm just quoting.
If I switch to passwd authentication, then I get two imap processes: 5826 ?? S 0:00.01 imap [bob 127.0.0.1] 5827 ?? I 0:00.00 imap: imap [bob:5826 rawlog] (imap)
fstat for the first is USER CMD PID FD MOUNT INUM MODE SZ|DV R/W bob imap 5826 root / 2 drwxr-xr-x 1024 r bob imap 5826 wd /usr 3960238 drwxr-xr-x 2560 r bob imap 5826 text /usr 2301620 -r-xr-xr-x 326420 r bob imap 5826 2 / 7957 crw-rw-rw- null r bob imap 5826 3* local dgram c6357640 <-> c6357f00 bob imap 5826 7* local stream c6357500 <-> c644bc80
The working directory is now /usr, and there's a local stream that wasn't present with LDAP.
fstat for the rawlog imap process is USER CMD PID FD MOUNT INUM MODE SZ|DV R/W bob imap 5827 root / 2 drwxr-xr-x 1024 r bob imap 5827 wd /usr 3960238 drwxr-xr-x 2560 r bob imap 5827 text /usr 2301620 -r-xr-xr-x 326420 r bob imap 5827 0* internet stream tcp c5ea06c0 bob imap 5827 1* internet stream tcp c5ea06c0 bob imap 5827 2 / 7957 crw-rw-rw- null r bob imap 5827 3* local dgram c6357640 <-> c6357f00 bob imap 5827 4 /usr 3976421 -rw------- 0 w bob imap 5827 5 /usr 3976422 -rw------- 0 w bob imap 5827 6* local stream c644bc80 <-> c6357500
This is probably way more info that you need, but little of this means anything to me, so I don't know what is significant and what isn't.
Bob Hall
On Friday, Aug 22, 2003, at 09:08 Europe/Helsinki, Bob Hall wrote:
Root directory? I'm not sure why it would be there.. It should be in /var/run/dovecot/ directory if home directory wasn't given.. Unless the home dir was root.
Even more strange, especially if it works. I don't think your home directory is /usr either?
Maybe the "wd" is buggy? Or something else weird is going on..
On Sat, Aug 23, 2003 at 09:46:42PM +0300, Timo Sirainen wrote:
Then shouldn't it be in /root instead of /?
I mounted a linux proc file system. It doesn't have the pwd file that you mentioned was present in a Linux OS, but it did have a status file. In both cases, the UID number was the number associated with whatever account was named in the IMAP login. 1 login <account name> <password> In neither case was it UID 0.
BTW, I've got to hand it to Torvalds; the Linux status file is cooler than the BSD status file. Much easier to understand.
I haven't a clue. It's all Finnish to me, Timo. :) You're the one who knows what he's doing. I just follow orders.
Bob
On Sun, 2003-08-24 at 01:56, Bob Hall wrote:
/root? Why? /root is usually root's home directory and you're not running imap as root.
You're the one who knows what he's doing. I just follow orders.
I'm pretty sure it's just not getting the homeDirectory out of LDAP. Maybe it's related to why uidNumber/gidNumber wasn't working either. My LDAP knowledge isn't very good, but maybe it's looking into wrong schema? Did you change user_attrs or user_filter in dovecot-ldap.conf?
And you did use the same dovecot-ldap.conf file for both passdb and userdb, right?
On Sun, Aug 24, 2003 at 03:21:28AM +0300, Timo Sirainen wrote:
LDAP knowledge isn't very good, but maybe it's looking into wrong schema? Did you change user_attrs or user_filter in dovecot-ldap.conf?
Bingo. When I was originally trying to get authentication to work, I removed everything from the user_attrs setting except uid. I copied the original setting back in from the original file, and now it works fine. The raw log directory is filling up like crazy.
Timo, you've got to explain this setting. Here's the original setting: user_attrs = uid,homeDirectory,,uid,uidNumber,gidNumber ^^^^^ There's an extra uid and an extra comma, but when I remove them, authentication fails. What's going on? What's with the duplicates?
Bob
On Sunday, Aug 24, 2003, at 07:10 Europe/Helsinki, Bob Hall wrote:
Great :)
The first uid is the virtual user name that you log in with. For example user@domain.com. The second uid is the system user id which is used to lookup groups from /etc/groups file. Hmm.. This should probably be said more clearly there. You really don't want to have it if you have only virtual users.
Two commas mean simply that the value in the middle of them doesn't exist. That'd be the MAIL environment which you can use to override config file's default_mail_env. By default it doesn't exist in LDAP so it's empty.
On Wed, 2003-08-20 at 22:24, Bob Hall wrote:
Actually .. :) It doesn't override anything. If it seems that way, LDAP lookup didn't return uidNumber or gidNumber for some reason.
Now it says:
Error: ldap(username): uidNumber not set and no default given in user_global_uid
participants (3)
-
Bob Hall
-
Eric S. Johansson
-
Timo Sirainen