- dovecot - dovecot.org

[Dovecot] Some benchmarking
by Timo Sirainen 11 Aug '03

11 Aug '03

So I finally got the new mail caching code fully working in normal conditions. Index rebuilding and some error correction still needs a bit work. mbox is also still broken. Anyway, I was mostly interested how fast Dovecot now works with my 367000 LKML mails, but thought I'd have to get something to compare the results against, so here's Dovecot vs. Courier. Results against older Dovecot would have been pretty boring, the older Dovecot could never perform threading without opening all the files and reading References-headers from them. Sorting would could have been done from cache if IMAP ENVELOPEs were cached, I'd estimate maybe 2-3x larger user CPU usage with it and a _lot_ larger cache file since it would have stored from, to, etc. fields in it. Results against Cyrus would be interesting. I'll probably do those sometimes later too.. Last I checked, Cyrus was very slow at inserting the mails into it's database, it took .. was it 1-3 hours to insert all those mails, while a simple copying took maybe 20 mins. Anyway, Cyrus's caching is pretty dummy as well so I'd expect Dovecot to be many times faster in threading and likely somewhat faster in sorting.

3 2

[Dovecot] Direct RADIUS support, not via PAM
by Jethro R Binks 08 Aug '03

08 Aug '03

Hi, I'm currently testing dovecot to authenticate user passwords against a radius server, via pam_radius_auth, and that seems to work fine. However, I am running other software (Exim MTA) that also authenticates via radius, using the radiusclient library from: http://www.mcs.de/~lf/radius/ Is there any chance that this library could be used by Dovecot as a further authentication option, and save me having two separate radius configurations for two different (but related) pieces of software, and the need to install pam_radius_auth, which I don't otherwise use? Jethro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jethro R Binks Computing Officer, IT Services University Of Strathclyde, Glasgow, UK

2 1

[Dovecot] Slow bulk deleting
by Leslie Viljoen 07 Aug '03

07 Aug '03

Hi! I am finding Dovecot very slow to delete large numbers of messages. IS this a known problem? Leslie

2 2

[Dovecot] compile with dietlibc
by Jens Gutzeit 07 Aug '03

07 Aug '03

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi all, I'm used to test the software which I use with dietlibc, just to see if it could run without the bloat of glibc. While compiling dovecot 0.99.10 I got these errors: In file included from lib.h:20, from alarm-hup.c:26: compat.h:30: conflicting types for `uint_fast32_t' /opt/diet/include/stdint.h:25: previous declaration of `uint_fast32_t' I've solved this in a dirty way, but is this something which could be fixed in the future? Thanks in advance, Jens Gutzeit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQE/L/etLPHiyaqJLscRAuNGAJ9jlOQzx86T+NS9h7FfjoDi86R4ygCgs6gb 1XyNB93ybwiG4gFatz797qo= =WZod -----END PGP SIGNATURE-----

3 3

[Dovecot] thinking seriously about converting
by Eric S. Johansson 06 Aug '03

06 Aug '03

to dovecot from UW imap. I think I will go all the way to maildir while I'm at it. I was having performance problems with mbox format files and converted to mbx but it's not really yielding a huge performance gain. Deletes and moves are still rather slow at times. my mailboxes are typically 2000 to 5000 messages. I've scanned the mailing lists a bit and picked up some hints on the conversion. Any bits of advice before I convert? Since it's only my own account I'm converting (my wife likes mutt), I'm torn between two different ways of converting the mailboxes. First is to convert all of the mbx format mailboxes back to mbox. Then convert from mbox to maildir. Second is running both imap server simultaneously (obviously one on a different port number) and running some form of a tool to read from one imap server and write to the other. any better ideas? many thanks ---eric

2 1

[Dovecot] corrupt mbox, mailboxes not found, and message read status
by Jared 04 Aug '03

04 Aug '03

I'm running dovecot 0.99.10-0.rc2 deb packages for Debian Woody from braincells.com. My old environment was a server running UW-IMAP for IMAP and POP services. We are still using mbox for all mailboxes. I use IMP for web based access. Forgive me if these issues have already been discussed or fixed in cvs. Any additional pointers to this info. would be appreciated. I have noticed a few issues. ---- I've seen a whole slew of entries like the following: pop3(username): Error indexing mbox file /home/username/mail/inbox: LF not found where expected According to this mailing list post from June 16, 2003: http://dovecot.procontrol.fi/list/dovecot/2003-June/001668.html it may be because of adjacent From mail headers in a message. I've had one user that this was not the case. They said they were deleting lots of messages in IMP and when he went back to his inbox, it was empty. When I looked at the mbox file, a message had been truncated. When I deleted the imap index and removed that truncated message, the mailbox became visible again. Has the adjacent From issue been fixed? Is there a way to find out more specifically what part of the mbox file dovecot is unhappy about? ie. a line number. ---- Inside IMP several users have set up a filter to send messages marked as spam by SpamAssassin into a spam folder. When they log-in or click the filter icon, the filter runs. However, now when the the filter runs, it reports: "Problem with filter rule 1: folder does not exist: mail/spam". However, the spam folder is still there, and when I open that folder in IMP, it opens just fine. I've already changed IMP's folder prefix from 'mail/' to ''. I've also tried deleting the imap index. The message still shows up and the filter fails to run. ---- With the old set-up, when I check mail with a pop3 client, the UW pop server modifies the Status and/or X-Keywords headers for the message. When the mailbox is viewed with an IMAP client such as IMP, messages that have already been downloaded via pop are marked as read. This doesn't seem to happen with dovecot. I found a Oct 19 2003 thread ( http://dovecot.procontrol.fi/list/dovecot/2002-October/000774.html ) on the mailing list where Timo said he was intending to use the X-Keywords trick. Did this ever get implemented? ---- Thanks, -jared

5 6

[Dovecot] why wont /usr/pkg/libexec/dovecot/imap see my Maildir
by Charlie Allom 04 Aug '03

04 Aug '03

When I invoke /usr/pkg/libexec/dovecot/imap from the command loine on the mail host, it logs me in, but won't see my emails: 13:22 yeled@lazy:~$ /usr/pkg/libexec/dovecot/imap * PREAUTH [CAPABILITY IMAP4rev1 SORT THREAD=REFERENCES MULTIAPPEND * UNSELECT LITERAL+ IDLE CHILDREN LISTEXT LIST-SUBSCRIBED] Logged in * as yeled 0 select inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] * Flags permitted. * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1059959635] UIDs valid * OK [UIDNEXT 1] Predicted next UID 0 OK [READ-WRITE] Select completed. I know I have many emails in there, and I can see them when I login via telnet localhost 443. Any tips? would be nice to use fetchmail with ssh here. C. -- charlie(a)rubberduck.com - Melbourne, Australia http://rubberduck.com/~yeled/ PGP: 0x14AA7941 || finger yeled(a)lazy.spodder.com

2 2

[Dovecot] Confusion
by Alejandro Martinez 02 Aug '03

02 Aug '03

Hi, I have a great confusion about mbox, maildir. I use a passwd-file to authenticate users, because I want to use non local users. I have a test user in that file that is this: test:VGRAafc3NHJwI:502:100:Usuario Virtual:/var/spool/mail/virtual/test:/no/shell I create inside /var/spool/mail/virtual/test a folder Maildir and inside Maildir /var/spool/mail/virtual/test/Maildir/cur/ /var/spool/mail/virtual/test/Maildir/new/ /var/spool/mail/virtual/test/Maildir/tmp/ I get an error when I login to the server admin@slk90 /$ telnet localhost 143 Trying 127.0.0.1... Connected to localhost.localdomain. Escape character is '^]'. * OK dovecot ready. . login test test . OK Logged in. Connection closed by foreign host. The /var/spool/mail/error is this "Aug 2 08:46:46 slk90 imap(test): Failed to create storage with data: /var/spool/mail/virtual/test/Maildir" I've tried setting default_mail_env = maildir:~/Maildir and mbox:~/Maildir, but nothing work. Any help ? Regards. --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com) Version: 6.0.506 / Virus Database: 303 - Release Date: 01/08/2003

2 1

RE: [Dovecot] corrupt mbox, mailboxes not found, and message rea d status
by Maikel Verheijen 31 Jul '03

31 Jul '03

[SNIP] > http://dovecot.procontrol.fi/list/dovecot/2002> -October/000774.html ) > > on the mailing list where Timo said he was intending to use the > > X-Keywords trick. Did this ever get implemented? > > X-Keywords have been used for a long time now. It's possible > to set \Seen flag for messages downloaded with POP3, but I > don't know if it's worth the trouble. Who uses POP3 and IMAP > mixed anyway? > A lot of our customers, since we only supported pop3 in the past, and our new webmail uses imap. :) And a lot of people don't click on "This is an imap server" when creating their account in for instance outlook-express. Kind regards, Maikel Verheijen.

3 2

[Dovecot] New index file code
by Timo Sirainen 31 Jul '03

31 Jul '03

I just had to write this even while it's not just yet in CVS, but it's only a few fixes away from being committed.. My year long dream has finally came true :) [Summary: Next release will have REALLY kickass indexes] Previously the biggest problem with caching message data in indexes was that we had to do it while syncing the mailbox, before client even saw those messages. This caused index rebuilds to be very slow since it had to go through all mails and cache all of them, even if client never would access them. Alternative was to not cache anything at all, but that wasn't really good idea either. Now Dovecot caches the data when client is actually requesting it. That way it will never do any extra work for caching something that client isn't currently interested in. There's still the anticipatory caching for data that client is likely to be interested in later, but that is done only when it doesn't cause extra disk I/O. The reason why it took this long was because it was previously too difficult with read/write locking that indexes required. I couldn't have relied on it working since changing from read lock -> write lock could deadlock and dropping the lock in the middle would have caused even more problems. So, what I did was to rewrite the cache file handling completely: - It doesn't require any locking to read from it. - It will be NFS-safe without being too costly. - The file format itself is now architecture-independent. - Uses less space than before - OpenBSD support is now worse than before, even worse than NFS-support. They should get that unified cache done. Cached data is also selected better: - Specifically requested message headers are cached (FETCH, SEARCH, SORT, THREAD, etc.) - IMAP ENVELOPE isn't treated in any special way anymore. It's treated just as if you had requested HEADER.FIELDS (Date Subject From ...etc.). - Maildir filenames aren't constantly updated in the cache file anymore. This takes more memory now, but should reduce disk I/O. There's a few more tweaks that I'll probably add later: - Maildir: When compressing cache file we update mail's filename to current one so if message's flags aren't changed often, it won't use any extra memory either. - A normal text/plain message doesn't need to have it's IMAP BODYSTRUCTURE cached. It only needs a single bit set in flags. Should reduce used space quite a lot. - Message with only one body part doesn't need to have it's body structure stored at all since it's mostly just useful for fetching body parts. Plus the body structure could be generated if message sizes were known. All this work was only for index cache file (ex. .data file). There's still the main .imap.index file and modify log files. Modify log should be easy to make NFS-safe - I'm currently using some stupid file locking to figure out when the file is safe to overwrite, but I should have simply replaced the old file with rename(). That will solve some other problems as well as make the code simpler. It should be possible to be made read-lockless as well. The main index file is more problematic though. It contains a lot of changing fields in the header, such as number of messages, number of seen messages, etc. I can't think of any way to make it safe to read these fields without locks (my previous ideas didn't actually work). Lockless reads are pretty much a must for scalable NFS-safety. I think it could be done by simply removing all the constantly changing headers. If you want to know how many seen messages there are, just read all the records in the file and count them. Expunging would be done by rewriting the file and rename()ing it over the old file. Appending new messages is a bit tricky to do safely, but I think I know how to do that too.. read/write locks allow the changing headers, but there's locking contention problems with shared mailboxes.. I think I'll make it optional how to do this. Oh, and I also thought how indexes would work with shared mailboxes. You could use one shared cache file, but each user would have their own main index and modify log. That would allow storing per-user flags in the index file and also expunge (hide) messages by removing them from user's index.

1 0