lines in it) will eat an inode, and a block (4k here)
mdbox is more compact on this way
Another difference: sdbox removes the message, mdbox does not : a
single
metadata update is performed, which may be packed with others if many
files are deleted at once
That said, I do not have experience with dovecot + cephfs, nor have
made
tests for sdbox vs mdbox
However, and this is a bit out of topic, I recommend you look at the
following dovecot's features (if not already done), as they are awesome
and will help you a lot:
I'm sending this message to both dovecot and ceph-users ML so please
don't
mind if something seems too obvious for you.
Hi,
I have a question for both dovecot and ceph lists and below I'll
explain
what's going on.
Regarding dbox format (https://wiki2.dovecot.org/MailboxFormat/dbox),
when
using sdbox, a new file is stored for each email message.
When using mdbox, multiple messages are appended to a single file
until
it
reaches/passes the rotate limit.
I would like to understand better how the mdbox format impacts on IO
performance.
I think it's generally expected that fewer larger file translate to
less
IO
and more troughput when compared to more small files, but how does
dovecot
handle that with mdbox?
If dovecot does flush data to storage upon each and every new email is
arrived and appended to the corresponding file, would that mean that
it
generate the same ammount of IO as it would do with one file per
message?
Also, if using mdbox many messages will be appended to a said file
before a
new file is created. That should mean that a file descriptor is kept
open
for sometime by dovecot process.
Using cephfs as backend, how would this impact cluster performance
regarding MDS caps and inodes cached when files from thousands of
users
are
opened and appended all over?
I would like to understand this better.
Why?
We are a small Business Email Hosting provider with bare metal, self
hosted
systems, using dovecot for servicing mailboxes and cephfs for email
storage.
We are currently working on dovecot and storage redesign to be in
production ASAP. The main objective is to serve more users with better
performance, high availability and scalability.
- high availability and load balancing is extremely important to us *
On our current model, we're using mdbox format with dovecot, having
dovecot's INDEXes stored in a replicated pool of SSDs, and messages
stored
in a replicated pool of HDDs (under a Cache Tier with a pool of SSDs).
All using cephfs / filestore backend.
Currently there are 3 clusters running dovecot 2.2.34 and ceph Jewel
(10.2.9-4).
- ~25K users from a few thousands of domains per cluster
- ~25TB of email data per cluster
- ~70GB of dovecot INDEX [meta]data per cluster
- ~100MB of cephfs metadata per cluster
Our goal is to build a single ceph cluster for storage that could
expand
in
capacity, be highly available and perform well enough. I know, that's
what
everyone wants.
Cephfs is an important choise because:
- there can be multiple mountpoints, thus multiple dovecot instances
on
different hosts
- the same storage backend is used for all dovecot instances
- no need of sharding domains
- dovecot is easily load balanced (with director sticking users to
the
same dovecot backend)
On the upcoming upgrade we intent to:
- upgrade ceph to 12.X (Luminous)
- drop the SSD Cache Tier (because it's deprecated)
- use bluestore engine
I was said on freenode/#dovecot that there are many cases where SDBOX
would
perform better with NFS sharing.
In case of cephfs, at first, I wouldn't think that would be true
because
more files == more generated IO, but thinking about what I said in the
beginning regarding sdbox vs mdbox that could be wrong.
Any thoughts will be highlt appreciated.
Regards,
Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com