[dovecot/core] e6a3d0: lib-storage: Try harder to rename a corrupted mail...

Fri Nov 25 14:00:14 UTC 2016

  Branch: refs/heads/master-2.2
  Home:   https://github.com/dovecot/core
  Commit: e6a3d022b64defe98480587ee24e23180c46344a
      https://github.com/dovecot/core/commit/e6a3d022b64defe98480587ee24e23180c46344a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/list/mailbox-list-index-backend.c

  Log Message:
  -----------
  lib-storage: Try harder to rename a corrupted mailbox name to its old name.

If the old name exists, use it as a prefix for the new name. This is
especially useful when restoring autocreated mailboxes. A new mailbox
could have already been autocreated, but it's still useful to have
the broken one renamed with the same prefix, so it'll be clear that
these mailboxes should be merged.

  Commit: 39815e86872d080ae096937bc6ed818594fbf6d2
      https://github.com/dovecot/core/commit/39815e86872d080ae096937bc6ed818594fbf6d2
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-private.h
    M src/lib-index/mail-index.c
    M src/lib-index/mail-index.h
    M src/lib-index/mail-transaction-log-private.h
    M src/lib-index/mail-transaction-log.c

  Log Message:
  -----------
  lib-index: Add mail_index_set_log_rotation()

  Commit: 0882459e93d919d9608d555438c0e91ccc54467a
      https://github.com/dovecot/core/commit/0882459e93d919d9608d555438c0e91ccc54467a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/list/mailbox-list-index.c

  Log Message:
  -----------
  lib-storage: Rotate dovecot.list.index.log* more often.

The history in these files isn't as important as in mailbox indexes.
Reduce disk space usage by rotating them more often and deleting the
.log.2 more quickly.

  Commit: 3e1407670fa423633cc690d1db9cce02ea1a48fa
      https://github.com/dovecot/core/commit/3e1407670fa423633cc690d1db9cce02ea1a48fa
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-transaction-log.c

  Log Message:
  -----------
  lib-index: Compiler warning fix.

  Commit: 1868f86ec81626563ef048decb5d545181b0617e
      https://github.com/dovecot/core/commit/1868f86ec81626563ef048decb5d545181b0617e
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib/macros.h

  Log Message:
  -----------
  lib: Add ATTR_UNSIGNED_WRAPS for disabling clang -fsanitize=integer

  Commit: 7acacaedd738612b15400d11ef920d8c2ffa91cc
      https://github.com/dovecot/core/commit/7acacaedd738612b15400d11ef920d8c2ffa91cc
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-map.c

  Log Message:
  -----------
  lib-index: mail_index_map_register_ext(ext_offset=-1) now sets hdr_offset=-1

It shouldn't make any difference, but this is a less confusing value.

  Commit: 5791bc28bff63f4a2794ba05fb647f7368ea8943
      https://github.com/dovecot/core/commit/5791bc28bff63f4a2794ba05fb647f7368ea8943
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib/md4.c
    M src/lib/md5.c
    M src/lib/sha1.c
    M src/lib/sha2.c

  Log Message:
  -----------
  lib: Mark md4/md5/sha1/sha2 code with ATTR_UNSIGNED_WRAPS

  Commit: 1215cceac448a35ab5fb90c10341dca6500e0b36
      https://github.com/dovecot/core/commit/1215cceac448a35ab5fb90c10341dca6500e0b36
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-mail/message-address.c
    M src/lib-mail/message-snippet.c
    M src/lib/sha3.c

  Log Message:
  -----------
  global: Avoid loops unnecessarily decreasing below zero.

Avoids complains from clang -fsanitize=integer

  Commit: 8dc068f46bfea9a00c69f648724a970fd597cb1f
      https://github.com/dovecot/core/commit/8dc068f46bfea9a00c69f648724a970fd597cb1f
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/indexer/master-connection.c
    M src/lib-mail/test-istream-header-filter.c
    M src/lib-storage/index/mbox/istream-raw-mbox.c

  Log Message:
  -----------
  global: Avoid unnecessary unsigned integer wraps.

Avoids complains from clang -fsanitize=integer

  Commit: 56c5e1c7add26c3e10d116e4839f853107dfd022
      https://github.com/dovecot/core/commit/56c5e1c7add26c3e10d116e4839f853107dfd022
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-dcrypt/test-stream.c
    M src/lib-index/mail-cache-lookup.c
    M src/lib-storage/index/dbox-multi/mdbox-storage.c
    M src/lib-storage/index/index-mailbox-size.c
    M src/lib/buffer.c
    M src/lib/istream.c

  Log Message:
  -----------
  global: Code cleanup - avoid passing NULL to functions with non-null parameter

  Commit: 62a17f66bc2c11fef596345d67735e0dd10f2ff2
      https://github.com/dovecot/core/commit/62a17f66bc2c11fef596345d67735e0dd10f2ff2
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib/buffer.h
    M src/lib/test-buffer.c

  Log Message:
  -----------
  lib: Clarify that buffer_write() zero-fills buffer when writing past its size

  Commit: 582003e3dd2ccc7846c4ee4a8eae054cb2a5339f
      https://github.com/dovecot/core/commit/582003e3dd2ccc7846c4ee4a8eae054cb2a5339f
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-mail/mail-html2text.c
    M src/lib-mail/test-mail-html2text.c

  Log Message:
  -----------
  lib-mail: Fix assert-crash in mail_html2text_more() with invalid input.

parse_data() continues forward thinking that it might have valid input,
until it has enough data and realizes that there's nothing valid. This
triggers:

Panic: file mail-html2text.c: line 312 (mail_html2text_more): assertion failed: (pos >= buf_orig_size)

  Commit: 6bde01d39e20e8693329be2c95b841ae40200451
      https://github.com/dovecot/core/commit/6bde01d39e20e8693329be2c95b841ae40200451
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-mail/test-mail-html2text.c

  Log Message:
  -----------
  lib-mail: Add randomness test to test-mail-html2text

  Commit: 34722782166836b6b941b4bf07691225805007cb
      https://github.com/dovecot/core/commit/34722782166836b6b941b4bf07691225805007cb
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-transaction-log.c

  Log Message:
  -----------
  lib-index: Compiler warning fix

  Commit: 67bb90dd1c527e40d0fede2923bbea8834adcc67
      https://github.com/dovecot/core/commit/67bb90dd1c527e40d0fede2923bbea8834adcc67
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/mail-namespace.c

  Log Message:
  -----------
  lib-storage: Fail if no namespaces have list=yes

The previous check allowed all namespaces to have list=children. This
crashed later on in mail_namespaces_get_root_sep(), because it couldn't
find any list=yes namespaces.

  Commit: 25a1ca4ee3784c6b21a769d9921cca6a08ba1f1e
      https://github.com/dovecot/core/commit/25a1ca4ee3784c6b21a769d9921cca6a08ba1f1e
  Author: Aki Tuomi <aki.tuomi at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-dcrypt/dcrypt.c

  Log Message:
  -----------
  lib-dcrypt: Add assert that vfs is initialized

  Commit: 582971057de6e37b02cf470dde5f7bba0c5dc162
      https://github.com/dovecot/core/commit/582971057de6e37b02cf470dde5f7bba0c5dc162
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-fsck.c

  Log Message:
  -----------
  lib-index: fsck: Fix log_file_head|tail_offset properly

  Commit: a37771a18e3258791f4cad06669fae2ff228a2ac
      https://github.com/dovecot/core/commit/a37771a18e3258791f4cad06669fae2ff228a2ac
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-map-read.c

  Log Message:
  -----------
  lib-index: If index open fails with fsck, retry opening once.

The fsck should have fixed the log offsets and open should work.

  Commit: 745c851f59ca66b4bd7e8b23c2ca413709c31ee2
      https://github.com/dovecot/core/commit/745c851f59ca66b4bd7e8b23c2ca413709c31ee2
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/list/mailbox-list-index-sync.c
    M src/lib-storage/list/mailbox-list-index.c
    M src/lib-storage/list/mailbox-list-index.h

  Log Message:
  -----------
  lib-storage: Rename mailbox_list_index.corrupted to corrupted_names_or_parents

Makes it clearer what exactly the flag means.

  Commit: ead5dced72859c1fb1579cf790a305b2ce697738
      https://github.com/dovecot/core/commit/ead5dced72859c1fb1579cf790a305b2ce697738
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-fsck.c
    M src/lib-index/mail-index-sync-update.c
    M src/lib-index/mail-index.h

  Log Message:
  -----------
  lib-index: fsck now adds MAIL_INDEX_HDR_FLAG_FSCKD to header.

It can only be removed by an explicit header update.

  Commit: 219dd30e1be110e390afe5a573b3276d24f958e8
      https://github.com/dovecot/core/commit/219dd30e1be110e390afe5a573b3276d24f958e8
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-index/mail-index-transaction-update.c
    M src/lib-index/mail-index.h

  Log Message:
  -----------
  lib-index: Add mail_index_unset_fscked()

This can be used to easily remove MAIL_INDEX_HDR_FLAG_FSCKD. It takes a
transaction parameter instead of sync_ctx because some index rebuilds
are done with a separate transaction while the sync_ctx is rolled back.

  Commit: 243d0d9f14c7a89692fbad60b102397aa793ccf1
      https://github.com/dovecot/core/commit/243d0d9f14c7a89692fbad60b102397aa793ccf1
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/fail-mail-storage.c
    M src/lib-storage/index/cydir/cydir-storage.c
    M src/lib-storage/index/dbox-multi/mdbox-deleted-storage.c
    M src/lib-storage/index/dbox-multi/mdbox-storage.c
    M src/lib-storage/index/dbox-single/sdbox-storage.c
    M src/lib-storage/index/imapc/imapc-storage.c
    M src/lib-storage/index/maildir/maildir-storage.c
    M src/lib-storage/index/mbox/mbox-storage.c
    M src/lib-storage/index/pop3c/pop3c-storage.c
    M src/lib-storage/index/raw/raw-storage.c
    M src/lib-storage/index/shared/shared-storage.c
    M src/lib-storage/mail-storage-private.h
    M src/plugins/virtual/virtual-storage.c

  Log Message:
  -----------
  lib-storage: Add mail_storage.list_index_corrupted()

The actual implementation is in the next commit.

  Commit: a81c40fb46f6d447c1edc858be905e6626dae97a
      https://github.com/dovecot/core/commit/a81c40fb46f6d447c1edc858be905e6626dae97a
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/list/mailbox-list-index-notify.c
    M src/lib-storage/list/mailbox-list-index-sync.c
    M src/lib-storage/list/mailbox-list-index.c
    M src/lib-storage/list/mailbox-list-index.h

  Log Message:
  -----------
  lib-storge: Call mail_storage.list_index_corrupted() when needed

The callback is called whenever mailbox list index appears to be corrupted
with LAYOUT=index. The storage is responsible for adding to the index any
mailboxes that are missing.

  Commit: 265b58526784d4119678f15489d7b312185a2bd2
      https://github.com/dovecot/core/commit/265b58526784d4119678f15489d7b312185a2bd2
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/index/dbox-multi/mdbox-map.c
    M src/lib-storage/index/dbox-multi/mdbox-map.h
    M src/lib-storage/index/dbox-multi/mdbox-storage-rebuild.c
    M src/lib-storage/index/dbox-multi/mdbox-sync.c

  Log Message:
  -----------
  mdbox: Rebuild index after it's been fsck'd

  Commit: 0805d0f892892230139787e0cb41da40339b914f
      https://github.com/dovecot/core/commit/0805d0f892892230139787e0cb41da40339b914f
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M doc/mkcert.sh

  Log Message:
  -----------
  mkcert.sh: Use umask to create key file as 0600

Fixes a race condition between creation of the file and a later chmod.
This script was mostly meant as an example though, and not really for
production use. Especially because it generates self-signed certs.
CVE-2016-4983

  Commit: 162b8a3532b8ece53bd602828e5b05b775684af8
      https://github.com/dovecot/core/commit/162b8a3532b8ece53bd602828e5b05b775684af8
  Author: Timo Sirainen <timo.sirainen at dovecot.fi>
  Date:   2016-11-25 (Fri, 25 Nov 2016)

  Changed paths:
    M src/lib-storage/list/mailbox-list-index.c

  Log Message:
  -----------
  lib-storage: Fix error handling in mailbox_list_index_refresh_force()

Broken by recent changes.

Compare: https://github.com/dovecot/core/compare/565ae33cb556...162b8a3532b8