From pigeonhole at rename-it.nl Sun Feb 1 12:23:24 2015 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Sun, 01 Feb 2015 13:23:24 +0100 Subject: dovecot-2.2-pigeonhole: Updated TODO. Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/012ce8594501 changeset: 1987:012ce8594501 user: Stephan Bosch date: Sun Feb 01 13:23:15 2015 +0100 description: Updated TODO. diffstat: TODO | 45 ++++++++++++++++++++++----------------------- 1 files changed, 22 insertions(+), 23 deletions(-) diffs (75 lines): diff -r cd8194a2469e -r 012ce8594501 TODO --- a/TODO Fri Jan 16 18:25:51 2015 +0100 +++ b/TODO Sun Feb 01 13:23:15 2015 +0100 @@ -1,32 +1,34 @@ Current activities: +* Rework string matching: + - Add support for stream matching for handling large values, e.g. from the + body extension. + - Improve efficiency :matches and :contains match types. +* Build proper comparator support: + - Add normalize() method to comparators to normalize the string before + matching (for efficiency). + - Allow for the existence of dynamic comparators (i.e. specified by + variables). + - Implement comparator-i;unicode-casemap. + +Parallel plugin-based efforts (on hold at the moment): + +* Implement enotify xmpp method as a plugin. + +Next (mostly in order of descending priority/precedence): + +* Properly implement Sieve internationalization support (utf-8 handling), + currently it is not complete: + - Make this implementation fully conform section 2.7.2 of RFC5228 (Comparisons + Across Character Sets). + - Verify validity of utf8 where necessary. * Finish LDAP Sieve script storage for read-only access. - Consolidate LDAP connections when more than a single Sieve script must be loaded from different storages linked to the same LDAP server. - Adjust Sieve script API to support asynchronous script retrieval to retrieve scripts in parallel when possible. - -Parallel plugin-based efforts (on hold at the moment): - -* Implement enotify xmpp method as a plugin. -* Implement metadata and servermetadata extensions as a plugin. - - Update to native Dovecot metadata implementation once it is created. - -Next (mostly in order of descending priority/precedence): - -* Implement index extension * Implement message modification and extraction API in order to: - Implement replace, enclose, foreverypart, mime and extracttext extensions -* Add normalize() method to comparators to normalize the string before matching - (for efficiency). -* Properly implement Sieve internationalization support (utf-8 handling), - currently it is not complete: - - Make this implementation fully conform section 2.7.2 of RFC5228 (Comparisons - Across Character Sets). - - Verify validity of utf8 where necessary. - - Implement comparator-i;unicode-casemap. -* Add support for stream matching for handling large values, e.g. from the body - extension. * Clean up doveadm sync implementation: - Mostly comments referring specifically to the file storage type. - API can be polished a lot as well. @@ -49,8 +51,6 @@ vacation command's :addresses list. * Fix remaining RFC deviations: - Fix issues listed in doc/rfc/RFC-questions.txt based on answers - - Allow for the existence of dynamic comparators (i.e. specified by - variables). - Verify outgoing mail addresses at runtime when necessary (e.g. after variables substitution) - Improve handling of invalid addresses in headers (requires Dovecot changes) @@ -87,7 +87,6 @@ * Provide a solution for mail_get_headers_utf8 reparsing the whole message each time it is called (header and address test; Timo might provide solution from within Dovecot) -* Use lib/str-find.h for :contains and :matches match types * Warn during compile if using non-existent folders. * Implement IMAP plugin for IMAPSieve support: From dovecot at dovecot.org Mon Feb 2 08:27:00 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 02 Feb 2015 08:27:00 +0000 Subject: dovecot-2.2: lib: array - two helpers to avoid wasteful array_count Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/b0ac11ef7f51 changeset: 18211:b0ac11ef7f51 user: Phil Carmody date: Mon Feb 02 10:24:42 2015 +0200 description: lib: array - two helpers to avoid wasteful array_count There's no need to dereference array->element_size and perform a division when all you care about is the boolean isempty/nonempty predicate: $ git grep 'array_count(.*) > 0' | wc -l 77 $ git grep 'array_count(.*) == 0' | wc -l 95 Changing 6 of them has the following impact on the code: $ size src/lib-imap-client/imapc-connection.o text data bss dec hex filename 20879 0 4 20883 5193 src/lib-imap-client/imapc-connection.o $ size src/lib-imap-client/imapc-connection.o text data bss dec hex filename 20796 0 4 20800 5140 src/lib-imap-client/imapc-connection.o Signed-off-by: Phil Carmody diffstat: src/lib/array.h | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-) diffs (15 lines): diff -r ad04a5e6a766 -r b0ac11ef7f51 src/lib/array.h --- a/src/lib/array.h Sat Jan 31 01:10:56 2015 +0200 +++ b/src/lib/array.h Mon Feb 02 10:24:42 2015 +0200 @@ -156,6 +156,11 @@ } #define array_count(array) \ array_count_i(&(array)->arr) +/* No need for the real count if all we're doing is comparing againts 0 */ +#define array_is_empty(array) \ + ((array)->arr.buffer->used == 0) +#define array_not_empty(array) \ + ((array)->arr.buffer->used > 0) static inline void array_append_i(struct array *array, const void *data, unsigned int count) From dovecot at dovecot.org Mon Feb 2 08:27:01 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 02 Feb 2015 08:27:01 +0000 Subject: dovecot-2.2: lib: array - test count/isempty/nonepty Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/90e1829ba80e changeset: 18212:90e1829ba80e user: Phil Carmody date: Mon Feb 02 10:26:44 2015 +0200 description: lib: array - test count/isempty/nonepty Signed-off-by: Phil Carmody diffstat: src/lib/test-array.c | 20 ++++++++++++++++++++ 1 files changed, 20 insertions(+), 0 deletions(-) diffs (37 lines): diff -r b0ac11ef7f51 -r 90e1829ba80e src/lib/test-array.c --- a/src/lib/test-array.c Mon Feb 02 10:24:42 2015 +0200 +++ b/src/lib/test-array.c Mon Feb 02 10:26:44 2015 +0200 @@ -9,6 +9,25 @@ unsigned int a, b, c; }; +static void test_array_count(void) +{ + ARRAY(struct foo) foos; + struct foo nfoo; + + test_begin("array count/empty"); + t_array_init(&foos, 32); + + test_assert(array_count(&foos) == 0); + test_assert(array_is_empty(&foos)); + test_assert(!array_not_empty(&foos)); + nfoo.a = nfoo.b = nfoo.c = 9; + array_append(&foos, &nfoo, 1); + test_assert(array_count(&foos) == 1); + test_assert(!array_is_empty(&foos)); + test_assert(array_not_empty(&foos)); + + test_end(); +} static void test_array_foreach(void) { ARRAY(struct foo) foos; @@ -210,6 +229,7 @@ void test_array(void) { + test_array_count(); test_array_foreach(); test_array_reverse(); test_array_cmp(); From dovecot at dovecot.org Mon Feb 2 21:48:47 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 02 Feb 2015 21:48:47 +0000 Subject: dovecot-2.2: lib-http: client: Fixed double unref of request whe... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/5205f3bd1a27 changeset: 18213:5205f3bd1a27 user: Stephan Bosch date: Mon Feb 02 23:48:30 2015 +0200 description: lib-http: client: Fixed double unref of request when error occurs during http_client_request_try_retry(). Much like when the request is first submitted, any errors that occur while attempting a retry from within the callback are now delayed in a zero timer. diffstat: src/lib-http/http-client-request.c | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-) diffs (19 lines): diff -r 90e1829ba80e -r 5205f3bd1a27 src/lib-http/http-client-request.c --- a/src/lib-http/http-client-request.c Mon Feb 02 10:26:44 2015 +0200 +++ b/src/lib-http/http-client-request.c Mon Feb 02 23:48:30 2015 +0200 @@ -912,10 +912,11 @@ if (req->queue != NULL) http_client_queue_drop_request(req->queue, req); - if (!req->submitted) { - /* we're still in http_client_request_submit(). delay - reporting the error, so the caller doesn't have to handle - immediate callbacks. */ + if (!req->submitted || + req->state == HTTP_REQUEST_STATE_GOT_RESPONSE) { + /* we're still in http_client_request_submit() or in the callback + during a retry attempt. delay reporting the error, so the caller + doesn't have to handle immediate or nested callbacks. */ i_assert(req->delayed_error == NULL); req->delayed_error = p_strdup(req->pool, error); req->delayed_error_status = status; From dovecot at dovecot.org Tue Feb 3 08:15:58 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 03 Feb 2015 08:15:58 +0000 Subject: dovecot-2.2: lib-imap-client: Implemented support for GMail [THR... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6c2ea1d6ab58 changeset: 18214:6c2ea1d6ab58 user: Timo Sirainen date: Tue Feb 03 10:15:38 2015 +0200 description: lib-imap-client: Implemented support for GMail [THROTTLED] resp-text-code. If we receive it, start throttling future commands by waiting exponentially longer until we no longer receive [THROTTLED]. Max wait time is currently 16 seconds. diffstat: src/lib-imap-client/imapc-connection.c | 59 ++++++++++++++++++++++++++++++++++ 1 files changed, 59 insertions(+), 0 deletions(-) diffs (125 lines): diff -r 5205f3bd1a27 -r 6c2ea1d6ab58 src/lib-imap-client/imapc-connection.c --- a/src/lib-imap-client/imapc-connection.c Mon Feb 02 23:48:30 2015 +0200 +++ b/src/lib-imap-client/imapc-connection.c Tue Feb 03 10:15:38 2015 +0200 @@ -8,6 +8,7 @@ #include "base64.h" #include "write-full.h" #include "str.h" +#include "time-util.h" #include "dns-lookup.h" #include "dsasl-client.h" #include "iostream-rawlog.h" @@ -24,6 +25,10 @@ #define IMAPC_COMMAND_STATE_AUTHENTICATE_CONTINUE 10000 #define IMAPC_MAX_INLINE_LITERAL_SIZE (1024*32) +/* max seconds to wait after receiving [THROTTLED] as + 2^IMAPC_THROTTLE_COUNTER_MAX_EXP */ +#define IMAPC_THROTTLE_COUNTER_MAX_EXP 4 + enum imapc_input_state { IMAPC_INPUT_STATE_NONE = 0, IMAPC_INPUT_STATE_PLUS, @@ -116,6 +121,11 @@ struct imapc_connection_literal literal; ARRAY(struct imapc_arg_file) literal_files; + unsigned int throttle_counter; + bool throttle_pending; + struct timeval throttle_end_timeval; + struct timeout *to_throttle; + unsigned int idling:1; unsigned int idle_stopping:1; unsigned int idle_plus_waiting:1; @@ -190,6 +200,8 @@ conn->io = io_loop_move_io(&conn->io); if (conn->to != NULL) conn->to = io_loop_move_timeout(&conn->to); + if (conn->to_throttle != NULL) + conn->to_throttle = io_loop_move_timeout(&conn->to_throttle); if (conn->output != NULL) o_stream_switch_ioloop(conn->output); if (conn->dns_lookup != NULL) @@ -373,6 +385,8 @@ timeout_remove(&conn->to); if (conn->to_output != NULL) timeout_remove(&conn->to_output); + if (conn->to_throttle != NULL) + timeout_remove(&conn->to_throttle); if (conn->parser != NULL) imap_parser_unref(&conn->parser); if (conn->io != NULL) @@ -611,6 +625,15 @@ conn->selecting_box = NULL; } } + if (strcasecmp(key, "THROTTLED") == 0 && !conn->throttle_pending) { + /* GMail throttling - start slowing down commands. */ + conn->throttle_end_timeval = ioloop_timeval; + timeval_add_msecs(&conn->throttle_end_timeval, + (1U << conn->throttle_counter) * 1000); + conn->throttle_pending = TRUE; + if (conn->throttle_counter < IMAPC_THROTTLE_COUNTER_MAX_EXP) + conn->throttle_counter++; + } return 0; } @@ -1157,6 +1180,13 @@ } else { reply.text_without_resp = reply.text_full; } + if (!conn->throttle_pending && + timeval_cmp(&ioloop_timeval, &conn->throttle_end_timeval) >= 0) { + /* tagged reply without [THROTTLED] and it was received after + the throttling ended. we can completely reset the throttling + state now */ + conn->throttle_counter = 0; + } /* find the command. it's either the first command in send queue (literal failed) or somewhere in wait list. */ @@ -1740,6 +1770,32 @@ } } +static bool imapc_connection_is_throttled(struct imapc_connection *conn) +{ + if (conn->to_throttle != NULL) + timeout_remove(&conn->to_throttle); + + if (conn->throttle_counter == 0) { + /* we haven't received [THROTTLED] recently */ + return FALSE; + } + if (array_count(&conn->cmd_wait_list) > 0) { + /* wait until we have received the existing commands' tagged + replies to see if we're still throttled */ + return TRUE; + } + if (timeval_cmp(&ioloop_timeval, &conn->throttle_end_timeval) > 0) { + /* we reached the throttle timeout - send the next command */ + conn->throttle_pending = FALSE; + return FALSE; + } + + /* we're still being throttled - wait for it to end */ + conn->to_throttle = timeout_add_absolute(&conn->throttle_end_timeval, + imapc_command_send_more, conn); + return TRUE; +} + static void imapc_command_send_more(struct imapc_connection *conn) { struct imapc_command *const *cmds, *cmd; @@ -1748,6 +1804,9 @@ unsigned int count, seek_pos, start_pos, end_pos, size; int ret; + if (imapc_connection_is_throttled(conn)) + return; + cmds = array_get(&conn->cmd_send_queue, &count); if (count == 0) return; From dovecot at dovecot.org Tue Feb 3 16:33:40 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 03 Feb 2015 16:33:40 +0000 Subject: dovecot-2.2: imapc: Added imapc_features=search support for send... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/405959839f48 changeset: 18215:405959839f48 user: Timo Sirainen date: Tue Feb 03 18:33:12 2015 +0200 description: imapc: Added imapc_features=search support for sending SEARCH commands. Currently requires the remote server to support ESEARCH (but this would be easy to avoid). This is only minimally tested for now, so bugs may exist (especially related to sub-queries). diffstat: src/lib-imap-client/imapc-client.c | 1 + src/lib-imap-client/imapc-client.h | 1 + src/lib-storage/index/imapc/Makefile.am | 2 + src/lib-storage/index/imapc/imapc-mailbox.c | 7 +- src/lib-storage/index/imapc/imapc-search.c | 357 +++++++++++++++++++++++++++ src/lib-storage/index/imapc/imapc-search.h | 16 + src/lib-storage/index/imapc/imapc-settings.c | 1 + src/lib-storage/index/imapc/imapc-settings.h | 3 +- src/lib-storage/index/imapc/imapc-storage.c | 7 +- src/lib-storage/index/imapc/imapc-storage.h | 1 + 10 files changed, 391 insertions(+), 5 deletions(-) diffs (truncated from 513 to 300 lines): diff -r 6c2ea1d6ab58 -r 405959839f48 src/lib-imap-client/imapc-client.c --- a/src/lib-imap-client/imapc-client.c Tue Feb 03 10:15:38 2015 +0200 +++ b/src/lib-imap-client/imapc-client.c Tue Feb 03 18:33:12 2015 +0200 @@ -24,6 +24,7 @@ { "CONDSTORE", IMAPC_CAPABILITY_CONDSTORE }, { "NAMESPACE", IMAPC_CAPABILITY_NAMESPACE }, { "UNSELECT", IMAPC_CAPABILITY_UNSELECT }, + { "ESEARCH", IMAPC_CAPABILITY_ESEARCH }, { "IMAP4REV1", IMAPC_CAPABILITY_IMAP4REV1 }, { NULL, 0 } diff -r 6c2ea1d6ab58 -r 405959839f48 src/lib-imap-client/imapc-client.h --- a/src/lib-imap-client/imapc-client.h Tue Feb 03 10:15:38 2015 +0200 +++ b/src/lib-imap-client/imapc-client.h Tue Feb 03 18:33:12 2015 +0200 @@ -23,6 +23,7 @@ IMAPC_CAPABILITY_CONDSTORE = 0x100, IMAPC_CAPABILITY_NAMESPACE = 0x200, IMAPC_CAPABILITY_UNSELECT = 0x400, + IMAPC_CAPABILITY_ESEARCH = 0x800, IMAPC_CAPABILITY_IMAP4REV1 = 0x40000000 }; diff -r 6c2ea1d6ab58 -r 405959839f48 src/lib-storage/index/imapc/Makefile.am --- a/src/lib-storage/index/imapc/Makefile.am Tue Feb 03 10:15:38 2015 +0200 +++ b/src/lib-storage/index/imapc/Makefile.am Tue Feb 03 18:33:12 2015 +0200 @@ -18,6 +18,7 @@ imapc-mail-fetch.c \ imapc-mailbox.c \ imapc-save.c \ + imapc-search.c \ imapc-settings.c \ imapc-sync.c \ imapc-storage.c @@ -25,6 +26,7 @@ headers = \ imapc-list.h \ imapc-mail.h \ + imapc-search.h \ imapc-settings.h \ imapc-storage.h \ imapc-sync.h diff -r 6c2ea1d6ab58 -r 405959839f48 src/lib-storage/index/imapc/imapc-mailbox.c --- a/src/lib-storage/index/imapc/imapc-mailbox.c Tue Feb 03 10:15:38 2015 +0200 +++ b/src/lib-storage/index/imapc/imapc-mailbox.c Tue Feb 03 18:33:12 2015 +0200 @@ -8,6 +8,7 @@ #include "imapc-client.h" #include "imapc-mail.h" #include "imapc-msgmap.h" +#include "imapc-search.h" #include "imapc-sync.h" #include "imapc-storage.h" @@ -438,7 +439,9 @@ i_free_and_null(mbox->sync_gmail_pop3_search_tag); - /* It should contain ALL */ + /* It should contain ALL or nonexistent if nothing matched */ + if (args[0].type == IMAP_ARG_EOL) + return; t_array_init(&rseqs, 64); if (!imap_arg_atom_equals(&args[0], "ALL") || !imap_arg_get_atom(&args[1], &atom) || @@ -493,6 +496,8 @@ if (mbox->sync_gmail_pop3_search_tag != NULL && strcmp(mbox->sync_gmail_pop3_search_tag, str) == 0) imapc_untagged_esearch_gmail_pop3(reply->args+1, mbox); + else + imapc_search_reply(reply->args+1, mbox); } static void diff -r 6c2ea1d6ab58 -r 405959839f48 src/lib-storage/index/imapc/imapc-search.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/lib-storage/index/imapc/imapc-search.c Tue Feb 03 18:33:12 2015 +0200 @@ -0,0 +1,357 @@ +/* Copyright (c) 2015 Dovecot authors, see the included COPYING file */ + +#include "lib.h" +#include "str.h" +#include "imap-arg.h" +#include "imap-date.h" +#include "imap-quote.h" +#include "imap-seqset.h" +#include "imap-util.h" +#include "mail-search.h" +#include "imapc-client.h" +#include "imapc-storage.h" +#include "imapc-search.h" + +#define IMAPC_CONTEXT(obj) \ + MODULE_CONTEXT(obj, imapc_storage_module) + +struct imapc_search_context { + union mail_search_module_context module_ctx; + + ARRAY_TYPE(seq_range) rseqs; + struct seq_range_iter iter; + unsigned int n; + bool finished; + bool success; +}; + +static MODULE_CONTEXT_DEFINE_INIT(imapc_storage_module, + &mail_storage_module_register); + +static bool +imapc_build_search_query_args(struct imapc_mailbox *mbox, + const struct mail_search_arg *args, + bool parent_or, string_t *str); + +static bool imapc_search_is_fast_local(const struct mail_search_arg *args) +{ + const struct mail_search_arg *arg; + + for (arg = args; arg != NULL; arg = arg->next) { + switch (arg->type) { + case SEARCH_OR: + case SEARCH_SUB: + if (!imapc_search_is_fast_local(arg->value.subargs)) + return FALSE; + break; + case SEARCH_ALL: + case SEARCH_SEQSET: + case SEARCH_UIDSET: + case SEARCH_FLAGS: + case SEARCH_KEYWORDS: + case SEARCH_MODSEQ: + case SEARCH_MAILBOX: + case SEARCH_MAILBOX_GUID: + case SEARCH_MAILBOX_GLOB: + case SEARCH_REAL_UID: + break; + default: + return FALSE; + } + } + return TRUE; +} + +static bool +imapc_build_search_query_arg(struct imapc_mailbox *mbox, + const struct mail_search_arg *arg, + string_t *str) +{ + enum imapc_capability capa = + imapc_client_get_capabilities(mbox->storage->client->client); + + if (arg->match_not) + str_append(str, "NOT "); + switch (arg->type) { + case SEARCH_OR: + str_append_c(str, '('); + imapc_build_search_query_args(mbox, arg->value.subargs, TRUE, str); + str_append_c(str, ')'); + break; + case SEARCH_SUB: + str_append_c(str, '('); + imapc_build_search_query_args(mbox, arg->value.subargs, FALSE, str); + str_append_c(str, ')'); + break; + + case SEARCH_ALL: + str_append(str, "ALL"); + break; + case SEARCH_SEQSET: + /* translate to UIDs */ + T_BEGIN { + ARRAY_TYPE(seq_range) uids; + + t_array_init(&uids, 64); + mailbox_get_uid_range(&mbox->box, &arg->value.seqset, + &uids); + str_append(str, "UID "); + imap_write_seq_range(str, &uids); + } T_END; + break; + case SEARCH_UIDSET: + str_append(str, "UID "); + imap_write_seq_range(str, &arg->value.seqset); + break; + case SEARCH_FLAGS: + i_assert((arg->value.flags & MAIL_FLAGS_MASK) != 0); + str_append_c(str, '('); + if ((arg->value.flags & MAIL_ANSWERED) != 0) + str_append(str, "ANSWERED "); + if ((arg->value.flags & MAIL_FLAGGED) != 0) + str_append(str, "FLAGGED "); + if ((arg->value.flags & MAIL_DELETED) != 0) + str_append(str, "DELETED "); + if ((arg->value.flags & MAIL_SEEN) != 0) + str_append(str, "SEEN "); + if ((arg->value.flags & MAIL_DRAFT) != 0) + str_append(str, "DRAFT "); + if ((arg->value.flags & MAIL_RECENT) != 0) + str_append(str, "RECENT "); + str_truncate(str, str_len(str)-1); + str_append_c(str, ')'); + break; + case SEARCH_KEYWORDS: { + const struct mail_keywords *kw = arg->value.keywords; + const ARRAY_TYPE(keywords) *names_arr; + const char *const *namep; + unsigned int i; + + names_arr = mail_index_get_keywords(kw->index); + + str_append_c(str, '('); + for (i = 0; i < kw->count; i++) { + namep = array_idx(names_arr, kw->idx[i]); + if (i > 0) + str_append_c(str, ' '); + str_printfa(str, "KEYWORD %s", *namep); + } + str_append_c(str, ')'); + break; + } + + case SEARCH_BEFORE: + str_printfa(str, "BEFORE \"%s\"", imap_to_datetime(arg->value.time)); + break; + case SEARCH_ON: + str_printfa(str, "ON \"%s\"", imap_to_datetime(arg->value.time)); + break; + case SEARCH_SINCE: + str_printfa(str, "SINCE \"%s\"", imap_to_datetime(arg->value.time)); + break; + case SEARCH_SMALLER: + str_printfa(str, "SMALLER %llu", (unsigned long long)arg->value.size); + break; + case SEARCH_LARGER: + str_printfa(str, "LARGER %llu", (unsigned long long)arg->value.size); + break; + case SEARCH_HEADER: + case SEARCH_HEADER_ADDRESS: + case SEARCH_HEADER_COMPRESS_LWSP: + if (strcasecmp(arg->hdr_field_name, "From") == 0 || + strcasecmp(arg->hdr_field_name, "To") == 0 || + strcasecmp(arg->hdr_field_name, "Cc") == 0 || + strcasecmp(arg->hdr_field_name, "Bcc") == 0 || + strcasecmp(arg->hdr_field_name, "Subject") == 0) + str_append(str, arg->hdr_field_name); + else { + str_append(str, "HEADER "); + imap_append_astring(str, arg->hdr_field_name); + } + str_append_c(str, ' '); + imap_append_astring(str, arg->value.str); + break; + + case SEARCH_BODY: + str_append(str, "BODY "); + imap_append_astring(str, arg->value.str); + break; + case SEARCH_TEXT: + str_append(str, "TEXT "); + imap_append_astring(str, arg->value.str); + break; + + /* extensions */ + case SEARCH_MODSEQ: + if ((capa & IMAPC_CAPABILITY_CONDSTORE) == 0) + return FALSE; + str_printfa(str, "MODSEQ %llu", (unsigned long long)arg->value.modseq); + break; + case SEARCH_INTHREAD: + case SEARCH_GUID: + case SEARCH_MAILBOX: + case SEARCH_MAILBOX_GUID: + case SEARCH_MAILBOX_GLOB: + case SEARCH_REAL_UID: + return FALSE; + default: + return FALSE; + } + return TRUE; +} + +static bool +imapc_build_search_query_args(struct imapc_mailbox *mbox, + const struct mail_search_arg *args, + bool parent_or, string_t *str) +{ + const struct mail_search_arg *arg; + + for (arg = args; arg != NULL; arg = arg->next) { + if (parent_or && arg->next != NULL) + str_append(str, "OR "); + if (!imapc_build_search_query_arg(mbox, arg, str)) + return FALSE; + str_append_c(str, ' '); + } + str_truncate(str, str_len(str)-1); + return TRUE; +} + +static bool imapc_build_search_query(struct imapc_mailbox *mbox, + const struct mail_search_args *args, + const char **query_r) +{ From dovecot at dovecot.org Tue Feb 3 16:48:02 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 03 Feb 2015 16:48:02 +0000 Subject: dovecot-2.2: imapc: Fixed modseq search for previous imapc_featu... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/bb92d04e5f71 changeset: 18216:bb92d04e5f71 user: Timo Sirainen date: Tue Feb 03 18:47:54 2015 +0200 description: imapc: Fixed modseq search for previous imapc_features=search change diffstat: src/lib-storage/index/imapc/imapc-search.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 405959839f48 -r bb92d04e5f71 src/lib-storage/index/imapc/imapc-search.c --- a/src/lib-storage/index/imapc/imapc-search.c Tue Feb 03 18:33:12 2015 +0200 +++ b/src/lib-storage/index/imapc/imapc-search.c Tue Feb 03 18:47:54 2015 +0200 @@ -185,7 +185,7 @@ case SEARCH_MODSEQ: if ((capa & IMAPC_CAPABILITY_CONDSTORE) == 0) return FALSE; - str_printfa(str, "MODSEQ %llu", (unsigned long long)arg->value.modseq); + str_printfa(str, "MODSEQ %llu", (unsigned long long)arg->value.modseq->modseq); break; case SEARCH_INTHREAD: case SEARCH_GUID: From dovecot at dovecot.org Tue Feb 3 18:27:58 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 03 Feb 2015 18:27:58 +0000 Subject: dovecot-2.2: lib-storage: Avoid assert-crashing on mailbox_verif... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9bec5721d277 changeset: 18217:9bec5721d277 user: Timo Sirainen date: Tue Feb 03 20:23:52 2015 +0200 description: lib-storage: Avoid assert-crashing on mailbox_verify_*_name() with invalid namespace prefix. diffstat: src/lib-storage/mail-storage.c | 15 ++++++++++++++- 1 files changed, 14 insertions(+), 1 deletions(-) diffs (32 lines): diff -r bb92d04e5f71 -r 9bec5721d277 src/lib-storage/mail-storage.c --- a/src/lib-storage/mail-storage.c Tue Feb 03 18:47:54 2015 +0200 +++ b/src/lib-storage/mail-storage.c Tue Feb 03 20:23:52 2015 +0200 @@ -5,6 +5,7 @@ #include "array.h" #include "llist.h" #include "str.h" +#include "str-sanitize.h" #include "unichar.h" #include "istream.h" #include "eacces-error.h" @@ -871,7 +872,19 @@ if (ns->prefix_len > 0) { /* vname is either "namespace/box" or "namespace" */ - i_assert(strncmp(vname, ns->prefix, ns->prefix_len-1) == 0); + if (strncmp(vname, ns->prefix, ns->prefix_len-1) != 0 || + (vname[ns->prefix_len-1] != '\0' && + vname[ns->prefix_len-1] != ns->prefix[ns->prefix_len-1])) { + /* User input shouldn't normally be able to get us in + here. The main reason this isn't an assert is to + allow any input at all to mailbox_verify_*_name() + without crashing. */ + mail_storage_set_error(box->storage, MAIL_ERROR_PARAMS, + t_strdup_printf("Invalid mailbox name '%s': " + "Missing namespace prefix '%s'", + str_sanitize(vname, 80), ns->prefix)); + return -1; + } vname += ns->prefix_len - 1; if (vname[0] != '\0') { i_assert(vname[0] == ns->prefix[ns->prefix_len-1]); From dovecot at dovecot.org Tue Feb 3 18:27:59 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 03 Feb 2015 18:27:59 +0000 Subject: dovecot-2.2: dsync: Make sure when fixing mailbox names we don't... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6a74968f7978 changeset: 18218:6a74968f7978 user: Timo Sirainen date: Tue Feb 03 20:27:49 2015 +0200 description: dsync: Make sure when fixing mailbox names we don't try to change the namespace prefix. Although in normal installations this it wouldn't have happened in any case. diffstat: src/doveadm/dsync/dsync-brain-mailbox-tree.c | 56 ++++++++++++++++------------ 1 files changed, 32 insertions(+), 24 deletions(-) diffs (95 lines): diff -r 9bec5721d277 -r 6a74968f7978 src/doveadm/dsync/dsync-brain-mailbox-tree.c --- a/src/doveadm/dsync/dsync-brain-mailbox-tree.c Tue Feb 03 20:23:52 2015 +0200 +++ b/src/doveadm/dsync/dsync-brain-mailbox-tree.c Tue Feb 03 20:27:49 2015 +0200 @@ -180,59 +180,67 @@ } static void -dsync_fix_mailbox_name(struct mail_namespace *ns, string_t *vname, +dsync_fix_mailbox_name(struct mail_namespace *ns, string_t *vname_str, char alt_char) { const char *old_vname; - char *p, list_sep = mailbox_list_get_hierarchy_sep(ns->list); + char *vname, list_sep = mailbox_list_get_hierarchy_sep(ns->list); guid_128_t guid; + unsigned int i, start_pos; + + vname = str_c_modifiable(vname_str); + if (strncmp(vname, ns->prefix, ns->prefix_len) == 0) + start_pos = ns->prefix_len; + else + start_pos = 0; /* replace control chars */ - for (p = str_c_modifiable(vname); *p != '\0'; p++) { - if ((unsigned char)*p < ' ') - *p = alt_char; + for (i = start_pos; vname[i] != '\0'; i++) { + if ((unsigned char)vname[i] < ' ') + vname[i] = alt_char; } /* make it valid UTF8 */ - if (!uni_utf8_str_is_valid(str_c(vname))) { - old_vname = t_strdup(str_c(vname)); - str_truncate(vname, 0); + if (!uni_utf8_str_is_valid(vname)) { + old_vname = t_strdup(vname + start_pos); + str_truncate(vname_str, start_pos); if (uni_utf8_get_valid_data((const void *)old_vname, - strlen(old_vname), vname)) + strlen(old_vname), vname_str)) i_unreached(); + vname = str_c_modifiable(vname_str); } - if (dsync_is_valid_name(ns, str_c(vname))) + if (dsync_is_valid_name(ns, vname)) return; /* 1) change any real separators to alt separators (this wouldn't be necessary with listescape, but don't bother detecting it) */ if (list_sep != mail_namespace_get_sep(ns)) { - for (p = str_c_modifiable(vname); *p != '\0'; p++) { - if (*p == list_sep) - *p = alt_char; + for (i = start_pos; vname[i] != '\0'; i++) { + if (vname[i] == list_sep) + vname[i] = alt_char; } - if (dsync_is_valid_name(ns, str_c(vname))) + if (dsync_is_valid_name(ns, vname)) return; } /* 2) '/' characters aren't valid without listescape */ if (mail_namespace_get_sep(ns) != '/' && list_sep != '/') { - for (p = str_c_modifiable(vname); *p != '\0'; p++) { - if (*p == '/') - *p = alt_char; + for (i = start_pos; vname[i] != '\0'; i++) { + if (vname[i] == '/') + vname[i] = alt_char; } - if (dsync_is_valid_name(ns, str_c(vname))) + if (dsync_is_valid_name(ns, vname)) return; } /* 3) probably some reserved name (e.g. dbox-Mails) */ - str_insert(vname, ns->prefix_len, "_"); - if (dsync_is_valid_name(ns, str_c(vname))) + str_insert(vname_str, ns->prefix_len, "_"); + if (dsync_is_valid_name(ns, str_c(vname_str))) return; /* 4) name is too long? just give up and generate a unique name */ guid_128_generate(guid); - str_truncate(vname, 0); - str_append(vname, ns->prefix); - str_append(vname, guid_128_to_string(guid)); - i_assert(dsync_is_valid_name(ns, str_c(vname))); + str_truncate(vname_str, 0); + str_append(vname_str, ns->prefix); + str_append(vname_str, guid_128_to_string(guid)); + i_assert(dsync_is_valid_name(ns, str_c(vname_str))); } static int From dovecot at dovecot.org Thu Feb 5 17:20:31 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 05 Feb 2015 17:20:31 +0000 Subject: dovecot-2.2: dsync: Don't assert-crash in configs where INBOX do... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/00a57ac1b9fa changeset: 18219:00a57ac1b9fa user: Timo Sirainen date: Thu Feb 05 19:20:17 2015 +0200 description: dsync: Don't assert-crash in configs where INBOX doesn't have a parent namespace. The inbox=yes namespace of course exists, but if there is no prefix="" or prefix=INBOX/ namespace, then INBOX doesn't clearly belong to any namespace. diffstat: src/doveadm/dsync/dsync-mailbox-tree-fill.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diffs (19 lines): diff -r 6a74968f7978 -r 00a57ac1b9fa src/doveadm/dsync/dsync-mailbox-tree-fill.c --- a/src/doveadm/dsync/dsync-mailbox-tree-fill.c Tue Feb 03 20:27:49 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-fill.c Thu Feb 05 19:20:17 2015 +0200 @@ -20,9 +20,12 @@ struct dsync_mailbox_node *node; node = dsync_mailbox_tree_get(tree, info->vname); - if (node->ns != info->ns) { - i_assert(node->ns != NULL); - + if (node->ns == info->ns) + ; + else if (node->ns == NULL) { + i_assert(tree->root.ns == NULL); + node->ns = info->ns; + } else { i_error("Mailbox '%s' exists in two namespaces: '%s' and '%s'", info->vname, node->ns->prefix, info->ns->prefix); return -1; From dovecot at dovecot.org Thu Feb 5 17:36:39 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 05 Feb 2015 17:36:39 +0000 Subject: dovecot-2.2: dsync: Added assert to make sure namespace prefix i... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/560043558dd6 changeset: 18220:560043558dd6 user: Timo Sirainen date: Thu Feb 05 19:36:29 2015 +0200 description: dsync: Added assert to make sure namespace prefix isn't attempted to be renamed. This can happen, I'm just not sure how exactly.. The previous behavior caused assert-crashes also, this change just makes it happen earlier. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 18 ++++++++++++++++++ src/doveadm/dsync/test-dsync-mailbox-tree-sync.c | 5 +++++ 2 files changed, 23 insertions(+), 0 deletions(-) diffs (50 lines): diff -r 00a57ac1b9fa -r 560043558dd6 src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Thu Feb 05 19:20:17 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Thu Feb 05 19:36:29 2015 +0200 @@ -357,6 +357,22 @@ return node_has_existent_children(node, TRUE); } +static bool sync_node_is_namespace_prefix(struct dsync_mailbox_tree *tree, + struct dsync_mailbox_node *node) +{ + const char *full_name; + unsigned int prefix_len = node->ns->prefix_len; + + if (prefix_len == 0) + return FALSE; + + full_name = dsync_mailbox_node_get_full_name(tree, node); + if (node->ns->prefix[prefix_len-1] == mail_namespace_get_sep(node->ns)) + prefix_len--; + return strncmp(full_name, node->ns->prefix, prefix_len) == 0 && + full_name[prefix_len] == '\0'; +} + static void sync_rename_node_to_temp(struct dsync_mailbox_tree_sync_ctx *ctx, struct dsync_mailbox_tree *tree, @@ -369,6 +385,8 @@ buffer_t buf; unsigned int prefix_len, max_prefix_len, counter = 1; + i_assert(!sync_node_is_namespace_prefix(tree, node)); + buffer_create_from_data(&buf, name, sizeof(name)); max_prefix_len = TEMP_MAX_NAME_LEN - TEMP_SUFFIX_MAX_LEN - 1; if (node->sync_temporary_name) { diff -r 00a57ac1b9fa -r 560043558dd6 src/doveadm/dsync/test-dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/test-dsync-mailbox-tree-sync.c Thu Feb 05 19:20:17 2015 +0200 +++ b/src/doveadm/dsync/test-dsync-mailbox-tree-sync.c Thu Feb 05 19:36:29 2015 +0200 @@ -12,6 +12,11 @@ #define MAX_DEPTH 4 +char mail_namespace_get_sep(struct mail_namespace *ns ATTR_UNUSED) +{ + return '/'; +} + void mailbox_name_get_sha128(const char *name, guid_128_t guid_128_r) { unsigned char sha[SHA1_RESULTLEN]; From dovecot at dovecot.org Thu Feb 5 17:40:40 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 05 Feb 2015 17:40:40 +0000 Subject: dovecot-2.2: dsync: Fixed crash in earlier patch if node didn't ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d7defdcfe039 changeset: 18221:d7defdcfe039 user: Timo Sirainen date: Thu Feb 05 19:40:25 2015 +0200 description: dsync: Fixed crash in earlier patch if node didn't have a namespace. This shouldn't have happened except in test-dsync-mailbox-tree-sync, but easier and safer to fix it here. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r 560043558dd6 -r d7defdcfe039 src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Thu Feb 05 19:36:29 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Thu Feb 05 19:40:25 2015 +0200 @@ -361,7 +361,7 @@ struct dsync_mailbox_node *node) { const char *full_name; - unsigned int prefix_len = node->ns->prefix_len; + unsigned int prefix_len = node->ns == NULL ? 0 : node->ns->prefix_len; if (prefix_len == 0) return FALSE; From dovecot at dovecot.org Fri Feb 6 15:06:45 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 15:06:45 +0000 Subject: dovecot-2.2: lib: Added i_get_failure_prefix() Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/89c42f824f84 changeset: 18222:89c42f824f84 user: Timo Sirainen date: Fri Feb 06 17:04:52 2015 +0200 description: lib: Added i_get_failure_prefix() diffstat: src/lib/failures.c | 5 +++++ src/lib/failures.h | 2 ++ 2 files changed, 7 insertions(+), 0 deletions(-) diffs (27 lines): diff -r d7defdcfe039 -r 89c42f824f84 src/lib/failures.c --- a/src/lib/failures.c Thu Feb 05 19:40:25 2015 +0200 +++ b/src/lib/failures.c Fri Feb 06 17:04:52 2015 +0200 @@ -547,6 +547,11 @@ log_prefix_sent = FALSE; } +const char *i_get_failure_prefix(void) +{ + return log_prefix != NULL ? log_prefix : ""; +} + static int internal_send_split(string_t *full_str, unsigned int prefix_len) { string_t *str; diff -r d7defdcfe039 -r 89c42f824f84 src/lib/failures.h --- a/src/lib/failures.h Thu Feb 05 19:40:25 2015 +0200 +++ b/src/lib/failures.h Fri Feb 06 17:04:52 2015 +0200 @@ -115,6 +115,8 @@ void i_set_failure_prefix(const char *prefix_fmt, ...) ATTR_FORMAT(1, 2); /* Set prefix to "". */ void i_unset_failure_prefix(void); +/* Returns the current failure prefix (never NULL). */ +const char *i_get_failure_prefix(void); /* Prefix failures with a timestamp. fmt is in strftime() format. */ void i_set_failure_timestamp_format(const char *fmt); /* When logging with internal error protocol, update the process's current From dovecot at dovecot.org Fri Feb 6 15:06:51 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 15:06:51 +0000 Subject: dovecot-2.2: lib-storage: Set the user log prefix earlier when i... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6be98d5d829a changeset: 18223:6be98d5d829a user: Timo Sirainen date: Fri Feb 06 17:06:36 2015 +0200 description: lib-storage: Set the user log prefix earlier when initializing user. So the full prefix with session and other useful information is included in the log message as early as possible. Some fatal/panic errors wouldn't even have shown the username. diffstat: src/lib-storage/mail-storage-service.c | 117 ++++++++++++++++++++++++++------ 1 files changed, 94 insertions(+), 23 deletions(-) diffs (212 lines): diff -r 89c42f824f84 -r 6be98d5d829a src/lib-storage/mail-storage-service.c --- a/src/lib-storage/mail-storage-service.c Fri Feb 06 17:04:52 2015 +0200 +++ b/src/lib-storage/mail-storage-service.c Fri Feb 06 17:06:36 2015 +0200 @@ -379,8 +379,8 @@ static const struct var_expand_table * get_var_expand_table(struct master_service *service, struct mail_storage_service_user *user, - struct mail_storage_service_input *input, - struct mail_storage_service_privileges *priv) + const struct mail_storage_service_input *input, + const struct mail_storage_service_privileges *priv) { static struct var_expand_table static_tab[] = { { 'u', NULL, "user" }, @@ -411,8 +411,10 @@ tab[4].value = net_ip2addr(&input->local_ip); tab[5].value = net_ip2addr(&input->remote_ip); tab[6].value = my_pid; - tab[7].value = dec2str(priv->uid == (uid_t)-1 ? geteuid() : priv->uid); - tab[8].value = dec2str(priv->gid == (gid_t)-1 ? getegid() : priv->gid); + tab[7].value = priv == NULL ? NULL : + dec2str(priv->uid == (uid_t)-1 ? geteuid() : priv->uid); + tab[8].value = priv == NULL ? NULL : + dec2str(priv->gid == (gid_t)-1 ? getegid() : priv->gid); tab[9].value = input->session_id; if (user == NULL || user->auth_user == NULL) { tab[10].value = tab[0].value; @@ -1006,10 +1008,27 @@ return *p1 - *p2; } -int mail_storage_service_lookup(struct mail_storage_service_ctx *ctx, - const struct mail_storage_service_input *input, - struct mail_storage_service_user **user_r, - const char **error_r) +static void +mail_storage_service_set_log_prefix(struct mail_storage_service_ctx *ctx, + const struct mail_user_settings *user_set, + struct mail_storage_service_user *user, + const struct mail_storage_service_input *input, + const struct mail_storage_service_privileges *priv) +{ + string_t *str; + + str = t_str_new(256); + var_expand(str, user_set->mail_log_prefix, + get_var_expand_table(ctx->service, user, input, priv)); + i_set_failure_prefix("%s", str_c(str)); +} + +static int +mail_storage_service_lookup_real(struct mail_storage_service_ctx *ctx, + const struct mail_storage_service_input *input, + bool update_log_prefix, + struct mail_storage_service_user **user_r, + const char **error_r) { enum mail_storage_service_flags flags; struct mail_storage_service_user *user; @@ -1040,9 +1059,9 @@ if (ctx->config_permission_denied) { /* just restart and maybe next time we will open the config socket before dropping privileges */ - i_fatal("user %s: %s", username, error); + i_fatal("%s", error); } - i_error("user %s: %s", username, error); + i_error("%s", error); pool_unref(&user_pool); *error_r = MAIL_ERRSTR_CRITICAL_MSG; return -1; @@ -1055,11 +1074,15 @@ ctx->log_initialized = TRUE; master_service_init_log(ctx->service, t_strconcat(ctx->service->name, ": ", NULL)); + update_log_prefix = TRUE; } sets = master_service_settings_parser_get_others(master_service, set_parser); user_set = sets[0]; + if (update_log_prefix) + mail_storage_service_set_log_prefix(ctx, user_set, NULL, input, NULL); + if (ctx->conn == NULL) mail_storage_service_first_init(ctx, user_info, user_set); /* load global plugins */ @@ -1120,8 +1143,7 @@ auth_user_fields_parse(userdb_fields, temp_pool, &reply); array_sort(&reply.extra_fields, extra_field_key_cmp_p); if (user_reply_handle(ctx, user, &reply, &error) < 0) { - i_error("user %s: Invalid settings in userdb: %s", - username, error); + i_error("Invalid settings in userdb: %s", error); *error_r = ERRSTR_INVALID_USER_SETTINGS; ret = -2; } @@ -1138,6 +1160,41 @@ return ret; } +int mail_storage_service_lookup(struct mail_storage_service_ctx *ctx, + const struct mail_storage_service_input *input, + struct mail_storage_service_user **user_r, + const char **error_r) +{ + const char *old_log_prefix = i_get_failure_prefix(); + bool update_log_prefix; + int ret; + + if (io_loop_get_current_context(current_ioloop) == NULL) { + /* no user yet. log prefix should be just "imap:" or something + equally unhelpful. we don't know the proper log format yet, + but initialize it to something better until we know it. */ + i_set_failure_prefix("%s(%s%s,%s)", + master_service_get_name(ctx->service), input->username, + input->session_id == NULL ? "" : + t_strdup_printf(",%s", input->session_id), + input->remote_ip.family == 0 ? "" : + t_strdup_printf(",%s", net_ip2addr(&input->remote_ip))); + update_log_prefix = TRUE; + } else { + /* we might be here because we're doing a user lookup for a + shared user. the log prefix is likely already usable, so + just append our own without replacing the whole thing. */ + i_set_failure_prefix("%suser-lookup(%s)", + old_log_prefix, input->username); + update_log_prefix = FALSE; + } + + ret = mail_storage_service_lookup_real(ctx, input, update_log_prefix, + user_r, error_r); + i_set_failure_prefix("%s", old_log_prefix); + return ret; +} + void mail_storage_service_save_userdb_fields(struct mail_storage_service_ctx *ctx, pool_t pool, const char *const **userdb_fields_r) { @@ -1149,9 +1206,10 @@ *userdb_fields_r = NULL; } -int mail_storage_service_next(struct mail_storage_service_ctx *ctx, - struct mail_storage_service_user *user, - struct mail_user **mail_user_r) +static int +mail_storage_service_next_real(struct mail_storage_service_ctx *ctx, + struct mail_storage_service_user *user, + struct mail_user **mail_user_r) { struct mail_storage_service_privileges priv; const char *error; @@ -1163,14 +1221,13 @@ bool use_chroot; if (service_parse_privileges(ctx, user, &priv, &error) < 0) { - i_error("user %s: %s", user->input.username, error); + i_error("%s", error); return -2; } if (*priv.home != '/' && *priv.home != '\0') { - i_error("user %s: " - "Relative home directory paths not supported: %s", - user->input.username, priv.home); + i_error("Relative home directory paths not supported: %s", + priv.home); return -2; } @@ -1213,8 +1270,7 @@ if (service_drop_privileges(user, &priv, disallow_root, temp_priv_drop, FALSE, &error) < 0) { - i_error("user %s: Couldn't drop privileges: %s", - user->input.username, error); + i_error("Couldn't drop privileges: %s", error); return -1; } if (!temp_priv_drop || @@ -1228,13 +1284,28 @@ if (mail_storage_service_init_post(ctx, user, &priv, mail_user_r, &error) < 0) { - i_error("user %s: Initialization failed: %s", - user->input.username, error); + i_error("User initialization failed: %s", error); return -2; } return 0; } +int mail_storage_service_next(struct mail_storage_service_ctx *ctx, + struct mail_storage_service_user *user, + struct mail_user **mail_user_r) +{ + const char *old_log_prefix = i_get_failure_prefix(); + int ret; + + mail_storage_service_set_log_prefix(ctx, user->user_set, user, + &user->input, NULL); + i_set_failure_prefix("%s", old_log_prefix); + ret = mail_storage_service_next_real(ctx, user, mail_user_r); + if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_LOG_INIT) != 0) + i_set_failure_prefix("%s", old_log_prefix); + return ret; +} + void mail_storage_service_restrict_setenv(struct mail_storage_service_ctx *ctx, struct mail_storage_service_user *user) { From dovecot at dovecot.org Fri Feb 6 16:18:30 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 16:18:30 +0000 Subject: dovecot-2.2: lib-storage: Previous log prefix changing code star... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/280246f95e74 changeset: 18224:280246f95e74 user: Timo Sirainen date: Fri Feb 06 18:17:10 2015 +0200 description: lib-storage: Previous log prefix changing code started accessing freed memory. diffstat: src/lib-storage/mail-storage-service.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diffs (37 lines): diff -r 6be98d5d829a -r 280246f95e74 src/lib-storage/mail-storage-service.c --- a/src/lib-storage/mail-storage-service.c Fri Feb 06 17:06:36 2015 +0200 +++ b/src/lib-storage/mail-storage-service.c Fri Feb 06 18:17:10 2015 +0200 @@ -1165,7 +1165,7 @@ struct mail_storage_service_user **user_r, const char **error_r) { - const char *old_log_prefix = i_get_failure_prefix(); + char *old_log_prefix = i_strdup(i_get_failure_prefix()); bool update_log_prefix; int ret; @@ -1192,6 +1192,7 @@ ret = mail_storage_service_lookup_real(ctx, input, update_log_prefix, user_r, error_r); i_set_failure_prefix("%s", old_log_prefix); + i_free(old_log_prefix); return ret; } @@ -1294,7 +1295,7 @@ struct mail_storage_service_user *user, struct mail_user **mail_user_r) { - const char *old_log_prefix = i_get_failure_prefix(); + char *old_log_prefix = i_strdup(i_get_failure_prefix()); int ret; mail_storage_service_set_log_prefix(ctx, user->user_set, user, @@ -1303,6 +1304,7 @@ ret = mail_storage_service_next_real(ctx, user, mail_user_r); if ((user->flags & MAIL_STORAGE_SERVICE_FLAG_NO_LOG_INIT) != 0) i_set_failure_prefix("%s", old_log_prefix); + i_free(old_log_prefix); return ret; } From dovecot at dovecot.org Fri Feb 6 16:18:31 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 16:18:31 +0000 Subject: dovecot-2.2: lib: Added module_dir_try_load_missing() that retur... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/88b09d5912c0 changeset: 18225:88b09d5912c0 user: Timo Sirainen date: Fri Feb 06 18:18:10 2015 +0200 description: lib: Added module_dir_try_load_missing() that returns error instead of logging it. Also module_names now can be given for it without i_fatal() being called. diffstat: src/lib/module-dir.c | 143 +++++++++++++++++++++++++++++++++----------------- src/lib/module-dir.h | 6 ++ 2 files changed, 99 insertions(+), 50 deletions(-) diffs (truncated from 317 to 300 lines): diff -r 280246f95e74 -r 88b09d5912c0 src/lib/module-dir.c --- a/src/lib/module-dir.c Fri Feb 06 18:17:10 2015 +0200 +++ b/src/lib/module-dir.c Fri Feb 06 18:18:10 2015 +0200 @@ -74,7 +74,7 @@ static bool module_check_wrong_binary_dependency(const struct module_dir_load_settings *set, - struct module *module) + struct module *module, const char **error_r) { const char *symbol_name, *binary_dep, *const *names; string_t *errstr; @@ -99,14 +99,15 @@ else str_printfa(errstr, "binaries: %s", binary_dep); str_printfa(errstr, " (we're %s)", set->binary_name); - i_error("%s", str_c(errstr)); + *error_r = str_c(errstr); return FALSE; } static bool module_check_missing_plugin_dependencies(const struct module_dir_load_settings *set, struct module *module, - struct module *all_modules) + struct module *all_modules, + const char **error_r) { const char **deps; struct module *m; @@ -137,7 +138,7 @@ set->setting_name, set->setting_name, *deps); } - i_error("%s", str_c(errmsg)); + *error_r = str_c(errmsg); return FALSE; } } @@ -184,15 +185,18 @@ return FALSE; } -static struct module * +static int module_load(const char *path, const char *name, const struct module_dir_load_settings *set, - struct module *all_modules) + struct module *all_modules, + struct module **module_r, const char **error_r) { void *handle; struct module *module; const char *const *module_version; - bool failed = FALSE; + + *module_r = NULL; + *error_r = NULL; if (set->ignore_dlopen_errors) { handle = quiet_dlopen(path, RTLD_GLOBAL | RTLD_NOW); @@ -204,21 +208,21 @@ "so just ignore this message)", name, dlerror()); } - return NULL; + return 0; } } else { handle = dlopen(path, RTLD_GLOBAL | RTLD_NOW); if (handle == NULL) { - i_error("dlopen(%s) failed: %s", path, dlerror()); + *error_r = t_strdup_printf("dlopen(%s) failed: %s", + path, dlerror()); #ifdef RTLD_LAZY - failed = TRUE; /* try to give a better error message by lazily loading the plugin and checking its dependencies */ handle = dlopen(path, RTLD_LAZY); if (handle == NULL) - return NULL; + return -1; #else - return NULL; + return -1; #endif } } @@ -232,10 +236,11 @@ get_symbol(module, t_strconcat(name, "_version", NULL), TRUE); if (module_version != NULL && !versions_equal(*module_version, set->abi_version)) { - i_error("Module is for different ABI version %s (we have %s): %s", + *error_r = t_strdup_printf( + "Module is for different ABI version %s (we have %s): %s", *module_version, set->abi_version, path); module_free(module); - return NULL; + return -1; } /* get our init func */ @@ -248,23 +253,26 @@ if ((module->init == NULL || module->deinit == NULL) && set->require_init_funcs) { - i_error("Module doesn't have %s function: %s", + *error_r = t_strdup_printf( + "Module doesn't have %s function: %s", module->init == NULL ? "init" : "deinit", path); - failed = TRUE; - } else if (!module_check_wrong_binary_dependency(set, module) || - !module_check_missing_plugin_dependencies(set, module, - all_modules)) - failed = TRUE; + } else if (!module_check_wrong_binary_dependency(set, module, error_r)) { + /* failed */ + } else if (!module_check_missing_plugin_dependencies(set, module, + all_modules, error_r)) { + /* failed */ + } - if (failed) { + if (*error_r != NULL) { module->deinit = NULL; module_free(module); - return NULL; + return -1; } if (set->debug) i_debug("Module loaded: %s", path); - return module; + *module_r = module; + return 1; } static int module_name_cmp(const char *const *n1, const char *const *n2) @@ -368,22 +376,26 @@ return TRUE; } -static struct module * -module_dir_load_real(struct module *old_modules, +static int +module_dir_load_real(struct module **_modules, const char *dir, const char **module_names, - const struct module_dir_load_settings *set) + const struct module_dir_load_settings *set, + char **error_r) { DIR *dirp; struct dirent *d; - const char *name, *p, *const *names_p; - struct module *modules, *module, **module_pos; + const char *name, *p, *error, *const *names_p; + struct module *modules, *module, **module_pos, *old_modules = *_modules; unsigned int i, count; ARRAY_TYPE(const_string) names; pool_t pool; + int ret; + + *error_r = NULL; if (module_names != NULL) { if (module_dir_is_all_loaded(old_modules, module_names)) - return old_modules; + return 0; } if (set->debug) @@ -391,21 +403,20 @@ dirp = opendir(dir); if (dirp == NULL) { + *error_r = i_strdup_printf("opendir(%s) failed: %m", dir); if (module_names != NULL) { /* we were given a list of modules to load. we can't fail. */ - i_fatal("opendir(%s) failed: %m", dir); + return -1; } - if (errno != ENOENT) - i_error("opendir(%s) failed: %m", dir); - return NULL; + return errno == ENOENT ? 0 : -1; } pool = pool_alloconly_create("module loader", 4096); p_array_init(&names, pool, 32); modules = NULL; - while ((d = readdir(dirp)) != NULL) { + for (errno = 0; (d = readdir(dirp)) != NULL; errno = 0) { name = d->d_name; if (name[0] == '.') @@ -422,8 +433,14 @@ name = p_strdup(pool, d->d_name); array_append(&names, &name, 1); } - if (closedir(dirp) < 0) - i_error("closedir(%s) failed: %m", dir); + if (errno != 0) + *error_r = i_strdup_printf("readdir(%s) failed: %m", dir); + if (closedir(dirp) < 0 && *error_r == NULL) + *error_r = i_strdup_printf("closedir(%s) failed: %m", dir); + if (*error_r != NULL) { + pool_unref(&pool); + return -1; + } array_sort(&names, module_name_cmp); names_p = array_get(&names, &count); @@ -443,9 +460,12 @@ module = NULL; else { path = t_strconcat(dir, "/", name, NULL); - module = module_load(path, stripped_name, set, modules); - if (module == NULL && module_names != NULL) - i_fatal("Couldn't load required plugins"); + ret = module_load(path, stripped_name, set, modules, &module, &error); + if (ret < 0 && module_names != NULL) { + *error_r = i_strdup_printf("Couldn't load required plugins: %s", error); + t_move_to_parent_frame(); + i = count; + } } if (module != NULL) { @@ -453,26 +473,29 @@ module_pos = &module->next; } } T_END; + pool_unref(&pool); - if (module_names != NULL && !set->ignore_missing) { + if (module_names != NULL && *error_r == NULL && !set->ignore_missing) { /* make sure all modules were found */ for (; *module_names != NULL; module_names++) { if (**module_names != '\0') { - i_fatal("Plugin '%s' not found from directory %s", + *error_r = i_strdup_printf("Plugin '%s' not found from directory %s", *module_names, dir); + break; } } } - pool_unref(&pool); - return modules; + *_modules = modules; + return *error_r != NULL ? -1 : 0; } -struct module * -module_dir_load_missing(struct module *old_modules, - const char *dir, const char *module_names, - const struct module_dir_load_settings *set) +int module_dir_try_load_missing(struct module **modules, + const char *dir, const char *module_names, + const struct module_dir_load_settings *set, + const char **error_r) { - struct module *modules; + char *error = NULL; + int ret; T_BEGIN { const char **arr = NULL; @@ -482,9 +505,29 @@ module_names_fix(arr); } - modules = module_dir_load_real(old_modules, dir, arr, set); + ret = module_dir_load_real(modules, dir, arr, set, &error); } T_END; - return modules; + *error_r = t_strdup(error); + i_free(error); + return ret; +} + +struct module * +module_dir_load_missing(struct module *old_modules, + const char *dir, const char *module_names, + const struct module_dir_load_settings *set) +{ + struct module *new_modules = old_modules; + const char *error; + + if (module_dir_try_load_missing(&new_modules, dir, module_names, + set, &error) < 0) { + if (module_names != NULL) + i_fatal("%s", error); + else + i_error("%s", error); + } + return new_modules; } From dovecot at dovecot.org Fri Feb 6 16:18:36 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 16:18:36 +0000 Subject: dovecot-2.2: lib-storage: Use module_dir_try_load_missing() to l... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f64cf4d075c8 changeset: 18226:f64cf4d075c8 user: Timo Sirainen date: Fri Feb 06 18:18:19 2015 +0200 description: lib-storage: Use module_dir_try_load_missing() to load user plugins. diffstat: src/lib-storage/mail-storage-service.c | 33 ++++++++++++++++++++++----------- 1 files changed, 22 insertions(+), 11 deletions(-) diffs (71 lines): diff -r 88b09d5912c0 -r f64cf4d075c8 src/lib-storage/mail-storage-service.c --- a/src/lib-storage/mail-storage-service.c Fri Feb 06 18:18:10 2015 +0200 +++ b/src/lib-storage/mail-storage-service.c Fri Feb 06 18:18:19 2015 +0200 @@ -968,17 +968,18 @@ auth_master_init(user_set->auth_socket_path, flags)); } -static void +static int mail_storage_service_load_modules(struct mail_storage_service_ctx *ctx, const struct setting_parser_info *user_info, - const struct mail_user_settings *user_set) + const struct mail_user_settings *user_set, + const char **error_r) { struct module_dir_load_settings mod_set; if (*user_set->mail_plugins == '\0') - return; + return 0; if ((ctx->flags & MAIL_STORAGE_SERVICE_FLAG_NO_PLUGINS) != 0) - return; + return 0; memset(&mod_set, 0, sizeof(mod_set)); mod_set.abi_version = DOVECOT_ABI_VERSION; @@ -987,10 +988,10 @@ mod_set.require_init_funcs = TRUE; mod_set.debug = mail_user_set_get_mail_debug(user_info, user_set); - mail_storage_service_modules = - module_dir_load_missing(mail_storage_service_modules, - user_set->mail_plugin_dir, - user_set->mail_plugins, &mod_set); + return module_dir_try_load_missing(&mail_storage_service_modules, + user_set->mail_plugin_dir, + user_set->mail_plugins, + &mod_set, error_r); } static int extra_field_key_cmp_p(const char *const *s1, const char *const *s2) @@ -1086,7 +1087,12 @@ if (ctx->conn == NULL) mail_storage_service_first_init(ctx, user_info, user_set); /* load global plugins */ - mail_storage_service_load_modules(ctx, user_info, user_set); + if (mail_storage_service_load_modules(ctx, user_info, user_set, &error) < 0) { + i_error("%s", error); + pool_unref(&user_pool); + *error_r = MAIL_ERRSTR_CRITICAL_MSG; + return -1; + } if (ctx->userdb_next_pool == NULL) temp_pool = pool_alloconly_create("userdb lookup", 2048); @@ -1152,8 +1158,13 @@ /* load per-user plugins */ if (ret > 0) { - mail_storage_service_load_modules(ctx, user_info, - user->user_set); + if (mail_storage_service_load_modules(ctx, user_info, + user->user_set, + &error) < 0) { + i_error("%s", error); + *error_r = MAIL_ERRSTR_CRITICAL_MSG; + ret = -2; + } } *user_r = user; From dovecot at dovecot.org Fri Feb 6 17:03:32 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 06 Feb 2015 17:03:32 +0000 Subject: dovecot-2.2: lib: Accidentally committed a nonexistent function ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/3ae5d3a2a7d7 changeset: 18227:3ae5d3a2a7d7 user: Timo Sirainen date: Fri Feb 06 19:03:20 2015 +0200 description: lib: Accidentally committed a nonexistent function call in previous commit. diffstat: src/lib/module-dir.c | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diffs (11 lines): diff -r f64cf4d075c8 -r 3ae5d3a2a7d7 src/lib/module-dir.c --- a/src/lib/module-dir.c Fri Feb 06 18:18:19 2015 +0200 +++ b/src/lib/module-dir.c Fri Feb 06 19:03:20 2015 +0200 @@ -463,7 +463,6 @@ ret = module_load(path, stripped_name, set, modules, &module, &error); if (ret < 0 && module_names != NULL) { *error_r = i_strdup_printf("Couldn't load required plugins: %s", error); - t_move_to_parent_frame(); i = count; } } From dovecot at dovecot.org Sat Feb 7 16:15:04 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 07 Feb 2015 16:15:04 +0000 Subject: dovecot-2.2: log: Minor cleanup: Update ioloop_timeval whenever ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2f15ae9bc1b3 changeset: 18228:2f15ae9bc1b3 user: Timo Sirainen date: Sat Feb 07 18:14:53 2015 +0200 description: log: Minor cleanup: Update ioloop_timeval whenever reading input from log client. diffstat: src/log/log-connection.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diffs (14 lines): diff -r 3ae5d3a2a7d7 -r 2f15ae9bc1b3 src/log/log-connection.c --- a/src/log/log-connection.c Fri Feb 06 19:03:20 2015 +0200 +++ b/src/log/log-connection.c Sat Feb 07 18:14:53 2015 +0200 @@ -296,8 +296,8 @@ while ((ret = i_stream_read(log->input)) > 0 || ret == -2) { /* get new timestamps for every read() */ - if (gettimeofday(&now, NULL) < 0) - i_panic("gettimeofday() failed: %m"); + io_loop_time_refresh(); + now = ioloop_timeval; tm = *localtime(&now.tv_sec); while ((line = i_stream_next_line(log->input)) != NULL) From dovecot at dovecot.org Sat Feb 7 16:18:11 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 07 Feb 2015 16:18:11 +0000 Subject: dovecot-2.2: log: Small cleanup: Include log connection prefix w... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/46e28c6ef002 changeset: 18229:46e28c6ef002 user: Timo Sirainen date: Sat Feb 07 18:17:59 2015 +0200 description: log: Small cleanup: Include log connection prefix when logging read() errors. diffstat: src/log/log-connection.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diffs (37 lines): diff -r 2f15ae9bc1b3 -r 46e28c6ef002 src/log/log-connection.c --- a/src/log/log-connection.c Sat Feb 07 18:14:53 2015 +0200 +++ b/src/log/log-connection.c Sat Feb 07 18:17:59 2015 +0200 @@ -243,7 +243,7 @@ ret = i_stream_read(log->input); if (ret < 0) { - i_error("read(log pipe) failed: %m"); + i_error("read(log %s) failed: %m", log->default_prefix); return -1; } if ((size_t)ret < sizeof(handshake)) { @@ -264,6 +264,7 @@ i_error("Missing prefix data in handshake"); return -1; } + i_free(log->default_prefix); log->default_prefix = i_strndup(data + sizeof(handshake), handshake.prefix_len); i_stream_skip(log->input, sizeof(handshake) + handshake.prefix_len); @@ -307,7 +308,7 @@ if (log->input->eof) log_connection_destroy(log); else if (log->input->stream_errno != 0) { - i_error("read(log pipe) failed: %m"); + i_error("read(log %s) failed: %m", log->default_prefix); log_connection_destroy(log); } else { i_assert(!log->input->closed); @@ -325,6 +326,7 @@ log->listen_fd = listen_fd; log->io = io_add(fd, IO_READ, log_connection_input, log); log->input = i_stream_create_fd(fd, PIPE_BUF, FALSE); + log->default_prefix = i_strdup_printf("listen_fd %d", listen_fd); hash_table_create_direct(&log->clients, default_pool, 0); array_idx_set(&logs_by_fd, listen_fd, &log); From dovecot at dovecot.org Sat Feb 7 16:39:17 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 07 Feb 2015 16:39:17 +0000 Subject: dovecot-2.2: log: Don't spend more than 100 msecs at a time logg... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/787ce1ab9f85 changeset: 18230:787ce1ab9f85 user: Timo Sirainen date: Sat Feb 07 18:23:58 2015 +0200 description: log: Don't spend more than 100 msecs at a time logging one connection. diffstat: src/log/log-connection.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-) diffs (48 lines): diff -r 46e28c6ef002 -r 787ce1ab9f85 src/log/log-connection.c --- a/src/log/log-connection.c Sat Feb 07 18:17:59 2015 +0200 +++ b/src/log/log-connection.c Sat Feb 07 18:23:58 2015 +0200 @@ -6,6 +6,7 @@ #include "istream.h" #include "llist.h" #include "hash.h" +#include "time-util.h" #include "master-interface.h" #include "master-service.h" #include "log-error-buffer.h" @@ -16,6 +17,7 @@ #include #define FATAL_QUEUE_TIMEOUT_MSECS 500 +#define MAX_MSECS_PER_CONNECTION 100 struct log_client { struct ip_addr ip; @@ -285,7 +287,7 @@ { const char *line; ssize_t ret; - struct timeval now; + struct timeval now, start_timeval; struct tm tm; if (!log->handshaked) { @@ -295,14 +297,18 @@ } } + io_loop_time_refresh(); + start_timeval = ioloop_timeval; while ((ret = i_stream_read(log->input)) > 0 || ret == -2) { /* get new timestamps for every read() */ - io_loop_time_refresh(); now = ioloop_timeval; tm = *localtime(&now.tv_sec); while ((line = i_stream_next_line(log->input)) != NULL) log_it(log, line, &now, &tm); + io_loop_time_refresh(); + if (timeval_diff_msecs(&ioloop_timeval, &start_timeval) > MAX_MSECS_PER_CONNECTION) + break; } if (log->input->eof) From dovecot at dovecot.org Sat Feb 7 16:39:17 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 07 Feb 2015 16:39:17 +0000 Subject: dovecot-2.2: log: Log a warning if some connection is logging fa... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/dec13d8f524c changeset: 18231:dec13d8f524c user: Timo Sirainen date: Sat Feb 07 18:39:05 2015 +0200 description: log: Log a warning if some connection is logging faster than we can write. diffstat: src/log/log-connection.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diffs (41 lines): diff -r 787ce1ab9f85 -r dec13d8f524c src/log/log-connection.c --- a/src/log/log-connection.c Sat Feb 07 18:23:58 2015 +0200 +++ b/src/log/log-connection.c Sat Feb 07 18:39:05 2015 +0200 @@ -19,6 +19,12 @@ #define FATAL_QUEUE_TIMEOUT_MSECS 500 #define MAX_MSECS_PER_CONNECTION 100 +/* Log a warning after 1 secs when we've been all the time busy writing the + log connection. */ +#define LOG_WARN_PENDING_COUNT (1000 / MAX_MSECS_PER_CONNECTION) +/* If we keep beeing busy, log a warning every 60 seconds. */ +#define LOG_WARN_PENDING_INTERVAL (60 * LOG_WARN_PENDING_COUNT) + struct log_client { struct ip_addr ip; char *prefix; @@ -37,6 +43,8 @@ char *default_prefix; HASH_TABLE(void *, struct log_client *) clients; + unsigned int pending_count; + unsigned int master:1; unsigned int handshaked:1; }; @@ -318,6 +326,15 @@ log_connection_destroy(log); } else { i_assert(!log->input->closed); + if (ret == 0) + log->pending_count = 0; + else if (++log->pending_count >= LOG_WARN_PENDING_COUNT) { + if (log->pending_count == LOG_WARN_PENDING_COUNT || + (log->pending_count % LOG_WARN_PENDING_INTERVAL) == 0) { + i_warning("Log connection fd %d listen_fd %d prefix '%s' is sending input faster than we can write", + log->fd, log->listen_fd, log->default_prefix); + } + } } } From dovecot at dovecot.org Mon Feb 9 02:03:26 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 02:03:26 +0000 Subject: dovecot-2.2: log: If too much logging data is sent, show it in p... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/8f7dee2e7e05 changeset: 18232:8f7dee2e7e05 user: Timo Sirainen date: Mon Feb 09 04:03:12 2015 +0200 description: log: If too much logging data is sent, show it in process title. diffstat: src/log/log-connection.c | 55 ++++++++++++++++++++++++++++++++++++++----- src/log/log-connection.h | 2 + src/log/main.c | 3 ++ src/master/service-process.c | 2 + 4 files changed, 55 insertions(+), 7 deletions(-) diffs (127 lines): diff -r dec13d8f524c -r 8f7dee2e7e05 src/log/log-connection.c --- a/src/log/log-connection.c Sat Feb 07 18:39:05 2015 +0200 +++ b/src/log/log-connection.c Mon Feb 09 04:03:12 2015 +0200 @@ -7,6 +7,7 @@ #include "llist.h" #include "hash.h" #include "time-util.h" +#include "process-title.h" #include "master-interface.h" #include "master-service.h" #include "log-error-buffer.h" @@ -51,9 +52,37 @@ static struct log_connection *log_connections = NULL; static ARRAY(struct log_connection *) logs_by_fd; +static unsigned int global_pending_count; +static struct log_connection *last_pending_log; static void log_connection_destroy(struct log_connection *log); +static void log_refresh_proctitle(void) +{ + if (!verbose_proctitle) + return; + + if (global_pending_count == 0) + process_title_set(""); + else if (last_pending_log == NULL) { + process_title_set(t_strdup_printf( + "[%u services too fast]", global_pending_count)); + } else if (global_pending_count > 1) { + process_title_set(t_strdup_printf( + "[%u services too fast, last: %d/%d/%s]", + global_pending_count, + last_pending_log->fd, + last_pending_log->listen_fd, + last_pending_log->default_prefix)); + } else { + process_title_set(t_strdup_printf( + "[service too fast: %d/%d/%s]", + last_pending_log->fd, + last_pending_log->listen_fd, + last_pending_log->default_prefix)); + } +} + static struct log_client *log_client_get(struct log_connection *log, pid_t pid) { struct log_client *client; @@ -326,14 +355,26 @@ log_connection_destroy(log); } else { i_assert(!log->input->closed); - if (ret == 0) - log->pending_count = 0; - else if (++log->pending_count >= LOG_WARN_PENDING_COUNT) { - if (log->pending_count == LOG_WARN_PENDING_COUNT || - (log->pending_count % LOG_WARN_PENDING_INTERVAL) == 0) { - i_warning("Log connection fd %d listen_fd %d prefix '%s' is sending input faster than we can write", - log->fd, log->listen_fd, log->default_prefix); + if (ret == 0) { + if (log->pending_count > 0) { + log->pending_count = 0; + i_assert(global_pending_count > 0); + global_pending_count--; + if (log == last_pending_log) + last_pending_log = NULL; + log_refresh_proctitle(); } + return; + } + last_pending_log = log; + if (log->pending_count++ == 0) { + global_pending_count++; + log_refresh_proctitle(); + } + if (log->pending_count == LOG_WARN_PENDING_COUNT || + (log->pending_count % LOG_WARN_PENDING_INTERVAL) == 0) { + i_warning("Log connection fd %d listen_fd %d prefix '%s' is sending input faster than we can write", + log->fd, log->listen_fd, log->default_prefix); } } } diff -r dec13d8f524c -r 8f7dee2e7e05 src/log/log-connection.h --- a/src/log/log-connection.h Sat Feb 07 18:39:05 2015 +0200 +++ b/src/log/log-connection.h Mon Feb 09 04:03:12 2015 +0200 @@ -3,6 +3,8 @@ struct log_connection; +extern bool verbose_proctitle; + void log_connection_create(struct log_error_buffer *errorbuf, int fd, int listen_fd); diff -r dec13d8f524c -r 8f7dee2e7e05 src/log/main.c --- a/src/log/main.c Sat Feb 07 18:39:05 2015 +0200 +++ b/src/log/main.c Mon Feb 09 04:03:12 2015 +0200 @@ -12,6 +12,7 @@ #include +bool verbose_proctitle; static struct log_error_buffer *errorbuf; static void @@ -71,6 +72,8 @@ i_fatal("Error reading configuration: %s", error); master_service_init_log(master_service, "log: "); + verbose_proctitle = master_service_settings_get(master_service)->verbose_proctitle; + restrict_access_by_env(NULL, FALSE); restrict_access_allow_coredumps(TRUE); diff -r dec13d8f524c -r 8f7dee2e7e05 src/master/service-process.c --- a/src/master/service-process.c Sat Feb 07 18:39:05 2015 +0200 +++ b/src/master/service-process.c Mon Feb 09 04:03:12 2015 +0200 @@ -227,6 +227,8 @@ env_put(t_strconcat("DEBUG_LOG_PATH=", set->debug_log_path, NULL)); env_put(t_strconcat("LOG_TIMESTAMP=", set->log_timestamp, NULL)); env_put(t_strconcat("SYSLOG_FACILITY=", set->syslog_facility, NULL)); + if (set->verbose_proctitle) + env_put("VERBOSE_PROCTITLE=1"); env_put("SSL=no"); break; default: From dovecot at dovecot.org Mon Feb 9 13:52:25 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 13:52:25 +0000 Subject: dovecot-2.2: doveadm: Reverted changeset 3b89f2f4ffb3, which avo... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/acebb486714e changeset: 18233:acebb486714e user: Timo Sirainen date: Mon Feb 09 13:03:47 2015 +0200 description: doveadm: Reverted changeset 3b89f2f4ffb3, which avoided calling init() for proxied commands. The init() is required for all commands that print something, because otherwise the doveadm_print_header*() isn't called and the commands crash. The reason for the original change was that some commands' init() attempted to do too much work on the proxy, which ended up failing. So looks like the proper fix is to just remove this kind of extra work from init(). diffstat: src/doveadm/doveadm-mail.c | 18 +++++++----------- 1 files changed, 7 insertions(+), 11 deletions(-) diffs (49 lines): diff -r 8f7dee2e7e05 -r acebb486714e src/doveadm/doveadm-mail.c --- a/src/doveadm/doveadm-mail.c Mon Feb 09 04:03:12 2015 +0200 +++ b/src/doveadm/doveadm-mail.c Mon Feb 09 13:03:47 2015 +0200 @@ -277,7 +277,7 @@ } static int -doveadm_mail_next_user(struct doveadm_mail_cmd_context *ctx, bool init_user, +doveadm_mail_next_user(struct doveadm_mail_cmd_context *ctx, const struct mail_storage_service_input *input, const char **error_r) { @@ -296,14 +296,6 @@ if (ret != 0) return ret; - if (init_user) { - /* call init() after we've checked whether we want to do this - via doveadm-server or not */ - ctx->v.init(ctx, ctx->args); - if (hook_doveadm_mail_init != NULL) - hook_doveadm_mail_init(ctx); - } - ret = mail_storage_service_lookup(ctx->storage_service, input, &ctx->cur_service_user, &error); if (ret <= 0) { @@ -349,7 +341,11 @@ ctx->storage_service_input = *input; ctx->storage_service = mail_storage_service_init(master_service, NULL, ctx->service_flags); - return doveadm_mail_next_user(ctx, TRUE, input, error_r); + ctx->v.init(ctx, ctx->args); + if (hook_doveadm_mail_init != NULL) + hook_doveadm_mail_init(ctx); + + return doveadm_mail_next_user(ctx, input, error_r); } static void sig_die(const siginfo_t *si, void *context ATTR_UNUSED) @@ -394,7 +390,7 @@ ctx->cur_username = user; doveadm_print_sticky("username", user); T_BEGIN { - ret = doveadm_mail_next_user(ctx, FALSE, &input, &error); + ret = doveadm_mail_next_user(ctx, &input, &error); if (ret < 0) i_error("%s", error); else if (ret == 0) From dovecot at dovecot.org Mon Feb 9 13:52:25 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 13:52:25 +0000 Subject: dovecot-2.2: doveadm: Added comments to struct doveadm_mail_cmd_... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/111f19992c06 changeset: 18234:111f19992c06 user: Timo Sirainen date: Mon Feb 09 15:21:55 2015 +0200 description: doveadm: Added comments to struct doveadm_mail_cmd_vfuncs diffstat: src/doveadm/doveadm-mail.h | 22 +++++++++++++++++++++- 1 files changed, 21 insertions(+), 1 deletions(-) diffs (42 lines): diff -r acebb486714e -r 111f19992c06 src/doveadm/doveadm-mail.h --- a/src/doveadm/doveadm-mail.h Mon Feb 09 13:03:47 2015 +0200 +++ b/src/doveadm/doveadm-mail.h Mon Feb 09 15:21:55 2015 +0200 @@ -15,17 +15,37 @@ struct doveadm_mail_cmd_context; struct doveadm_mail_cmd_vfuncs { - bool (*parse_arg)(struct doveadm_mail_cmd_context *ctx,int c); + /* Parse one getopt() parameter. This is called for each parameter. */ + bool (*parse_arg)(struct doveadm_mail_cmd_context *ctx, int c); + /* Usually not needed. The preinit() is called just after parsing all + parameters, but before any userdb lookups are done. This allows the + preinit() to alter the userdb lookup behavior (especially + service_flags). */ void (*preinit)(struct doveadm_mail_cmd_context *ctx); + /* Initialize the command. Most importantly if the function prints + anything, this should initialize the headers. It shouldn't however + do any actual work. The init() is called also when doveadm is + performing the work via doveadm-server, which could be running + remotely with completely different Dovecot configuration. */ void (*init)(struct doveadm_mail_cmd_context *ctx, const char *const args[]); + /* Usually not needed. When iterating through multiple users, use this + function to get the next username. Overriding this is usually done + only when there's a known username filter, such as the expire + plugin. */ int (*get_next_user)(struct doveadm_mail_cmd_context *ctx, const char **username_r); + /* Usually not needed. This is called between + mail_storage_service_lookup() and mail_storage_service_next() for + each user. */ int (*prerun)(struct doveadm_mail_cmd_context *ctx, struct mail_storage_service_user *service_user, const char **error_r); + /* This is the main function which performs all the work for the + command. This is called once per each user. */ int (*run)(struct doveadm_mail_cmd_context *ctx, struct mail_user *mail_user); + /* Deinitialize the command. Called once at the end. */ void (*deinit)(struct doveadm_mail_cmd_context *ctx); }; From dovecot at dovecot.org Mon Feb 9 13:52:25 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 13:52:25 +0000 Subject: dovecot-2.2: doveadm copy/move/import: Delay initializing source... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/b4662ccd0b5e changeset: 18235:b4662ccd0b5e user: Timo Sirainen date: Mon Feb 09 15:38:42 2015 +0200 description: doveadm copy/move/import: Delay initializing source user until run() This fixes using them with doveadm proxying. diffstat: src/doveadm/doveadm-mail-copymove.c | 10 ++++-- src/doveadm/doveadm-mail-import.c | 53 +++++++++++++++++++++--------------- 2 files changed, 38 insertions(+), 25 deletions(-) diffs (141 lines): diff -r 111f19992c06 -r b4662ccd0b5e src/doveadm/doveadm-mail-copymove.c --- a/src/doveadm/doveadm-mail-copymove.c Mon Feb 09 15:21:55 2015 +0200 +++ b/src/doveadm/doveadm-mail-copymove.c Mon Feb 09 15:38:42 2015 +0200 @@ -13,6 +13,7 @@ struct copy_cmd_context { struct doveadm_mail_cmd_context ctx; + const char *source_username; struct mail_storage_service_user *source_service_user; struct mail_user *source_user; @@ -71,13 +72,13 @@ } static void -cmd_copy_alloc_source_user(struct copy_cmd_context *ctx, const char *username) +cmd_copy_alloc_source_user(struct copy_cmd_context *ctx) { struct mail_storage_service_input input; const char *error; input = ctx->ctx.storage_service_input; - input.username = username; + input.username = ctx->source_username; if (mail_storage_service_lookup_next(ctx->ctx.storage_service, &input, &ctx->source_service_user, @@ -100,6 +101,9 @@ const struct mailbox_info *info; int ret = 0; + if (ctx->source_username != NULL && ctx->source_user == NULL) + cmd_copy_alloc_source_user(ctx); + ns = mail_namespace_find(user->namespaces, ctx->destname); destbox = mailbox_alloc(ns->list, ctx->destname, MAILBOX_FLAG_SAVEONLY); if (mailbox_open(destbox) < 0) { @@ -146,7 +150,7 @@ MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP) == 0) i_fatal("Use -u parameter to specify destination user"); - cmd_copy_alloc_source_user(ctx, args[1]); + ctx->source_username = p_strdup(_ctx->pool, args[1]); args += 2; } diff -r 111f19992c06 -r b4662ccd0b5e src/doveadm/doveadm-mail-import.c --- a/src/doveadm/doveadm-mail-import.c Mon Feb 09 15:21:55 2015 +0200 +++ b/src/doveadm/doveadm-mail-import.c Mon Feb 09 15:38:42 2015 +0200 @@ -13,6 +13,7 @@ struct import_cmd_context { struct doveadm_mail_cmd_context ctx; + const char *src_location; struct mail_user *src_user; const char *dest_parent; bool subscribe; @@ -152,6 +153,30 @@ return ret; } +static void cmd_import_init_source_user(struct import_cmd_context *ctx) +{ + struct mail_storage_service_input input; + struct mail_storage_service_user *service_user; + struct mail_user *user; + const char *error; + + /* create a user for accessing the source storage */ + memset(&input, 0, sizeof(input)); + input.module = "mail"; + input.username = "doveadm"; + input.flags_override_add = MAIL_STORAGE_SERVICE_FLAG_NO_NAMESPACES | + MAIL_STORAGE_SERVICE_FLAG_NO_RESTRICT_ACCESS; + input.flags_override_remove = MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP; + if (mail_storage_service_lookup_next(ctx->ctx.storage_service, &input, + &service_user, &user, &error) < 0) + i_fatal("Import user initialization failed: %s", error); + if (mail_namespaces_init_location(user, ctx->src_location, &error) < 0) + i_fatal("Import namespace initialization failed: %s", error); + + ctx->src_user = user; + mail_storage_service_user_free(&service_user); +} + static int cmd_import_run(struct doveadm_mail_cmd_context *_ctx, struct mail_user *user) { @@ -163,6 +188,9 @@ const struct mailbox_info *info; int ret = 0; + if (ctx->src_user == NULL) + cmd_import_init_source_user(ctx); + iter = doveadm_mailbox_list_iter_init(_ctx, ctx->src_user, _ctx->search_args, iter_flags); while ((info = doveadm_mailbox_list_iter_next(iter)) != NULL) T_BEGIN { @@ -178,39 +206,20 @@ const char *const args[]) { struct import_cmd_context *ctx = (struct import_cmd_context *)_ctx; - struct mail_storage_service_input input; - struct mail_storage_service_user *service_user; - struct mail_user *user; - const char *src_location, *error; if (str_array_length(args) < 3) doveadm_mail_help_name("import"); - src_location = args[0]; + ctx->src_location = p_strdup(_ctx->pool, args[0]); ctx->dest_parent = p_strdup(_ctx->pool, args[1]); ctx->ctx.search_args = doveadm_mail_build_search_args(args+2); - - /* create a user for accessing the source storage */ - memset(&input, 0, sizeof(input)); - input.module = "mail"; - input.username = "doveadm"; - input.flags_override_add = MAIL_STORAGE_SERVICE_FLAG_NO_NAMESPACES | - MAIL_STORAGE_SERVICE_FLAG_NO_RESTRICT_ACCESS; - input.flags_override_remove = MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP; - if (mail_storage_service_lookup_next(ctx->ctx.storage_service, &input, - &service_user, &user, &error) < 0) - i_fatal("Import user initialization failed: %s", error); - if (mail_namespaces_init_location(user, src_location, &error) < 0) - i_fatal("Import namespace initialization failed: %s", error); - - ctx->src_user = user; - mail_storage_service_user_free(&service_user); } static void cmd_import_deinit(struct doveadm_mail_cmd_context *_ctx) { struct import_cmd_context *ctx = (struct import_cmd_context *)_ctx; - mail_user_unref(&ctx->src_user); + if (ctx->src_user != NULL) + mail_user_unref(&ctx->src_user); } static bool cmd_import_parse_arg(struct doveadm_mail_cmd_context *_ctx, int c) From dovecot at dovecot.org Mon Feb 9 19:58:48 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 19:58:48 +0000 Subject: dovecot-2.2: dsync: Fixed assert-crash when the two mailbox tree... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/0eb8d396949a changeset: 18236:0eb8d396949a user: Timo Sirainen date: Mon Feb 09 21:45:04 2015 +0200 description: dsync: Fixed assert-crash when the two mailbox trees differed only by their directory names. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 41 ++++++++++++++++++++++++---- 1 files changed, 34 insertions(+), 7 deletions(-) diffs (76 lines): diff -r b4662ccd0b5e -r 0eb8d396949a src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Mon Feb 09 15:38:42 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Mon Feb 09 21:45:04 2015 +0200 @@ -498,8 +498,8 @@ } } -static int node_mailbox_guids_cmp2(struct dsync_mailbox_node *node1, - struct dsync_mailbox_node *node2) +static int node_mailbox_guids_cmp(struct dsync_mailbox_node *node1, + struct dsync_mailbox_node *node2) { int ret; @@ -516,8 +516,8 @@ if (ret != 0) return ret; - ret = node_mailbox_guids_cmp2(node1->first_child, - node2->first_child); + ret = node_mailbox_guids_cmp(node1->first_child, + node2->first_child); if (ret != 0) return ret; node1 = node1->next; @@ -528,12 +528,39 @@ return node1 != NULL ? -1 : 1; } -static int node_mailbox_guids_cmp(struct dsync_mailbox_node *node1, +static int node_mailbox_names_cmp(struct dsync_mailbox_node *node1, struct dsync_mailbox_node *node2) { int ret; - ret = node_mailbox_guids_cmp2(node1, node2); + while (node1 != NULL && node2 != NULL) { + ret = strcmp(node1->name, node2->name); + if (ret != 0) + return ret; + + ret = node_mailbox_names_cmp(node1->first_child, + node2->first_child); + if (ret != 0) + return ret; + node1 = node1->next; + node2 = node2->next; + } + if (node1 == NULL && node2 == NULL) + return 0; + return node1 != NULL ? -1 : 1; +} + +static int node_mailbox_trees_cmp(struct dsync_mailbox_node *node1, + struct dsync_mailbox_node *node2) +{ + int ret; + + ret = node_mailbox_guids_cmp(node1, node2); + if (ret == 0) { + /* only a directory name changed and all the timestamps + are equal. just pick the alphabetically smaller. */ + ret = node_mailbox_names_cmp(node1, node2); + } i_assert(ret != 0); return ret; } @@ -585,7 +612,7 @@ children. */ if (local_ts > remote_ts || (local_ts == remote_ts && - node_mailbox_guids_cmp(local_node1, remote_node2) < 0)) { + node_mailbox_trees_cmp(local_node1, remote_node2) < 0)) { /* local nodes have a higher timestamp. we only want to do renames where the destination parent is the current node's (local_node1/remote_node2) parent. */ From dovecot at dovecot.org Mon Feb 9 19:58:49 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Mon, 09 Feb 2015 19:58:49 +0000 Subject: dovecot-2.2: dsync: Don't try to rename namespace prefixes. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/1a0ece9b39e5 changeset: 18237:1a0ece9b39e5 user: Timo Sirainen date: Mon Feb 09 21:58:13 2015 +0200 description: dsync: Don't try to rename namespace prefixes. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 19 +++++++-- src/doveadm/dsync/test-dsync-mailbox-tree-sync.c | 46 ++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 4 deletions(-) diffs (109 lines): diff -r 0eb8d396949a -r 1a0ece9b39e5 src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Mon Feb 09 21:45:04 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Mon Feb 09 21:58:13 2015 +0200 @@ -759,6 +759,14 @@ return NULL; } +static bool sync_node_is_namespace_root(struct dsync_mailbox_tree *tree, + struct dsync_mailbox_node *node) +{ + if (node == &tree->root) + return TRUE; + return sync_node_is_namespace_prefix(tree, node); +} + static bool sync_rename_directory(struct dsync_mailbox_tree_sync_ctx *ctx, struct dsync_mailbox_node *local_node1, struct dsync_mailbox_node *remote_node2) @@ -766,15 +774,18 @@ struct dsync_mailbox_node *remote_node1, *local_node2; /* see if we can find matching mailbox branches based on the nodes' - child mailboxes (with GUIDs). */ + child mailboxes (with GUIDs). we can then rename the entire branch. + don't try to do this for namespace prefixes though. */ remote_node1 = sync_find_branch(ctx->local_tree, ctx->remote_tree, local_node1); local_node2 = sync_find_branch(ctx->remote_tree, ctx->local_tree, remote_node2); if (remote_node1 == NULL || local_node2 == NULL || - remote_node1 == &ctx->remote_tree->root || - local_node2 == &ctx->local_tree->root || - node_names_equal(remote_node1, local_node2)) + node_names_equal(remote_node1, local_node2) || + sync_node_is_namespace_root(ctx->remote_tree, remote_node1) || + sync_node_is_namespace_root(ctx->remote_tree, remote_node2) || + sync_node_is_namespace_root(ctx->local_tree, local_node1) || + sync_node_is_namespace_root(ctx->local_tree, local_node2)) return FALSE; return sync_rename_lower_ts(ctx, local_node1, remote_node1, diff -r 0eb8d396949a -r 1a0ece9b39e5 src/doveadm/dsync/test-dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/test-dsync-mailbox-tree-sync.c Mon Feb 09 21:45:04 2015 +0200 +++ b/src/doveadm/dsync/test-dsync-mailbox-tree-sync.c Mon Feb 09 21:58:13 2015 +0200 @@ -11,6 +11,12 @@ #include #define MAX_DEPTH 4 +#define TEST_NAMESPACE_NAME "INBOX" + +static struct mail_namespace inbox_namespace = { + .prefix = TEST_NAMESPACE_NAME"/", + .prefix_len = sizeof(TEST_NAMESPACE_NAME)-1 + 1 +}; char mail_namespace_get_sep(struct mail_namespace *ns ATTR_UNUSED) { @@ -212,11 +218,51 @@ dsync_mailbox_tree_deinit(&orig_tree2); } +static void +test_tree_nodes_add_namespace(struct dsync_mailbox_node *node, + struct mail_namespace *ns) +{ + for (; node != NULL; node = node->next) { + node->ns = ns; + test_tree_nodes_add_namespace(node->first_child, ns); + } +} + +static void +test_tree_add_namespace(struct dsync_mailbox_tree *tree, + struct mail_namespace *ns) +{ + struct dsync_mailbox_node *node, *n; + + node = dsync_mailbox_tree_get(tree, TEST_NAMESPACE_NAME); + node->existence = DSYNC_MAILBOX_NODE_EXISTS; + i_assert(tree->root.first_child == node); + i_assert(node->first_child == NULL); + node->first_child = node->next; + for (n = node->first_child; n != NULL; n = n->next) + n->parent = node; + node->next = NULL; + + test_tree_nodes_add_namespace(&tree->root, ns); +} + static void test_trees(struct dsync_mailbox_tree *tree1, struct dsync_mailbox_tree *tree2) { + struct dsync_mailbox_tree *tree1_dup, *tree2_dup; + + tree1_dup = dsync_mailbox_tree_dup(tree1); + tree2_dup = dsync_mailbox_tree_dup(tree2); + + /* test without namespace prefix */ test_trees_nofree(tree1, &tree2); dsync_mailbox_tree_deinit(&tree1); + + /* test with namespace prefix */ + test_tree_add_namespace(tree1_dup, &inbox_namespace); + test_tree_add_namespace(tree2_dup, &inbox_namespace); + test_trees_nofree(tree1_dup, &tree2_dup); + dsync_mailbox_tree_deinit(&tree1_dup); } static void test_dsync_mailbox_tree_sync_creates(void) From dovecot at dovecot.org Tue Feb 10 10:31:45 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 10 Feb 2015 10:31:45 +0000 Subject: dovecot-2.2: lib: istream create helpers for common cases Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/43a61a8bf9c5 changeset: 18238:43a61a8bf9c5 user: Phil Carmody date: Tue Feb 10 12:31:12 2015 +0200 description: lib: istream create helpers for common cases Several clients want to create streams from buffer_t and string_t, we may as well make it easy for them Signed-off-by: Phil Carmody diffstat: src/lib/istream.h | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diffs (14 lines): diff -r 1a0ece9b39e5 -r 43a61a8bf9c5 src/lib/istream.h --- a/src/lib/istream.h Mon Feb 09 21:58:13 2015 +0200 +++ b/src/lib/istream.h Tue Feb 10 12:31:12 2015 +0200 @@ -33,6 +33,10 @@ uoff_t start_offset, uoff_t v_size, bool autoclose_fd); struct istream *i_stream_create_from_data(const void *data, size_t size); +#define i_stream_create_from_buffer(buf) \ + i_stream_create_from_data((buf)->data, (buf)->used) +#define i_stream_create_from_string(str) \ + i_stream_create_from_data(str_data(str), str_len(str)) struct istream *i_stream_create_limit(struct istream *input, uoff_t v_size); struct istream *i_stream_create_range(struct istream *input, uoff_t v_offset, uoff_t v_size); From dovecot at dovecot.org Tue Feb 10 11:12:57 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 10 Feb 2015 11:12:57 +0000 Subject: dovecot-2.2: dovecot.m4: External plugins can now more easily ru... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d938a49c2045 changeset: 18239:d938a49c2045 user: Timo Sirainen date: Tue Feb 10 13:12:42 2015 +0200 description: dovecot.m4: External plugins can now more easily run their tests via Valgrind. DC_DOVECOT macro automatically adds RUN_TEST variable to Makefiles, which can be used to call any test programs. If valgrind exists, the tests are run through it. This is done by writing run-test.sh to the build directory, so the original run-test.sh in hg is no longer needed. diffstat: .hgignore | 1 + configure.ac | 8 +------- dovecot.m4 | 30 ++++++++++++++++++++++++++++++ run-test.sh | 15 --------------- 4 files changed, 32 insertions(+), 22 deletions(-) diffs (93 lines): diff -r 43a61a8bf9c5 -r d938a49c2045 .hgignore --- a/.hgignore Tue Feb 10 12:31:12 2015 +0200 +++ b/.hgignore Tue Feb 10 13:12:42 2015 +0200 @@ -36,6 +36,7 @@ ChangeLog Makefile Makefile.in +run-test.sh *.o *.lo diff -r 43a61a8bf9c5 -r d938a49c2045 configure.ac --- a/configure.ac Tue Feb 10 12:31:12 2015 +0200 +++ b/configure.ac Tue Feb 10 13:12:42 2015 +0200 @@ -2816,13 +2816,7 @@ fi AC_SUBST(docdir) -AC_CHECK_PROG(VALGRIND, valgrind, yes, no) -if test $VALGRIND = yes; then - RUN_TEST='$(SHELL) $(top_srcdir)/run-test.sh' -else - RUN_TEST='' -fi -AC_SUBST(RUN_TEST) +DC_DOVECOT_TEST_WRAPPER AC_SUBST(abs_top_builddir) AC_CONFIG_HEADERS([config.h]) diff -r 43a61a8bf9c5 -r d938a49c2045 dovecot.m4 --- a/dovecot.m4 Tue Feb 10 12:31:12 2015 +0200 +++ b/dovecot.m4 Tue Feb 10 13:12:42 2015 +0200 @@ -31,6 +31,35 @@ unset _plugin_deps ]) +AC_DEFUN([DC_DOVECOT_TEST_WRAPPER],[ + AC_CHECK_PROG(VALGRIND, valgrind, yes, no) + if test $VALGRIND = yes; then + cat > run-test.sh < details: http://hg.dovecot.org/dovecot-2.2/rev/51d04f935e61 changeset: 18240:51d04f935e61 user: Timo Sirainen date: Tue Feb 10 17:29:31 2015 +0200 description: dovecot-config: Added lib-sasl path into LIBDOVECOT_INCLUDE. diffstat: dovecot-config.in.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r d938a49c2045 -r 51d04f935e61 dovecot-config.in.in --- a/dovecot-config.in.in Tue Feb 10 13:12:42 2015 +0200 +++ b/dovecot-config.in.in Tue Feb 10 17:29:31 2015 +0200 @@ -20,7 +20,7 @@ LIBDOVECOT_STORAGE_DEPS="@LIBDOVECOT_STORAGE_DEPS@" LIBDOVECOT_DSYNC_DEPS="@LIBDOVECOT_DSYNC@" -LIBDOVECOT_INCLUDE="-I$(incdir) -I$(incdir)/src/lib -I$(incdir)/src/lib-dict -I$(incdir)/src/lib-dns -I$(incdir)/src/lib-http -I$(incdir)/src/lib-mail -I$(incdir)/src/lib-imap -I$(incdir)/src/lib-fs -I$(incdir)/src/lib-charset -I$(incdir)/src/lib-auth -I$(incdir)/src/lib-master -I$(incdir)/src/lib-ssl-iostream -I$(incdir)/src/lib-compression -I$(incdir)/src/lib-settings -I$(incdir)/src/lib-test" +LIBDOVECOT_INCLUDE="-I$(incdir) -I$(incdir)/src/lib -I$(incdir)/src/lib-dict -I$(incdir)/src/lib-dns -I$(incdir)/src/lib-http -I$(incdir)/src/lib-mail -I$(incdir)/src/lib-imap -I$(incdir)/src/lib-fs -I$(incdir)/src/lib-charset -I$(incdir)/src/lib-auth -I$(incdir)/src/lib-master -I$(incdir)/src/lib-ssl-iostream -I$(incdir)/src/lib-compression -I$(incdir)/src/lib-settings -I$(incdir)/src/lib-test -I$(incdir)/src/lib-sasl" LIBDOVECOT_LDA_INCLUDE="-I$(incdir)/src/lib-lda -I$(incdir)/src/lda" LIBDOVECOT_DOVEADM_INCLUDE="-I$(incdir)/src/doveadm" LIBDOVECOT_STORAGE_INCLUDE="-I$(incdir)/src/lib-index -I$(incdir)/src/lib-storage -I$(incdir)/src/lib-storage/list -I$(incdir)/src/lib-storage/index -I$(incdir)/src/lib-storage/index/raw -I$(incdir)/src/lib-imap-storage -I$(incdir)/src/plugins/quota" From dovecot at dovecot.org Wed Feb 11 09:57:15 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 11 Feb 2015 09:57:15 +0000 Subject: dovecot-2.2: fts: Fixed error printing if fts wasn't enabled for... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/2a262816191d changeset: 18241:2a262816191d user: Timo Sirainen date: Wed Feb 11 11:56:59 2015 +0200 description: fts: Fixed error printing if fts wasn't enabled for user's default namespace. diffstat: src/plugins/fts/doveadm-fts.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r 51d04f935e61 -r 2a262816191d src/plugins/fts/doveadm-fts.c --- a/src/plugins/fts/doveadm-fts.c Tue Feb 10 17:29:31 2015 +0200 +++ b/src/plugins/fts/doveadm-fts.c Wed Feb 11 11:56:59 2015 +0200 @@ -25,7 +25,8 @@ } if (fts_list_backend(ns->list) == NULL) { - i_error("fts not enabled for user's namespace %s", ns_prefix); + i_error("fts not enabled for user's namespace %s", + ns_prefix != NULL ? ns_prefix : "INBOX"); return -1; } *ns_r = ns; From dovecot at dovecot.org Thu Feb 12 08:49:45 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 12 Feb 2015 08:49:45 +0000 Subject: dovecot-2.2: fts: Fixed memory leak in HTML parsing Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/8b44a5a14704 changeset: 18242:8b44a5a14704 user: Timo Sirainen date: Thu Feb 12 10:49:28 2015 +0200 description: fts: Fixed memory leak in HTML parsing diffstat: src/plugins/fts/fts-parser-html.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 2a262816191d -r 8b44a5a14704 src/plugins/fts/fts-parser-html.c --- a/src/plugins/fts/fts-parser-html.c Wed Feb 11 11:56:59 2015 +0200 +++ b/src/plugins/fts/fts-parser-html.c Thu Feb 12 10:49:28 2015 +0200 @@ -51,6 +51,7 @@ { struct html_fts_parser *parser = (struct html_fts_parser *)_parser; + mail_html2text_deinit(&parser->html2text); buffer_free(&parser->output); i_free(parser); } From dovecot at dovecot.org Thu Feb 12 10:42:55 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 12 Feb 2015 10:42:55 +0000 Subject: dovecot-2.2: dovecot-config: Added LIBDOVECOT_IMAPC_INCLUDE and ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f5f9ce55eca3 changeset: 18243:f5f9ce55eca3 user: Timo Sirainen date: Thu Feb 12 12:42:39 2015 +0200 description: dovecot-config: Added LIBDOVECOT_IMAPC_INCLUDE and LIBDOVECOT_FTS_INCLUDE diffstat: dovecot-config.in.in | 2 ++ dovecot.m4 | 2 +- 2 files changed, 3 insertions(+), 1 deletions(-) diffs (24 lines): diff -r 8b44a5a14704 -r f5f9ce55eca3 dovecot-config.in.in --- a/dovecot-config.in.in Thu Feb 12 10:49:28 2015 +0200 +++ b/dovecot-config.in.in Thu Feb 12 12:42:39 2015 +0200 @@ -28,6 +28,8 @@ LIBDOVECOT_LOGIN_INCLUDE="-I$(incdir)/src/login-common" LIBDOVECOT_IMAP_INCLUDE="-I$(incdir)/src/imap" LIBDOVECOT_CONFIG_INCLUDE="-I$(incdir)/src/config" +LIBDOVECOT_IMAPC_INCLUDE="-I$(incdir)/src/lib-imap-client" +LIBDOVECOT_FTS_INCLUDE="-I$(incdir)/src/plugins/fts" dovecot_pkgincludedir= dovecot_pkglibdir= diff -r 8b44a5a14704 -r f5f9ce55eca3 dovecot.m4 --- a/dovecot.m4 Thu Feb 12 10:49:28 2015 +0200 +++ b/dovecot.m4 Thu Feb 12 12:42:39 2015 +0200 @@ -118,7 +118,7 @@ AX_SUBST_L([DOVECOT_CFLAGS], [DOVECOT_LIBS], [DOVECOT_SSL_LIBS], [DOVECOT_SQL_LIBS], [DOVECOT_COMPRESS_LIBS]) AX_SUBST_L([LIBDOVECOT], [LIBDOVECOT_LOGIN], [LIBDOVECOT_SQL], [LIBDOVECOT_SSL], [LIBDOVECOT_COMPRESS], [LIBDOVECOT_LDA], [LIBDOVECOT_STORAGE], [LIBDOVECOT_DSYNC]) AX_SUBST_L([LIBDOVECOT_DEPS], [LIBDOVECOT_LOGIN_DEPS], [LIBDOVECOT_SQL_DEPS], [LIBDOVECOT_SSL_DEPS], [LIBDOVECOT_COMPRESS_DEPS], [LIBDOVECOT_LDA_DEPS], [LIBDOVECOT_STORAGE_DEPS], [LIBDOVECOT_DSYNC_DEPS]) - AX_SUBST_L([LIBDOVECOT_INCLUDE], [LIBDOVECOT_LDA_INCLUDE], [LIBDOVECOT_DOVEADM_INCLUDE], [LIBDOVECOT_SERVICE_INCLUDE], [LIBDOVECOT_STORAGE_INCLUDE], [LIBDOVECOT_LOGIN_INCLUDE], [LIBDOVECOT_CONFIG_INCLUDE], [LIBDOVECOT_IMAP_INCLUDE], [LIBDOVECOT_DSYNC_INCLUDE]) + AX_SUBST_L([LIBDOVECOT_INCLUDE], [LIBDOVECOT_LDA_INCLUDE], [LIBDOVECOT_DOVEADM_INCLUDE], [LIBDOVECOT_SERVICE_INCLUDE], [LIBDOVECOT_STORAGE_INCLUDE], [LIBDOVECOT_LOGIN_INCLUDE], [LIBDOVECOT_CONFIG_INCLUDE], [LIBDOVECOT_IMAP_INCLUDE], [LIBDOVECOT_DSYNC_INCLUDE], [LIBDOVECOT_IMAPC_INCLUDE], [LIBDOVECOT_FTS_INCLUDE]) DC_PLUGIN_DEPS DC_DOVECOT_TEST_WRAPPER From dovecot at dovecot.org Thu Feb 12 11:29:41 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 12 Feb 2015 11:29:41 +0000 Subject: dovecot-2.2: dovecot-config: Added lib-storage/index/imapc also ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/524098f35b3b changeset: 18244:524098f35b3b user: Timo Sirainen date: Thu Feb 12 13:29:23 2015 +0200 description: dovecot-config: Added lib-storage/index/imapc also to LIBDOVECOT_IMAPC_INCLUDE diffstat: dovecot-config.in.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r f5f9ce55eca3 -r 524098f35b3b dovecot-config.in.in --- a/dovecot-config.in.in Thu Feb 12 12:42:39 2015 +0200 +++ b/dovecot-config.in.in Thu Feb 12 13:29:23 2015 +0200 @@ -28,7 +28,7 @@ LIBDOVECOT_LOGIN_INCLUDE="-I$(incdir)/src/login-common" LIBDOVECOT_IMAP_INCLUDE="-I$(incdir)/src/imap" LIBDOVECOT_CONFIG_INCLUDE="-I$(incdir)/src/config" -LIBDOVECOT_IMAPC_INCLUDE="-I$(incdir)/src/lib-imap-client" +LIBDOVECOT_IMAPC_INCLUDE="-I$(incdir)/src/lib-imap-client -I$(incdir)/src/lib-storage/index/imapc" LIBDOVECOT_FTS_INCLUDE="-I$(incdir)/src/plugins/fts" dovecot_pkgincludedir= From dovecot at dovecot.org Thu Feb 12 12:38:37 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 12 Feb 2015 12:38:37 +0000 Subject: dovecot-2.2: dovecot-config: Added LIBDOVECOT_NOTIFY_INCLUDE Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f10725a5eed8 changeset: 18245:f10725a5eed8 user: Timo Sirainen date: Thu Feb 12 14:38:18 2015 +0200 description: dovecot-config: Added LIBDOVECOT_NOTIFY_INCLUDE diffstat: dovecot-config.in.in | 1 + dovecot.m4 | 2 +- 2 files changed, 2 insertions(+), 1 deletions(-) diffs (23 lines): diff -r 524098f35b3b -r f10725a5eed8 dovecot-config.in.in --- a/dovecot-config.in.in Thu Feb 12 13:29:23 2015 +0200 +++ b/dovecot-config.in.in Thu Feb 12 14:38:18 2015 +0200 @@ -30,6 +30,7 @@ LIBDOVECOT_CONFIG_INCLUDE="-I$(incdir)/src/config" LIBDOVECOT_IMAPC_INCLUDE="-I$(incdir)/src/lib-imap-client -I$(incdir)/src/lib-storage/index/imapc" LIBDOVECOT_FTS_INCLUDE="-I$(incdir)/src/plugins/fts" +LIBDOVECOT_NOTIFY_INCLUDE="-I$(incdir)/src/plugins/notify" dovecot_pkgincludedir= dovecot_pkglibdir= diff -r 524098f35b3b -r f10725a5eed8 dovecot.m4 --- a/dovecot.m4 Thu Feb 12 13:29:23 2015 +0200 +++ b/dovecot.m4 Thu Feb 12 14:38:18 2015 +0200 @@ -118,7 +118,7 @@ AX_SUBST_L([DOVECOT_CFLAGS], [DOVECOT_LIBS], [DOVECOT_SSL_LIBS], [DOVECOT_SQL_LIBS], [DOVECOT_COMPRESS_LIBS]) AX_SUBST_L([LIBDOVECOT], [LIBDOVECOT_LOGIN], [LIBDOVECOT_SQL], [LIBDOVECOT_SSL], [LIBDOVECOT_COMPRESS], [LIBDOVECOT_LDA], [LIBDOVECOT_STORAGE], [LIBDOVECOT_DSYNC]) AX_SUBST_L([LIBDOVECOT_DEPS], [LIBDOVECOT_LOGIN_DEPS], [LIBDOVECOT_SQL_DEPS], [LIBDOVECOT_SSL_DEPS], [LIBDOVECOT_COMPRESS_DEPS], [LIBDOVECOT_LDA_DEPS], [LIBDOVECOT_STORAGE_DEPS], [LIBDOVECOT_DSYNC_DEPS]) - AX_SUBST_L([LIBDOVECOT_INCLUDE], [LIBDOVECOT_LDA_INCLUDE], [LIBDOVECOT_DOVEADM_INCLUDE], [LIBDOVECOT_SERVICE_INCLUDE], [LIBDOVECOT_STORAGE_INCLUDE], [LIBDOVECOT_LOGIN_INCLUDE], [LIBDOVECOT_CONFIG_INCLUDE], [LIBDOVECOT_IMAP_INCLUDE], [LIBDOVECOT_DSYNC_INCLUDE], [LIBDOVECOT_IMAPC_INCLUDE], [LIBDOVECOT_FTS_INCLUDE]) + AX_SUBST_L([LIBDOVECOT_INCLUDE], [LIBDOVECOT_LDA_INCLUDE], [LIBDOVECOT_DOVEADM_INCLUDE], [LIBDOVECOT_SERVICE_INCLUDE], [LIBDOVECOT_STORAGE_INCLUDE], [LIBDOVECOT_LOGIN_INCLUDE], [LIBDOVECOT_CONFIG_INCLUDE], [LIBDOVECOT_IMAP_INCLUDE], [LIBDOVECOT_DSYNC_INCLUDE], [LIBDOVECOT_IMAPC_INCLUDE], [LIBDOVECOT_FTS_INCLUDE], [LIBDOVECOT_NOTIFY_INCLUDE]) DC_PLUGIN_DEPS DC_DOVECOT_TEST_WRAPPER From dovecot at dovecot.org Fri Feb 13 12:36:12 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 13 Feb 2015 12:36:12 +0000 Subject: dovecot-2.2: dovecot-config: Added DOVECOT_INSTALLED parameter. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/16c21c12e21f changeset: 18246:16c21c12e21f user: Timo Sirainen date: Fri Feb 13 14:35:54 2015 +0200 description: dovecot-config: Added DOVECOT_INSTALLED parameter. Also added dovecot_installed_moduledir to dovecot.m4 These can be used by external plugins to access some headers and libraries that don't already have existing DOVECOT_* parameters in dovecot-config. diffstat: Makefile.am | 5 +++-- dovecot.m4 | 10 +++++++--- 2 files changed, 10 insertions(+), 5 deletions(-) diffs (67 lines): diff -r f10725a5eed8 -r 16c21c12e21f Makefile.am --- a/Makefile.am Thu Feb 12 14:38:18 2015 +0200 +++ b/Makefile.am Fri Feb 13 14:35:54 2015 +0200 @@ -37,7 +37,7 @@ dovecot-config: dovecot-config.in Makefile old=`pwd` && cd $(top_builddir) && abs_builddir=`pwd` && cd $$old && \ cd $(top_srcdir) && abs_srcdir=`pwd` && cd $$old && \ - cat dovecot-config.in | sed \ + (echo "DOVECOT_INSTALLED=no"; cat dovecot-config.in | sed \ -e "s|\$$(top_builddir)|$$abs_builddir|g" \ -e "s|\$$(incdir)|$$abs_srcdir|g" \ -e "s|\$$(LIBICONV)|$(LIBICONV)|g" \ @@ -47,7 +47,7 @@ -e "s|^\(dovecot_pkglibexecdir\)=|\1=$(libexecdir)/dovecot|" \ -e "s|^\(dovecot_docdir\)=|\1=$(docdir)|" \ -e "s|^\(dovecot_moduledir\)=|\1=$(moduledir)|" \ - > dovecot-config + ) > dovecot-config if HAVE_SYSTEMD %.service: %.service.in @@ -64,6 +64,7 @@ $(mkdir_p) $(DESTDIR)$(pkglibdir); \ grep -v '^LIBDOVECOT_.*_INCLUDE' dovecot-config | \ grep -v '^LIBDOVECOT.*_DEPS' | sed \ + -e "s|^\(DOVECOT_INSTALLED\)=.*$$|\1=yes|" \ -e "s|^\(LIBDOVECOT\)=.*$$|\1='-L$(pkglibdir) -ldovecot'|" \ -e "s|^\(LIBDOVECOT_LOGIN\)=.*$$|\1='-ldovecot-login $(SSL_LIBS)'|" \ -e "s|^\(LIBDOVECOT_SQL\)=.*$$|\1=-ldovecot-sql|" \ diff -r f10725a5eed8 -r 16c21c12e21f dovecot.m4 --- a/dovecot.m4 Thu Feb 12 14:38:18 2015 +0200 +++ b/dovecot.m4 Fri Feb 13 14:35:54 2015 +0200 @@ -6,7 +6,7 @@ # unlimited permission to copy and/or distribute it, with or without # modifications, as long as this notice is preserved. -# serial 13 +# serial 14 AC_DEFUN([DC_DOVECOT_MODULEDIR],[ AC_ARG_WITH(moduledir, @@ -104,6 +104,8 @@ eval `grep -i '^dovecot_[[a-z_]]*=' "$dovecotdir"/dovecot-config` eval `grep '^LIBDOVECOT[[A-Z_]]*=' "$dovecotdir"/dovecot-config` + dovecot_installed_moduledir="$dovecot_moduledir" + if test "$use_install_dirs" = "no"; then # the main purpose of these is to fix make distcheck for plugins # other than that, they don't really make much sense @@ -114,12 +116,14 @@ dovecot_moduledir='$(moduledir)' fi - AX_SUBST_L([DISTCHECK_CONFIGURE_FLAGS], [dovecotdir], [dovecot_moduledir], [dovecot_pkgincludedir], [dovecot_pkglibexecdir], [dovecot_pkglibdir], [dovecot_docdir]) - AX_SUBST_L([DOVECOT_CFLAGS], [DOVECOT_LIBS], [DOVECOT_SSL_LIBS], [DOVECOT_SQL_LIBS], [DOVECOT_COMPRESS_LIBS]) + AX_SUBST_L([DISTCHECK_CONFIGURE_FLAGS], [dovecotdir], [dovecot_moduledir], [dovecot_installed_moduledir], [dovecot_pkgincludedir], [dovecot_pkglibexecdir], [dovecot_pkglibdir], [dovecot_docdir]) + AX_SUBST_L([DOVECOT_INSTALLED], [DOVECOT_CFLAGS], [DOVECOT_LIBS], [DOVECOT_SSL_LIBS], [DOVECOT_SQL_LIBS], [DOVECOT_COMPRESS_LIBS]) AX_SUBST_L([LIBDOVECOT], [LIBDOVECOT_LOGIN], [LIBDOVECOT_SQL], [LIBDOVECOT_SSL], [LIBDOVECOT_COMPRESS], [LIBDOVECOT_LDA], [LIBDOVECOT_STORAGE], [LIBDOVECOT_DSYNC]) AX_SUBST_L([LIBDOVECOT_DEPS], [LIBDOVECOT_LOGIN_DEPS], [LIBDOVECOT_SQL_DEPS], [LIBDOVECOT_SSL_DEPS], [LIBDOVECOT_COMPRESS_DEPS], [LIBDOVECOT_LDA_DEPS], [LIBDOVECOT_STORAGE_DEPS], [LIBDOVECOT_DSYNC_DEPS]) AX_SUBST_L([LIBDOVECOT_INCLUDE], [LIBDOVECOT_LDA_INCLUDE], [LIBDOVECOT_DOVEADM_INCLUDE], [LIBDOVECOT_SERVICE_INCLUDE], [LIBDOVECOT_STORAGE_INCLUDE], [LIBDOVECOT_LOGIN_INCLUDE], [LIBDOVECOT_CONFIG_INCLUDE], [LIBDOVECOT_IMAP_INCLUDE], [LIBDOVECOT_DSYNC_INCLUDE], [LIBDOVECOT_IMAPC_INCLUDE], [LIBDOVECOT_FTS_INCLUDE], [LIBDOVECOT_NOTIFY_INCLUDE]) + AM_CONDITIONAL(DOVECOT_INSTALLED, test "$DOVECOT_INSTALLED" = "yes") + DC_PLUGIN_DEPS DC_DOVECOT_TEST_WRAPPER ]) From dovecot at dovecot.org Fri Feb 13 14:38:53 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 13 Feb 2015 14:38:53 +0000 Subject: dovecot-2.2: Link all libstorage.la dependencies into the librar... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/3ac5933082c4 changeset: 18247:3ac5933082c4 user: Timo Sirainen date: Fri Feb 13 16:38:35 2015 +0200 description: Link all libstorage.la dependencies into the library itself instead. Instead of keeping a lot of different libraries in LIBDOVECOT_STORAGE, which have circular dependencies and may cause linking issues. diffstat: configure.ac | 9 +++++---- src/lib-storage/Makefile.am | 15 +++++++-------- 2 files changed, 12 insertions(+), 12 deletions(-) diffs (77 lines): diff -r 16c21c12e21f -r 3ac5933082c4 configure.ac --- a/configure.ac Fri Feb 13 14:35:54 2015 +0200 +++ b/configure.ac Fri Feb 13 16:38:35 2015 +0200 @@ -2551,14 +2551,15 @@ else LIBDOVECOT_DEPS='$(top_builddir)/src/lib-master/libmaster.la $(top_builddir)/src/lib-settings/libsettings.la $(top_builddir)/src/lib-http/libhttp.la $(top_builddir)/src/lib-dict/libdict.la $(top_builddir)/src/lib-dns/libdns.la $(top_builddir)/src/lib-fs/libfs.la $(top_builddir)/src/lib-imap/libimap.la $(top_builddir)/src/lib-mail/libmail.la $(top_builddir)/src/lib-sasl/libsasl.la $(top_builddir)/src/lib-auth/libauth.la $(top_builddir)/src/lib-charset/libcharset.la $(top_builddir)/src/lib-ssl-iostream/libssl_iostream.la $(top_builddir)/src/lib-test/libtest.la $(top_builddir)/src/lib/liblib.la' LIBDOVECOT="$LIBDOVECOT_DEPS \$(LIBICONV) \$(MODULE_LIBS)" - LIBDOVECOT_STORAGE_LAST='$(top_builddir)/src/lib-storage/list/libstorage_list.la $(top_builddir)/src/lib-storage/index/libstorage_index.la $(top_builddir)/src/lib-storage/libstorage.la $(top_builddir)/src/lib-index/libindex.la $(top_builddir)/src/lib-imap-storage/libimap-storage.la' - LIBDOVECOT_STORAGE_FIRST='$(top_builddir)/src/lib-storage/libstorage_service.la $(top_builddir)/src/lib-storage/register/libstorage_register.la' - LIBDOVECOT_STORAGE_DEPS="$LIBDOVECOT_STORAGE_FIRST $LINKED_STORAGE_LIBS $LIBDOVECOT_STORAGE_LAST" + #LIBDOVECOT_STORAGE_LAST='$(top_builddir)/src/lib-storage/list/libstorage_list.la $(top_builddir)/src/lib-storage/index/libstorage_index.la $(top_builddir)/src/lib-storage/libstorage.la $(top_builddir)/src/lib-index/libindex.la $(top_builddir)/src/lib-imap-storage/libimap-storage.la' + #LIBDOVECOT_STORAGE_FIRST='$(top_builddir)/src/lib-storage/register/libstorage_register.la' + #LIBDOVECOT_STORAGE_DEPS="$LIBDOVECOT_STORAGE_FIRST $LINKED_STORAGE_LIBS $LIBDOVECOT_STORAGE_LAST" + LIBDOVECOT_STORAGE_DEPS='$(top_builddir)/src/lib-storage/libstorage.la' LIBDOVECOT_LOGIN='$(top_builddir)/src/login-common/liblogin.la' LIBDOVECOT_COMPRESS='$(top_builddir)/src/lib-compression/libcompression.la' LIBDOVECOT_LDA='$(top_builddir)/src/lib-lda/liblda.la' fi -LIBDOVECOT_STORAGE="$LIBDOVECOT_STORAGE_DEPS $LINKED_STORAGE_LDADD" +LIBDOVECOT_STORAGE="$LIBDOVECOT_STORAGE_DEPS" LIBDOVECOT_DSYNC='$(top_builddir)/src/doveadm/dsync/libdovecot-dsync.la' LIBDOVECOT_SQL='$(top_builddir)/src/lib-sql/libsql.la' AC_SUBST(LIBDOVECOT) diff -r 16c21c12e21f -r 3ac5933082c4 src/lib-storage/Makefile.am --- a/src/lib-storage/Makefile.am Fri Feb 13 14:35:54 2015 +0200 +++ b/src/lib-storage/Makefile.am Fri Feb 13 16:38:35 2015 +0200 @@ -1,6 +1,6 @@ SUBDIRS = list index register -noinst_LTLIBRARIES = libstorage.la libstorage_service.la +noinst_LTLIBRARIES = libstorage.la AM_CPPFLAGS = \ -I$(top_srcdir)/src/lib \ @@ -37,6 +37,7 @@ mail-search-register-imap.c \ mail-storage.c \ mail-storage-hooks.c \ + mail-storage-service.c \ mail-storage-settings.c \ mail-thread.c \ mail-user.c \ @@ -50,9 +51,6 @@ mailbox-tree.c \ mailbox-uidvalidity.c -libstorage_service_la_SOURCES = \ - mail-storage-service.c - headers = \ fail-mail-storage.h \ mail-copy.h \ @@ -81,8 +79,6 @@ shlibs = \ @LINKED_STORAGE_LIBS@ \ - libstorage.la \ - libstorage_service.la \ list/libstorage_list.la \ index/libstorage_index.la \ register/libstorage_register.la \ @@ -90,10 +86,13 @@ ../lib-imap-storage/libimap-storage.la \ ../lib-dovecot/libdovecot.la +libstorage_la_LIBADD = $(shlibs) +libstorage_la_DEPENDENCIES = $(shlibs) + pkglib_LTLIBRARIES = libdovecot-storage.la libdovecot_storage_la_SOURCES = -libdovecot_storage_la_LIBADD = $(shlibs) $(LINKED_STORAGE_LDADD) -libdovecot_storage_la_DEPENDENCIES = $(shlibs) +libdovecot_storage_la_LIBADD = libstorage.la $(LINKED_STORAGE_LDADD) +libdovecot_storage_la_DEPENDENCIES = libstorage.la libdovecot_storage_la_LDFLAGS = -export-dynamic test_programs = \ From dovecot at dovecot.org Fri Feb 13 14:39:50 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 13 Feb 2015 14:39:50 +0000 Subject: dovecot-2.2: configure: Removed unnecessary lines left by the pr... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9bd6ed657cb3 changeset: 18248:9bd6ed657cb3 user: Timo Sirainen date: Fri Feb 13 16:39:30 2015 +0200 description: configure: Removed unnecessary lines left by the previous commit. diffstat: configure.ac | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diffs (13 lines): diff -r 3ac5933082c4 -r 9bd6ed657cb3 configure.ac --- a/configure.ac Fri Feb 13 16:38:35 2015 +0200 +++ b/configure.ac Fri Feb 13 16:39:30 2015 +0200 @@ -2551,9 +2551,6 @@ else LIBDOVECOT_DEPS='$(top_builddir)/src/lib-master/libmaster.la $(top_builddir)/src/lib-settings/libsettings.la $(top_builddir)/src/lib-http/libhttp.la $(top_builddir)/src/lib-dict/libdict.la $(top_builddir)/src/lib-dns/libdns.la $(top_builddir)/src/lib-fs/libfs.la $(top_builddir)/src/lib-imap/libimap.la $(top_builddir)/src/lib-mail/libmail.la $(top_builddir)/src/lib-sasl/libsasl.la $(top_builddir)/src/lib-auth/libauth.la $(top_builddir)/src/lib-charset/libcharset.la $(top_builddir)/src/lib-ssl-iostream/libssl_iostream.la $(top_builddir)/src/lib-test/libtest.la $(top_builddir)/src/lib/liblib.la' LIBDOVECOT="$LIBDOVECOT_DEPS \$(LIBICONV) \$(MODULE_LIBS)" - #LIBDOVECOT_STORAGE_LAST='$(top_builddir)/src/lib-storage/list/libstorage_list.la $(top_builddir)/src/lib-storage/index/libstorage_index.la $(top_builddir)/src/lib-storage/libstorage.la $(top_builddir)/src/lib-index/libindex.la $(top_builddir)/src/lib-imap-storage/libimap-storage.la' - #LIBDOVECOT_STORAGE_FIRST='$(top_builddir)/src/lib-storage/register/libstorage_register.la' - #LIBDOVECOT_STORAGE_DEPS="$LIBDOVECOT_STORAGE_FIRST $LINKED_STORAGE_LIBS $LIBDOVECOT_STORAGE_LAST" LIBDOVECOT_STORAGE_DEPS='$(top_builddir)/src/lib-storage/libstorage.la' LIBDOVECOT_LOGIN='$(top_builddir)/src/login-common/liblogin.la' LIBDOVECOT_COMPRESS='$(top_builddir)/src/lib-compression/libcompression.la' From dovecot at dovecot.org Sat Feb 14 07:47:59 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 14 Feb 2015 07:47:59 +0000 Subject: dovecot-2.2: lib-imap-client: Improved error message a bit if se... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/9ecf16e01a87 changeset: 18249:9ecf16e01a87 user: Timo Sirainen date: Sat Feb 14 09:47:41 2015 +0200 description: lib-imap-client: Improved error message a bit if server disconnects. diffstat: src/lib-imap-client/imapc-connection.c | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diffs (25 lines): diff -r 9bd6ed657cb3 -r 9ecf16e01a87 src/lib-imap-client/imapc-connection.c --- a/src/lib-imap-client/imapc-connection.c Fri Feb 13 16:39:30 2015 +0200 +++ b/src/lib-imap-client/imapc-connection.c Sat Feb 14 09:47:41 2015 +0200 @@ -1300,15 +1300,17 @@ i_error("imapc(%s): Server disconnected with message: %s", conn->name, conn->disconnect_reason); } else if (conn->ssl_iostream == NULL) { - i_error("imapc(%s): Server disconnected unexpectedly", - conn->name); + errstr = conn->input->stream_errno == 0 ? "EOF" : + i_stream_get_error(conn->input); + i_error("imapc(%s): Server disconnected unexpectedly: %s", + conn->name, errstr); } else { errstr = ssl_iostream_get_last_error(conn->ssl_iostream); if (errstr == NULL) { errstr = conn->input->stream_errno == 0 ? "EOF" : - strerror(conn->input->stream_errno); + i_stream_get_error(conn->input); } - i_error("imapc(%s): Server disconnected: %s", + i_error("imapc(%s): Server disconnected unexpectedly: %s", conn->name, errstr); } imapc_connection_reconnect(conn); From dovecot at dovecot.org Sun Feb 15 08:11:19 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:19 +0000 Subject: dovecot-2.2: lib: If istream-seekable's parent stream returns er... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/20babbc0dc24 changeset: 18250:20babbc0dc24 user: Timo Sirainen date: Sun Feb 15 09:54:45 2015 +0200 description: lib: If istream-seekable's parent stream returns error, make sure we set eof=TRUE diffstat: src/lib/istream-seekable.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 9ecf16e01a87 -r 20babbc0dc24 src/lib/istream-seekable.c --- a/src/lib/istream-seekable.c Sat Feb 14 09:47:41 2015 +0200 +++ b/src/lib/istream-seekable.c Sun Feb 15 09:54:45 2015 +0200 @@ -143,6 +143,7 @@ "read(%s) failed: %s", i_stream_get_name(sstream->cur_input), i_stream_get_error(sstream->cur_input)); + sstream->istream.istream.eof = TRUE; sstream->istream.istream.stream_errno = sstream->cur_input->stream_errno; return -1; From dovecot at dovecot.org Sun Feb 15 08:11:19 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:19 +0000 Subject: dovecot-2.2: lib: If i_stream_read() sets stream_errno even if i... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d76d4351762b changeset: 18251:d76d4351762b user: Timo Sirainen date: Sun Feb 15 09:55:05 2015 +0200 description: lib: If i_stream_read() sets stream_errno even if it didn't return -1, make sure we set eof=TRUE diffstat: src/lib/istream.c | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diffs (26 lines): diff -r 20babbc0dc24 -r d76d4351762b src/lib/istream.c --- a/src/lib/istream.c Sun Feb 15 09:54:45 2015 +0200 +++ b/src/lib/istream.c Sun Feb 15 09:55:05 2015 +0200 @@ -137,6 +137,7 @@ ssize_t ret; if (unlikely(stream->closed || stream->stream_errno != 0)) { + stream->eof = TRUE; errno = stream->stream_errno; return -1; } @@ -174,6 +175,14 @@ break; } + if (stream->stream_errno != 0) { + /* error handling should be easier if we now just + assume the stream is now at EOF. Note that we could get here + even if read() didn't return -1, although that's a little + bit sloppy istream implementation. */ + stream->eof = TRUE; + } + i_stream_update(_stream); return ret; } From dovecot at dovecot.org Sun Feb 15 08:11:30 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:30 +0000 Subject: dovecot-2.2: lib-mail: istream-dot should have immediately retur... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/42b3ac799f2f changeset: 18252:42b3ac799f2f user: Timo Sirainen date: Sun Feb 15 09:55:56 2015 +0200 description: lib-mail: istream-dot should have immediately returned error if dot-line was missing at EOF. diffstat: src/lib-mail/istream-dot.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diffs (12 lines): diff -r d76d4351762b -r 42b3ac799f2f src/lib-mail/istream-dot.c --- a/src/lib-mail/istream-dot.c Sun Feb 15 09:55:05 2015 +0200 +++ b/src/lib-mail/istream-dot.c Sun Feb 15 09:55:56 2015 +0200 @@ -131,6 +131,8 @@ /* we have to update stream->pos before reading more data */ ret1 = i_stream_dot_return(stream, dest, 0); if ((ret = i_stream_dot_read_some(dstream)) <= 0) { + if (stream->istream.stream_errno != 0) + return -1; if (ret1 != 0) return ret1; dest = stream->pos; From dovecot at dovecot.org Sun Feb 15 08:11:30 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:30 +0000 Subject: dovecot-2.2: lib-mail: ostream-dot API changes Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/ca24e6d34345 changeset: 18253:ca24e6d34345 user: Timo Sirainen date: Sun Feb 15 10:03:10 2015 +0200 description: lib-mail: ostream-dot API changes The dot-line sending is now done when o_stream_flush() is called. This should be used instead of relying on close() doing it, because it still needs to write a few bytes and there are no guarantees that the parent stream allows sending them. Also added force_extra_crlf parameter, which specifies whether CRLF should always be written before the dot-line, even when it already contained CRLF. This is useful if the input is read with i_stream_create_dot(send_last_lf=FALSE), since it allows sending a stream that doesn't have to end with LF. diffstat: src/lib-mail/ostream-dot.c | 55 +++++++++++++++++++++++++++++++++------- src/lib-mail/ostream-dot.h | 12 ++++++-- src/lib-mail/test-ostream-dot.c | 3 +- 3 files changed, 56 insertions(+), 14 deletions(-) diffs (151 lines): diff -r 42b3ac799f2f -r ca24e6d34345 src/lib-mail/ostream-dot.c --- a/src/lib-mail/ostream-dot.c Sun Feb 15 09:55:56 2015 +0200 +++ b/src/lib-mail/ostream-dot.c Sun Feb 15 10:03:10 2015 +0200 @@ -10,25 +10,52 @@ STREAM_STATE_NONE, STREAM_STATE_CR, STREAM_STATE_CRLF, + STREAM_STATE_DONE }; struct dot_ostream { struct ostream_private ostream; enum dot_ostream_state state; + bool force_extra_crlf; }; -static void -o_stream_dot_close(struct iostream_private *stream, - bool close_parent) +static int +o_stream_dot_flush(struct ostream_private *stream) { struct dot_ostream *dstream = (struct dot_ostream *)stream; + int ret; - if (dstream->state == STREAM_STATE_CRLF) - (void)o_stream_send(dstream->ostream.parent, ".\r\n", 3); - else - (void)o_stream_send(dstream->ostream.parent, "\r\n.\r\n", 5); - (void)o_stream_flush(&dstream->ostream.ostream); + if (o_stream_get_buffer_avail_size(stream->parent) < 5) { + /* make space for the dot line */ + if ((ret = o_stream_flush(stream->parent)) <= 0) { + if (ret < 0) + o_stream_copy_error_from_parent(stream); + return ret; + } + } + + if (dstream->state == STREAM_STATE_DONE) + ; + else if (dstream->state == STREAM_STATE_CRLF && + !dstream->force_extra_crlf) { + ret = o_stream_send(stream->parent, ".\r\n", 3); + i_assert(ret == 3); + } else { + ret = o_stream_send(stream->parent, "\r\n.\r\n", 5); + i_assert(ret == 5); + } + dstream->state = STREAM_STATE_DONE; + + if ((ret = o_stream_flush(stream->parent)) < 0) + o_stream_copy_error_from_parent(stream); + return ret; +} + +static void +o_stream_dot_close(struct iostream_private *stream, bool close_parent) +{ + struct dot_ostream *dstream = (struct dot_ostream *)stream; if (close_parent) o_stream_close(dstream->ostream.parent); @@ -45,6 +72,8 @@ unsigned int count, i; ssize_t ret; + i_assert(dstream->state != STREAM_STATE_DONE); + if ((ret=o_stream_flush(stream->parent)) <= 0) { /* error / we still couldn't flush existing data to parent stream. */ @@ -54,7 +83,9 @@ /* check for dots */ t_array_init(&iov_arr, iov_count + 32); - max_bytes = o_stream_get_buffer_avail_size(stream->parent); // FIXME: what if max_buffer_size is 0? + max_bytes = o_stream_get_buffer_avail_size(stream->parent); + i_assert(max_bytes > 0); /* FIXME: not supported currently */ + sent = added = 0; for (i = 0; i < iov_count && max_bytes > 0; i++) { size_t size = iov[i].iov_len, chunk; @@ -113,6 +144,8 @@ break; } break; + case STREAM_STATE_DONE: + i_unreached(); } if (add != 0) { @@ -168,13 +201,15 @@ } struct ostream * -o_stream_create_dot(struct ostream *output) +o_stream_create_dot(struct ostream *output, bool force_extra_crlf) { struct dot_ostream *dstream; dstream = i_new(struct dot_ostream, 1); dstream->ostream.sendv = o_stream_dot_sendv; dstream->ostream.iostream.close = o_stream_dot_close; + dstream->ostream.flush = o_stream_dot_flush; dstream->ostream.max_buffer_size = output->real_stream->max_buffer_size; + dstream->force_extra_crlf = force_extra_crlf; return o_stream_create(&dstream->ostream, output, o_stream_get_fd(output)); } diff -r 42b3ac799f2f -r ca24e6d34345 src/lib-mail/ostream-dot.h --- a/src/lib-mail/ostream-dot.h Sun Feb 15 09:55:56 2015 +0200 +++ b/src/lib-mail/ostream-dot.h Sun Feb 15 10:03:10 2015 +0200 @@ -2,8 +2,14 @@ #define OSTREAM_DOT_H /* Create output stream for writing SMTP DATA style message: Add additional "." - to lines beginning with it. Write line that contains only "." upon close(). - */ -struct ostream *o_stream_create_dot(struct ostream *output); + to lines that start with ".". Write a line that contains only "." upon + o_stream_flush(). (This is also called at close(), but it shouldn't be + relied on since it could fail due to output buffer being full.) + + If output ends with CRLF, force_extra_crlf controls whether additional CRLF + is written before the "." line. This parameter should match + i_stream_create_dot()'s send_last_lf parameter (reversed). */ +struct ostream *o_stream_create_dot(struct ostream *output, + bool force_extra_crlf); #endif diff -r 42b3ac799f2f -r ca24e6d34345 src/lib-mail/test-ostream-dot.c --- a/src/lib-mail/test-ostream-dot.c Sun Feb 15 09:55:56 2015 +0200 +++ b/src/lib-mail/test-ostream-dot.c Sun Feb 15 10:03:10 2015 +0200 @@ -26,7 +26,7 @@ output_data = buffer_create_dynamic(pool_datastack_create(), 1024); test_output = o_stream_create_buffer(output_data); - output = o_stream_create_dot(test_output); + output = o_stream_create_dot(test_output, FALSE); while ((ret = i_stream_read(test_input)) > 0 || ret == -2) { data = i_stream_get_data(test_input, &size); @@ -39,6 +39,7 @@ test_assert(test_input->eof); + test_assert(o_stream_flush(output) > 0); o_stream_unref(&output); o_stream_unref(&test_output); From dovecot at dovecot.org Sun Feb 15 08:11:30 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:30 +0000 Subject: dovecot-2.2: doveadm: Added support for mail commands to read an... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/22a5eda76490 changeset: 18254:22a5eda76490 user: Timo Sirainen date: Sun Feb 15 10:09:19 2015 +0200 description: doveadm: Added support for mail commands to read an input stream (from stdin) This is done by calling doveadm_mail_get_input() from the command's init() function. Currently it reads the entire input into a seekable istream with hardcoded 5 minute timeout. The input stream sending works also through doveadm proxying. This could probably be used by dsync at some point to support proxying over doveadm proxies, but that would require some more work. Especially a flag for commands to specify that they allow non-blocking input streams. diffstat: src/doveadm/doveadm-dsync.c | 2 +- src/doveadm/doveadm-mail-server.c | 2 +- src/doveadm/doveadm-mail.c | 80 +++++++++++++++++++++++++++++++++++++++ src/doveadm/doveadm-mail.h | 7 +++ src/doveadm/server-connection.c | 79 +++++++++++++++++++++++++++++++++++++- src/doveadm/server-connection.h | 1 + 6 files changed, 166 insertions(+), 5 deletions(-) diffs (truncated from 321 to 300 lines): diff -r ca24e6d34345 -r 22a5eda76490 src/doveadm/doveadm-dsync.c --- a/src/doveadm/doveadm-dsync.c Sun Feb 15 10:03:10 2015 +0200 +++ b/src/doveadm/doveadm-dsync.c Sun Feb 15 10:09:19 2015 +0200 @@ -762,7 +762,7 @@ str_append_c(cmd, '\n'); ctx->tcp_conn = conn; - server_connection_cmd(conn, str_c(cmd), + server_connection_cmd(conn, str_c(cmd), NULL, dsync_connected_callback, ctx); io_loop_run(ioloop); ctx->tcp_conn = NULL; diff -r ca24e6d34345 -r 22a5eda76490 src/doveadm/doveadm-mail-server.c --- a/src/doveadm/doveadm-mail-server.c Sun Feb 15 10:03:10 2015 +0200 +++ b/src/doveadm/doveadm-mail-server.c Sun Feb 15 10:09:19 2015 +0200 @@ -155,7 +155,7 @@ servercmd = i_new(struct doveadm_mail_server_cmd, 1); servercmd->conn = conn; servercmd->username = i_strdup(username); - server_connection_cmd(conn, str_c(cmd), + server_connection_cmd(conn, str_c(cmd), cmd_ctx->cmd_input, doveadm_cmd_callback, servercmd); } diff -r ca24e6d34345 -r 22a5eda76490 src/doveadm/doveadm-mail.c --- a/src/doveadm/doveadm-mail.c Sun Feb 15 10:03:10 2015 +0200 +++ b/src/doveadm/doveadm-mail.c Sun Feb 15 10:09:19 2015 +0200 @@ -4,6 +4,9 @@ #include "array.h" #include "lib-signals.h" #include "ioloop.h" +#include "istream.h" +#include "istream-dot.h" +#include "istream-seekable.h" #include "str.h" #include "unichar.h" #include "module-dir.h" @@ -17,6 +20,7 @@ #include "mail-search-build.h" #include "mail-search-parser.h" #include "mailbox-list-iter.h" +#include "client-connection.h" #include "doveadm.h" #include "doveadm-settings.h" #include "doveadm-print.h" @@ -26,6 +30,8 @@ #include #include +#define DOVEADM_MAIL_CMD_INPUT_TIMEOUT_MSECS (5*60*1000) + ARRAY_TYPE(doveadm_mail_cmd) doveadm_mail_cmds; void (*hook_doveadm_mail_init)(struct doveadm_mail_cmd_context *ctx); struct doveadm_mail_cmd_module_register @@ -144,6 +150,76 @@ return ctx; } +static void doveadm_mail_cmd_input_input(struct doveadm_mail_cmd_context *ctx) +{ + while (i_stream_read(ctx->cmd_input) > 0) + i_stream_skip(ctx->cmd_input, i_stream_get_data_size(ctx->cmd_input)); + if (!ctx->cmd_input->eof) + return; + + if (ctx->cmd_input->stream_errno != 0) { + i_error("read(%s) failed: %s", + i_stream_get_name(ctx->cmd_input), + i_stream_get_error(ctx->cmd_input)); + } + io_loop_stop(current_ioloop); +} + +static void doveadm_mail_cmd_input_timeout(struct doveadm_mail_cmd_context *ctx) +{ + struct istream *input; + + input = i_stream_create_error_str(ETIMEDOUT, "Timed out in %u secs", + DOVEADM_MAIL_CMD_INPUT_TIMEOUT_MSECS/1000); + i_stream_set_name(input, i_stream_get_name(ctx->cmd_input)); + i_stream_destroy(&ctx->cmd_input); + ctx->cmd_input = input; + io_loop_stop(current_ioloop); +} + +static void doveadm_mail_cmd_input_read(struct doveadm_mail_cmd_context *ctx) +{ + struct ioloop *ioloop; + struct io *io; + struct timeout *to; + + ioloop = io_loop_create(); + io = io_add(ctx->cmd_input_fd, IO_READ, + doveadm_mail_cmd_input_input, ctx); + to = timeout_add(DOVEADM_MAIL_CMD_INPUT_TIMEOUT_MSECS, + doveadm_mail_cmd_input_timeout, ctx); + io_loop_run(ioloop); + io_remove(&io); + timeout_remove(&to); + io_loop_destroy(&ioloop); + + i_assert(ctx->cmd_input->eof); + i_stream_seek(ctx->cmd_input, 0); +} + +void doveadm_mail_get_input(struct doveadm_mail_cmd_context *ctx) +{ + struct istream *inputs[2]; + + if (ctx->cmd_input != NULL) + return; + + if (ctx->conn != NULL) + inputs[0] = i_stream_create_dot(ctx->conn->input, FALSE); + else { + inputs[0] = i_stream_create_fd(STDIN_FILENO, 1024*1024, FALSE); + i_stream_set_name(inputs[0], "stdin"); + } + inputs[1] = NULL; + ctx->cmd_input_fd = i_stream_get_fd(inputs[0]); + ctx->cmd_input = i_stream_create_seekable_path(inputs, 1024*256, + "/tmp/doveadm."); + i_stream_set_name(ctx->cmd_input, i_stream_get_name(inputs[0])); + i_stream_unref(&inputs[0]); + + doveadm_mail_cmd_input_read(ctx); +} + struct mailbox * doveadm_mailbox_find(struct mail_user *user, const char *mailbox) { @@ -322,6 +398,8 @@ return ret; } + if (ctx->cmd_input != NULL) + i_stream_seek(ctx->cmd_input, 0); if (ctx->v.run(ctx, ctx->cur_mail_user) < 0) { i_assert(ctx->exit_code != 0); } @@ -551,6 +629,8 @@ /* service deinit unloads mail plugins, so do it late */ mail_storage_service_deinit(&ctx->storage_service); + if (ctx->cmd_input != NULL) + i_stream_unref(&ctx->cmd_input); if (ctx->exit_code != 0) doveadm_exit_code = ctx->exit_code; pool_unref(&ctx->pool); diff -r ca24e6d34345 -r 22a5eda76490 src/doveadm/doveadm-mail.h --- a/src/doveadm/doveadm-mail.h Sun Feb 15 10:03:10 2015 +0200 +++ b/src/doveadm/doveadm-mail.h Sun Feb 15 10:09:19 2015 +0200 @@ -81,6 +81,9 @@ struct mail_user *cur_mail_user; struct doveadm_mail_cmd_vfuncs v; + struct istream *cmd_input; + int cmd_input_fd; + ARRAY(union doveadm_mail_cmd_module_context *) module_contexts; /* if non-zero, exit with this code */ @@ -133,6 +136,10 @@ const char **error_r); void doveadm_mail_server_flush(void); +/* Request input stream to be read (from stdin). This must be called from + the command's init() function. */ +void doveadm_mail_get_input(struct doveadm_mail_cmd_context *ctx); + struct mailbox * doveadm_mailbox_find(struct mail_user *user, const char *mailbox); int doveadm_mailbox_find_and_sync(struct mail_user *user, const char *mailbox, diff -r ca24e6d34345 -r 22a5eda76490 src/doveadm/server-connection.c --- a/src/doveadm/server-connection.c Sun Feb 15 10:03:10 2015 +0200 +++ b/src/doveadm/server-connection.c Sun Feb 15 10:09:19 2015 +0200 @@ -7,6 +7,7 @@ #include "net.h" #include "istream.h" #include "ostream.h" +#include "ostream-dot.h" #include "str.h" #include "strescape.h" #include "iostream-ssl.h" @@ -42,6 +43,8 @@ struct ostream *output; struct ssl_iostream *ssl_iostream; + struct istream *cmd_input; + struct ostream *cmd_output; const char *delayed_cmd; server_cmd_callback_t *callback; void *context; @@ -78,6 +81,58 @@ } } +static int server_connection_send_cmd_input_more(struct server_connection *conn) +{ + off_t ret; + + /* ostream-dot writes only up to max buffer size, so keep it non-zero */ + o_stream_set_max_buffer_size(conn->cmd_output, IO_BLOCK_SIZE); + ret = o_stream_send_istream(conn->cmd_output, conn->cmd_input); + o_stream_set_max_buffer_size(conn->cmd_output, (size_t)-1); + + if (ret >= 0 && i_stream_have_bytes_left(conn->cmd_input)) { + o_stream_set_flush_pending(conn->cmd_output, TRUE); + return 0; + } + if (conn->cmd_input->stream_errno != 0) { + i_error("read(%s) failed: %s", + i_stream_get_name(conn->cmd_input), + i_stream_get_error(conn->cmd_input)); + } else if (conn->cmd_output->stream_errno != 0 || + o_stream_flush(conn->cmd_output) < 0) { + i_error("write(%s) failed: %s", + o_stream_get_name(conn->cmd_output), + o_stream_get_error(conn->cmd_output)); + } + + i_stream_destroy(&conn->cmd_input); + o_stream_destroy(&conn->cmd_output); + return ret < 0 ? -1 : 1; +} + +static void server_connection_send_cmd_input(struct server_connection *conn) +{ + if (conn->cmd_input == NULL) + return; + + conn->cmd_output = o_stream_create_dot(conn->output, TRUE); + (void)server_connection_send_cmd_input_more(conn); +} + +static int server_connection_output(struct server_connection *conn) +{ + int ret; + + o_stream_cork(conn->output); + ret = o_stream_flush(conn->output); + if (ret > 0 && conn->cmd_input != NULL && conn->delayed_cmd == NULL) + ret = server_connection_send_cmd_input_more(conn); + if (ret < 0) + server_connection_destroy(&conn); + o_stream_uncork(conn->output); + return ret; +} + static void server_connection_callback(struct server_connection *conn, int exit_code, const char *error) @@ -171,6 +226,7 @@ if (conn->delayed_cmd != NULL) { o_stream_nsend_str(conn->output, conn->delayed_cmd); conn->delayed_cmd = NULL; + server_connection_send_cmd_input(conn); } } @@ -401,6 +457,10 @@ conn->io = io_add(conn->fd, IO_READ, server_connection_input, conn); conn->input = i_stream_create_fd(conn->fd, MAX_INBUF_SIZE, FALSE); conn->output = o_stream_create_fd(conn->fd, (size_t)-1, FALSE); + o_stream_set_flush_callback(conn->output, server_connection_output, conn); + + i_stream_set_name(conn->input, server->name); + o_stream_set_name(conn->output, server->name); array_append(&conn->server->connections, &conn, 1); @@ -452,6 +512,11 @@ i_stream_destroy(&conn->input); if (conn->output != NULL) o_stream_destroy(&conn->output); + if (conn->cmd_input != NULL) + i_stream_destroy(&conn->cmd_input); + /* close cmd_output after its parent, so the "." isn't sent */ + if (conn->cmd_output != NULL) + o_stream_destroy(&conn->cmd_output); if (conn->ssl_iostream != NULL) ssl_iostream_unref(&conn->ssl_iostream); if (conn->io != NULL) @@ -470,15 +535,23 @@ } void server_connection_cmd(struct server_connection *conn, const char *line, + struct istream *cmd_input, server_cmd_callback_t *callback, void *context) { i_assert(conn->delayed_cmd == NULL); conn->state = SERVER_REPLY_STATE_PRINT; - if (conn->authenticated) + if (cmd_input != NULL) { + i_assert(conn->cmd_input == NULL); + i_stream_ref(cmd_input); + conn->cmd_input = cmd_input; + } + if (!conn->authenticated) From dovecot at dovecot.org Sun Feb 15 08:11:30 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sun, 15 Feb 2015 08:11:30 +0000 Subject: dovecot-2.2: doveadm: Added "save" command to directly save mail... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d3bb7541ca5e changeset: 18255:d3bb7541ca5e user: Timo Sirainen date: Sun Feb 15 10:11:01 2015 +0200 description: doveadm: Added "save" command to directly save mail to specified mailbox. The mail is read from stdin. diffstat: src/doveadm/Makefile.am | 1 + src/doveadm/doveadm-mail-save.c | 123 ++++++++++++++++++++++++++++++++++++++++ src/doveadm/doveadm-mail.c | 1 + src/doveadm/doveadm-mail.h | 1 + 4 files changed, 126 insertions(+), 0 deletions(-) diffs (160 lines): diff -r 22a5eda76490 -r d3bb7541ca5e src/doveadm/Makefile.am --- a/src/doveadm/Makefile.am Sun Feb 15 10:09:19 2015 +0200 +++ b/src/doveadm/Makefile.am Sun Feb 15 10:11:01 2015 +0200 @@ -93,6 +93,7 @@ doveadm-mail-mailbox-status.c \ doveadm-mail-copymove.c \ doveadm-mailbox-list-iter.c \ + doveadm-mail-save.c \ doveadm-mail-search.c \ doveadm-mail-server.c diff -r 22a5eda76490 -r d3bb7541ca5e src/doveadm/doveadm-mail-save.c --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/src/doveadm/doveadm-mail-save.c Sun Feb 15 10:11:01 2015 +0200 @@ -0,0 +1,123 @@ +/* Copyright (c) 2015 Dovecot authors, see the included COPYING file */ + +#include "lib.h" +#include "istream.h" +#include "mail-storage.h" +#include "doveadm-mail.h" + +struct save_cmd_context { + struct doveadm_mail_cmd_context ctx; + const char *mailbox; +}; + +static int +cmd_save_to_mailbox(struct save_cmd_context *ctx, struct mailbox *box, + struct istream *input) +{ + struct mail_storage *storage = mailbox_get_storage(box); + struct mailbox_transaction_context *trans; + struct mail_save_context *save_ctx; + ssize_t ret; + bool save_failed = FALSE; + + if (mailbox_open(box) < 0) { + i_error("Failed to open mailbox %s: %s", + mailbox_get_vname(box), mailbox_get_last_error(box, NULL)); + doveadm_mail_failed_storage(&ctx->ctx, storage); + return -1; + } + + trans = mailbox_transaction_begin(box, MAILBOX_TRANSACTION_FLAG_EXTERNAL); + save_ctx = mailbox_save_alloc(trans); + if (mailbox_save_begin(&save_ctx, input) < 0) { + i_error("Saving failed: %s", mailbox_get_last_error(box, NULL)); + doveadm_mail_failed_storage(&ctx->ctx, storage); + mailbox_transaction_rollback(&trans); + return -1; + } + while ((ret = i_stream_read(input)) > 0 || ret == -2) { + if (mailbox_save_continue(save_ctx) < 0) { + save_failed = TRUE; + ret = -1; + break; + } + } + i_assert(ret == -1); + + if (input->stream_errno != 0) { + i_error("read(msg input) failed: %s", i_stream_get_error(input)); + doveadm_mail_failed_error(&ctx->ctx, MAIL_ERROR_TEMP); + } else if (save_failed) { + i_error("Saving failed: %s", mailbox_get_last_error(box, NULL)); + doveadm_mail_failed_storage(&ctx->ctx, storage); + } else if (mailbox_save_finish(&save_ctx) < 0) { + i_error("Saving failed: %s", + mailbox_get_last_error(box, NULL)); + doveadm_mail_failed_storage(&ctx->ctx, storage); + } else if (mailbox_transaction_commit(&trans) < 0) { + i_error("Save transaction commit failed: %s", + mailbox_get_last_error(box, NULL)); + doveadm_mail_failed_storage(&ctx->ctx, storage); + } else { + ret = 0; + } + if (save_ctx != NULL) + mailbox_save_cancel(&save_ctx); + if (trans != NULL) + mailbox_transaction_rollback(&trans); + i_assert(input->eof); + return ret < 0 ? -1 : 0; +} + +static int +cmd_save_run(struct doveadm_mail_cmd_context *_ctx, struct mail_user *user) +{ + struct save_cmd_context *ctx = (struct save_cmd_context *)_ctx; + struct mail_namespace *ns; + struct mailbox *box; + int ret; + + ns = mail_namespace_find(user->namespaces, ctx->mailbox); + box = mailbox_alloc(ns->list, ctx->mailbox, MAILBOX_FLAG_SAVEONLY); + ret = cmd_save_to_mailbox(ctx, box, _ctx->cmd_input); + mailbox_free(&box); + return ret; +} + +static void cmd_save_init(struct doveadm_mail_cmd_context *_ctx, + const char *const args[] ATTR_UNUSED) +{ + doveadm_mail_get_input(_ctx); +} + +static bool +cmd_mailbox_save_parse_arg(struct doveadm_mail_cmd_context *_ctx, int c) +{ + struct save_cmd_context *ctx = (struct save_cmd_context *)_ctx; + + switch (c) { + case 'm': + ctx->mailbox = optarg; + break; + default: + return FALSE; + } + return TRUE; +} + +static struct doveadm_mail_cmd_context *cmd_save_alloc(void) +{ + struct save_cmd_context *ctx; + + ctx = doveadm_mail_cmd_alloc(struct save_cmd_context); + ctx->ctx.getopt_args = "m:"; + ctx->ctx.v.parse_arg = cmd_mailbox_save_parse_arg; + ctx->ctx.v.init = cmd_save_init; + ctx->ctx.v.run = cmd_save_run; + ctx->mailbox = "INBOX"; + return &ctx->ctx; +} + +struct doveadm_mail_cmd cmd_save = { + cmd_save_alloc, "save", "[-m mailbox]" +}; diff -r 22a5eda76490 -r d3bb7541ca5e src/doveadm/doveadm-mail.c --- a/src/doveadm/doveadm-mail.c Sun Feb 15 10:09:19 2015 +0200 +++ b/src/doveadm/doveadm-mail.c Sun Feb 15 10:11:01 2015 +0200 @@ -787,6 +787,7 @@ &cmd_force_resync, &cmd_purge, &cmd_expunge, + &cmd_save, &cmd_search, &cmd_fetch, &cmd_flags_add, diff -r 22a5eda76490 -r d3bb7541ca5e src/doveadm/doveadm-mail.h --- a/src/doveadm/doveadm-mail.h Sun Feb 15 10:09:19 2015 +0200 +++ b/src/doveadm/doveadm-mail.h Sun Feb 15 10:11:01 2015 +0200 @@ -167,6 +167,7 @@ struct mailbox_list *list); extern struct doveadm_mail_cmd cmd_expunge; +extern struct doveadm_mail_cmd cmd_save; extern struct doveadm_mail_cmd cmd_search; extern struct doveadm_mail_cmd cmd_fetch; extern struct doveadm_mail_cmd cmd_flags_add; From dovecot at dovecot.org Tue Feb 17 14:03:13 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 14:03:13 +0000 Subject: dovecot-2.2: imapc: Fixed STATUS_FIRST_RECENT_UID to return the ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f13c6f9e0174 changeset: 18256:f13c6f9e0174 user: Timo Sirainen date: Tue Feb 17 16:02:49 2015 +0200 description: imapc: Fixed STATUS_FIRST_RECENT_UID to return the (mostly) correct UID. diffstat: src/lib-storage/index/imapc/imapc-mailbox.c | 10 ++++++++-- src/lib-storage/index/imapc/imapc-storage.c | 5 ++++- src/lib-storage/index/imapc/imapc-storage.h | 1 + 3 files changed, 13 insertions(+), 3 deletions(-) diffs (59 lines): diff -r d3bb7541ca5e -r f13c6f9e0174 src/lib-storage/index/imapc/imapc-mailbox.c --- a/src/lib-storage/index/imapc/imapc-mailbox.c Sun Feb 15 10:11:01 2015 +0200 +++ b/src/lib-storage/index/imapc/imapc-mailbox.c Tue Feb 17 16:02:49 2015 +0200 @@ -303,14 +303,20 @@ have_labels = TRUE; } } - /* FIXME: need to do something about recent flags */ - flags &= ~MAIL_RECENT; imapc_mailbox_init_delayed_trans(mbox); if (imapc_mailbox_msgmap_update(mbox, rseq, fetch_uid, &lseq, &uid) < 0 || uid == 0) return; + if ((flags & MAIL_RECENT) == 0 && mbox->highest_nonrecent_uid < uid) { + /* remember for STATUS_FIRST_RECENT_UID */ + mbox->highest_nonrecent_uid = uid; + } + /* FIXME: we should ideally also pass these through so they show up + to clients. */ + flags &= ~MAIL_RECENT; + /* if this is a reply to some FETCH request, update the mail's fields */ array_foreach(&mbox->fetch_requests, fetch_requestp) { array_foreach(&(*fetch_requestp)->mails, mailp) { diff -r d3bb7541ca5e -r f13c6f9e0174 src/lib-storage/index/imapc/imapc-storage.c --- a/src/lib-storage/index/imapc/imapc-storage.c Sun Feb 15 10:11:01 2015 +0200 +++ b/src/lib-storage/index/imapc/imapc-storage.c Tue Feb 17 16:02:49 2015 +0200 @@ -676,6 +676,8 @@ index_storage_get_open_status(&mbox->box, items, status_r); if ((items & STATUS_PERMANENT_FLAGS) != 0) status_r->permanent_flags = mbox->permanent_flags; + if ((items & STATUS_FIRST_RECENT_UID) != 0) + status_r->first_recent_uid = mbox->highest_nonrecent_uid + 1; } static int imapc_mailbox_delete(struct mailbox *box) @@ -735,7 +737,8 @@ if (box->opened) { imapc_mailbox_get_selected_status(mbox, items, status_r); } else if ((items & (STATUS_FIRST_UNSEEN_SEQ | STATUS_KEYWORDS | - STATUS_PERMANENT_FLAGS)) != 0) { + STATUS_PERMANENT_FLAGS | + STATUS_FIRST_RECENT_UID)) != 0) { /* getting these requires opening the mailbox */ if (mailbox_open(box) < 0) return -1; diff -r d3bb7541ca5e -r f13c6f9e0174 src/lib-storage/index/imapc/imapc-storage.h --- a/src/lib-storage/index/imapc/imapc-storage.h Sun Feb 15 10:11:01 2015 +0200 +++ b/src/lib-storage/index/imapc/imapc-storage.h Tue Feb 17 16:02:49 2015 +0200 @@ -101,6 +101,7 @@ ARRAY(struct imapc_mailbox_event_callback) resp_text_callbacks; enum mail_flags permanent_flags; + uint32_t highest_nonrecent_uid; ARRAY_TYPE(uint32_t) delayed_expunged_uids; uint32_t sync_uid_validity; From dovecot at dovecot.org Tue Feb 17 18:44:33 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 18:44:33 +0000 Subject: dovecot-2.2: lib-fs: Added flag for iteration returning object IDs. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/29bd2bcd56d6 changeset: 18257:29bd2bcd56d6 user: Timo Sirainen date: Tue Feb 17 20:42:59 2015 +0200 description: lib-fs: Added flag for iteration returning object IDs. The flag is only allowed to be used if FS_PROPERTY_OBJECTIDS is set (to avoid writing extra code for backends that don't support this). diffstat: src/lib-fs/fs-api.c | 3 +++ src/lib-fs/fs-api.h | 9 ++++++++- 2 files changed, 11 insertions(+), 1 deletions(-) diffs (39 lines): diff -r f13c6f9e0174 -r 29bd2bcd56d6 src/lib-fs/fs-api.c --- a/src/lib-fs/fs-api.c Tue Feb 17 16:02:49 2015 +0200 +++ b/src/lib-fs/fs-api.c Tue Feb 17 20:42:59 2015 +0200 @@ -712,6 +712,9 @@ { struct fs_iter *iter; + i_assert((flags & FS_ITER_FLAG_OBJECTIDS) == 0 || + (fs_get_properties(fs) & FS_PROPERTY_OBJECTIDS) != 0); + T_BEGIN { iter = fs->v.iter_init(fs, path, flags); } T_END; diff -r f13c6f9e0174 -r 29bd2bcd56d6 src/lib-fs/fs-api.h --- a/src/lib-fs/fs-api.h Tue Feb 17 16:02:49 2015 +0200 +++ b/src/lib-fs/fs-api.h Tue Feb 17 20:42:59 2015 +0200 @@ -28,6 +28,8 @@ FS_PROPERTY_COPY_METADATA = 0x400, /* Backend support asynchronous file operations. */ FS_PROPERTY_ASYNC = 0x800, + /* Backend supports FS_ITER_FLAG_OBJECTIDS. */ + FS_PROPERTY_OBJECTIDS = 0x1000 }; enum fs_open_mode { @@ -71,7 +73,12 @@ /* Iterate only directories, not files */ FS_ITER_FLAG_DIRS = 0x01, /* Request asynchronous iteration. */ - FS_ITER_FLAG_ASYNC = 0x02 + FS_ITER_FLAG_ASYNC = 0x02, + /* Instead of returning object names, return /. + If this isn't supported, the is returned empty. The + object IDs are always hex-encoded data. This flag can be used only + if FS_PROPERTY_OBJECTIDS is enabled. */ + FS_ITER_FLAG_OBJECTIDS = 0x04 }; struct fs_settings { From dovecot at dovecot.org Tue Feb 17 18:44:33 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 18:44:33 +0000 Subject: dovecot-2.2: lib-fs: Added FS_METADATA_OBJECTID macro. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/a27eed623099 changeset: 18258:a27eed623099 user: Timo Sirainen date: Tue Feb 17 20:43:57 2015 +0200 description: lib-fs: Added FS_METADATA_OBJECTID macro. diffstat: src/lib-fs/fs-api.h | 7 +++++++ src/lib-fs/fs-metawrap.c | 4 ++++ 2 files changed, 11 insertions(+), 0 deletions(-) diffs (31 lines): diff -r 29bd2bcd56d6 -r a27eed623099 src/lib-fs/fs-api.h --- a/src/lib-fs/fs-api.h Tue Feb 17 20:42:59 2015 +0200 +++ b/src/lib-fs/fs-api.h Tue Feb 17 20:43:57 2015 +0200 @@ -7,6 +7,13 @@ struct fs_lock; struct hash_method; +/* Metadata with this prefix shouldn't actually be sent to storage. */ +#define FS_METADATA_INTERNAL_PREFIX ":/X-Dovecot-fs-api-" +/* fs_write*() may return a hex-encoded object ID after write is finished. + This can be later on used to optimize reads by setting it before reading + the file. */ +#define FS_METADATA_OBJECTID FS_METADATA_INTERNAL_PREFIX"ObjectID" + enum fs_properties { FS_PROPERTY_METADATA = 0x01, FS_PROPERTY_LOCKS = 0x02, diff -r 29bd2bcd56d6 -r a27eed623099 src/lib-fs/fs-metawrap.c --- a/src/lib-fs/fs-metawrap.c Tue Feb 17 20:42:59 2015 +0200 +++ b/src/lib-fs/fs-metawrap.c Tue Feb 17 20:43:57 2015 +0200 @@ -286,6 +286,10 @@ const struct fs_metadata *metadata; array_foreach(&file->file.metadata, metadata) { + if (strncmp(metadata->key, FS_METADATA_INTERNAL_PREFIX, + strlen(FS_METADATA_INTERNAL_PREFIX)) == 0) + continue; + str_append_tabescaped(str, metadata->key); str_append_c(str, ':'); str_append_tabescaped(str, metadata->value); From dovecot at dovecot.org Tue Feb 17 20:06:37 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 20:06:37 +0000 Subject: dovecot-2.2: doveadm fetch: Removed unused code. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/f30bc964ac34 changeset: 18259:f30bc964ac34 user: Timo Sirainen date: Tue Feb 17 22:06:14 2015 +0200 description: doveadm fetch: Removed unused code. diffstat: src/doveadm/doveadm-mail-fetch.c | 12 ------------ 1 files changed, 0 insertions(+), 12 deletions(-) diffs (43 lines): diff -r a27eed623099 -r f30bc964ac34 src/doveadm/doveadm-mail-fetch.c --- a/src/doveadm/doveadm-mail-fetch.c Tue Feb 17 20:43:57 2015 +0200 +++ b/src/doveadm/doveadm-mail-fetch.c Tue Feb 17 22:06:14 2015 +0200 @@ -25,7 +25,6 @@ struct fetch_cmd_context { struct doveadm_mail_cmd_context ctx; - struct ostream *output; struct mail *mail; ARRAY(struct fetch_field) fields; @@ -525,13 +524,6 @@ return ret; } -static void cmd_fetch_deinit(struct doveadm_mail_cmd_context *_ctx) -{ - struct fetch_cmd_context *ctx = (struct fetch_cmd_context *)_ctx; - - o_stream_unref(&ctx->output); -} - static void cmd_fetch_init(struct doveadm_mail_cmd_context *_ctx, const char *const args[]) { @@ -543,9 +535,6 @@ parse_fetch_fields(ctx, fetch_fields); _ctx->search_args = doveadm_mail_build_search_args(args + 1); - - ctx->output = o_stream_create_fd(STDOUT_FILENO, 0, FALSE); - o_stream_set_no_error_handling(ctx->output, TRUE); } static struct doveadm_mail_cmd_context *cmd_fetch_alloc(void) @@ -555,7 +544,6 @@ ctx = doveadm_mail_cmd_alloc(struct fetch_cmd_context); ctx->ctx.v.init = cmd_fetch_init; ctx->ctx.v.run = cmd_fetch_run; - ctx->ctx.v.deinit = cmd_fetch_deinit; doveadm_print_init("pager"); return &ctx->ctx; } From dovecot at dovecot.org Tue Feb 17 21:14:46 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 21:14:46 +0000 Subject: dovecot-2.2: doveadm fs delete: When recursively deleting files,... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/359bc38c5e90 changeset: 18260:359bc38c5e90 user: Timo Sirainen date: Tue Feb 17 23:07:00 2015 +0200 description: doveadm fs delete: When recursively deleting files, delete directories with "/" prefix. Some backends require this to properly delete the file. Those that don't can ignore it easily. diffstat: src/doveadm/doveadm-fs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r f30bc964ac34 -r 359bc38c5e90 src/doveadm/doveadm-fs.c --- a/src/doveadm/doveadm-fs.c Tue Feb 17 22:06:14 2015 +0200 +++ b/src/doveadm/doveadm-fs.c Tue Feb 17 23:07:00 2015 +0200 @@ -263,7 +263,7 @@ t_array_init(&fnames, 8); iter = fs_iter_init(fs, path, FS_ITER_FLAG_DIRS); while ((fname = fs_iter_next(iter)) != NULL) { - fname = t_strdup(fname); + fname = t_strconcat(fname, "/", NULL); array_append(&fnames, &fname, 1); } if (fs_iter_deinit(&iter) < 0) { From dovecot at dovecot.org Tue Feb 17 21:14:46 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 21:14:46 +0000 Subject: dovecot-2.2: doveadm fs delete -R: Delete the root directory also. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/d7d35fd179e8 changeset: 18261:d7d35fd179e8 user: Timo Sirainen date: Tue Feb 17 23:13:00 2015 +0200 description: doveadm fs delete -R: Delete the root directory also. diffstat: src/doveadm/doveadm-fs.c | 16 +++++++++++++++- 1 files changed, 15 insertions(+), 1 deletions(-) diffs (28 lines): diff -r 359bc38c5e90 -r d7d35fd179e8 src/doveadm/doveadm-fs.c --- a/src/doveadm/doveadm-fs.c Tue Feb 17 23:07:00 2015 +0200 +++ b/src/doveadm/doveadm-fs.c Tue Feb 17 23:13:00 2015 +0200 @@ -335,9 +335,23 @@ cmd_fs_delete_recursive(int argc, char *argv[], unsigned int async_count) { struct fs *fs; + struct fs_file *file; + const char *path; fs = cmd_fs_init(&argc, &argv, 1, cmd_fs_delete); - cmd_fs_delete_dir_recursive(fs, async_count, argv[0]); + path = argv[0]; + + cmd_fs_delete_dir_recursive(fs, async_count, path); + if ((fs_get_properties(fs) & FS_PROPERTY_DIRECTORIES) != 0) { + /* delete the root itself */ + file = fs_file_init(fs, path, FS_OPEN_MODE_READONLY); + if (fs_delete(file) < 0) { + i_error("fs_delete(%s) failed: %s", + fs_file_path(file), fs_file_last_error(file)); + doveadm_exit_code = EX_TEMPFAIL; + } + fs_file_deinit(&file); + } fs_deinit(&fs); } From dovecot at dovecot.org Tue Feb 17 21:14:51 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 17 Feb 2015 21:14:51 +0000 Subject: dovecot-2.2: lib-fs: fs-posix needs to return FS_PROPERTY_DIRECT... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/87b22053f4cb changeset: 18262:87b22053f4cb user: Timo Sirainen date: Tue Feb 17 23:13:22 2015 +0200 description: lib-fs: fs-posix needs to return FS_PROPERTY_DIRECTORIES diffstat: src/lib-fs/fs-posix.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r d7d35fd179e8 -r 87b22053f4cb src/lib-fs/fs-posix.c --- a/src/lib-fs/fs-posix.c Tue Feb 17 23:13:00 2015 +0200 +++ b/src/lib-fs/fs-posix.c Tue Feb 17 23:13:22 2015 +0200 @@ -133,7 +133,8 @@ static enum fs_properties fs_posix_get_properties(struct fs *fs ATTR_UNUSED) { return FS_PROPERTY_LOCKS | FS_PROPERTY_FASTCOPY | FS_PROPERTY_RENAME | - FS_PROPERTY_STAT | FS_PROPERTY_ITER | FS_PROPERTY_RELIABLEITER; + FS_PROPERTY_STAT | FS_PROPERTY_ITER | FS_PROPERTY_RELIABLEITER | + FS_PROPERTY_DIRECTORIES; } static int fs_posix_mkdir_parents(struct posix_fs *fs, const char *path) From pigeonhole at rename-it.nl Thu Feb 19 00:00:24 2015 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Thu, 19 Feb 2015 01:00:24 +0100 Subject: dovecot-2.2-pigeonhole: managesieve: Forgot to dereference scrip... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/281a90a91f95 changeset: 1988:281a90a91f95 user: Stephan Bosch date: Thu Feb 19 01:00:19 2015 +0100 description: managesieve: Forgot to dereference script object in RENAMESCRIPT command. diffstat: src/managesieve/cmd-renamescript.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r 012ce8594501 -r 281a90a91f95 src/managesieve/cmd-renamescript.c --- a/src/managesieve/cmd-renamescript.c Sun Feb 01 13:23:15 2015 +0100 +++ b/src/managesieve/cmd-renamescript.c Thu Feb 19 01:00:19 2015 +0100 @@ -34,6 +34,7 @@ else client_send_ok(client, "Renamescript completed."); + sieve_script_unref(&script); return TRUE; } From dovecot at dovecot.org Thu Feb 19 08:24:34 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 08:24:34 +0000 Subject: dovecot-2.2: Makefile: Removed run-test.sh from EXTRA_DIST since... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/86d382b0c4d8 changeset: 18263:86d382b0c4d8 user: Timo Sirainen date: Thu Feb 19 10:24:09 2015 +0200 description: Makefile: Removed run-test.sh from EXTRA_DIST since it's now generated by configure diffstat: Makefile.am | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diffs (11 lines): diff -r 87b22053f4cb -r 86d382b0c4d8 Makefile.am --- a/Makefile.am Tue Feb 17 23:13:22 2015 +0200 +++ b/Makefile.am Thu Feb 19 10:24:09 2015 +0200 @@ -13,7 +13,6 @@ COPYING.MIT \ ChangeLog \ is-tagged.py \ - run-test.sh \ cc-wrapper.sh.in \ update-version.sh \ $(conf_DATA) From dovecot at dovecot.org Thu Feb 19 10:00:36 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 10:00:36 +0000 Subject: dovecot-2.2: fs-posix: Removed the FS_PROPERTY_DIRECTORIES after... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/faed0c7403f8 changeset: 18264:faed0c7403f8 user: Timo Sirainen date: Thu Feb 19 12:00:11 2015 +0200 description: fs-posix: Removed the FS_PROPERTY_DIRECTORIES after all. We could have kept it if we removed the auto-rmdir()-parents feature in fs_delete(), but SIS code already somewhat relies on it so it's better not to change it for now at least. The downside here though is that directories are rmdir()ed only if fs_settings.root_dir is set. So for example "doveadm fs delete -R" doesn't ever rmdir() any directories. But that's probably not a real problem. diffstat: src/lib-fs/fs-posix.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) diffs (17 lines): diff -r 86d382b0c4d8 -r faed0c7403f8 src/lib-fs/fs-posix.c --- a/src/lib-fs/fs-posix.c Thu Feb 19 10:24:09 2015 +0200 +++ b/src/lib-fs/fs-posix.c Thu Feb 19 12:00:11 2015 +0200 @@ -132,9 +132,11 @@ static enum fs_properties fs_posix_get_properties(struct fs *fs ATTR_UNUSED) { + /* FS_PROPERTY_DIRECTORIES not returned because fs_delete() + automatically rmdir()s parents. This could be changed later though, + but SIS code at least would need to be changed to support it. */ return FS_PROPERTY_LOCKS | FS_PROPERTY_FASTCOPY | FS_PROPERTY_RENAME | - FS_PROPERTY_STAT | FS_PROPERTY_ITER | FS_PROPERTY_RELIABLEITER | - FS_PROPERTY_DIRECTORIES; + FS_PROPERTY_STAT | FS_PROPERTY_ITER | FS_PROPERTY_RELIABLEITER; } static int fs_posix_mkdir_parents(struct posix_fs *fs, const char *path) From dovecot at dovecot.org Thu Feb 19 11:04:13 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 11:04:13 +0000 Subject: dovecot-2.2: lib: Replaced two test_assert()s checking for NULLs... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/c8aeb0cb4a6e changeset: 18265:c8aeb0cb4a6e user: Timo Sirainen date: Thu Feb 19 13:03:46 2015 +0200 description: lib: Replaced two test_assert()s checking for NULLs with i_assert()s These can never happen anyway, and keeping them as test_assert()s cause static analyzer to give warnings. diffstat: src/lib/test-data-stack.c | 2 +- src/lib/test-printf-format-fix.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diffs (24 lines): diff -r faed0c7403f8 -r c8aeb0cb4a6e src/lib/test-data-stack.c --- a/src/lib/test-data-stack.c Thu Feb 19 12:00:11 2015 +0200 +++ b/src/lib/test-data-stack.c Thu Feb 19 13:03:46 2015 +0200 @@ -102,7 +102,7 @@ for (i = 0; i < number; i++) { ps[i] = t_malloc(size/2); bool re = t_try_realloc(ps[i], size); - test_assert_idx(ps[i] != NULL, i); + i_assert(ps[i] != NULL); if (!re) { try_fails++; ps[i] = t_malloc(size); diff -r faed0c7403f8 -r c8aeb0cb4a6e src/lib/test-printf-format-fix.c --- a/src/lib/test-printf-format-fix.c Thu Feb 19 12:00:11 2015 +0200 +++ b/src/lib/test-printf-format-fix.c Thu Feb 19 13:03:46 2015 +0200 @@ -52,7 +52,7 @@ errno = EINVAL; needle = strerror(errno); - test_assert(needle != NULL); + i_assert(needle != NULL); needlen = strlen(needle); for (i = 0; i < N_ELEMENTS(tests); ++i) { From dovecot at dovecot.org Thu Feb 19 11:06:14 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 11:06:14 +0000 Subject: dovecot-2.2: lib: Another test_assert() -> i_assert() change to ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/ca7441fc784e changeset: 18266:ca7441fc784e user: Timo Sirainen date: Thu Feb 19 13:05:31 2015 +0200 description: lib: Another test_assert() -> i_assert() change to avoid static analyzer warnings. diffstat: src/lib/test-data-stack.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r c8aeb0cb4a6e -r ca7441fc784e src/lib/test-data-stack.c --- a/src/lib/test-data-stack.c Thu Feb 19 13:03:46 2015 +0200 +++ b/src/lib/test-data-stack.c Thu Feb 19 13:05:31 2015 +0200 @@ -96,7 +96,7 @@ int try_fails = 0; unsigned int t_id = t_push_named("test_ds_recurse[%i]", depth); ps = t_buffer_get_type(char *, number); - test_assert_idx(ps != NULL, depth); + i_assert(ps != NULL); t_buffer_alloc_type(char *, number); for (i = 0; i < number; i++) { From dovecot at dovecot.org Thu Feb 19 11:06:14 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 11:06:14 +0000 Subject: dovecot-2.2: lib: Make static analyzer happier Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/8d95e460094d changeset: 18267:8d95e460094d user: Timo Sirainen date: Thu Feb 19 13:05:49 2015 +0200 description: lib: Make static analyzer happier diffstat: src/lib/test-istream.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diffs (23 lines): diff -r ca7441fc784e -r 8d95e460094d src/lib/test-istream.c --- a/src/lib/test-istream.c Thu Feb 19 13:05:31 2015 +0200 +++ b/src/lib/test-istream.c Thu Feb 19 13:05:49 2015 +0200 @@ -32,8 +32,7 @@ test_assert(size == 3 && memcmp(data, "123", 3) == 0); /* child1 check middle again.. the parent has been modified, so it can't return the original data (without some code changes). */ - data = i_stream_get_data(child1, &size); - test_assert(size == 0); + test_assert(i_stream_get_data_size(child1) == 0); i_stream_skip(child1, 3); /* child1 read end */ test_assert(i_stream_read(child1) == 3); @@ -42,8 +41,7 @@ i_stream_skip(child1, 3); test_assert(i_stream_read(child1) == -1); /* child2 check beginning again.. */ - data = i_stream_get_data(child2, &size); - test_assert(size == 0); + test_assert(i_stream_get_data_size(child1) == 0); i_stream_skip(child2, 3); /* child2 read middle */ test_assert(i_stream_read(child2) == 3); From dovecot at dovecot.org Thu Feb 19 11:45:48 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 11:45:48 +0000 Subject: dovecot-2.2: Makefile: Added run-test.sh to be cleaned by distclean Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/98eeca072492 changeset: 18268:98eeca072492 user: Timo Sirainen date: Thu Feb 19 13:45:23 2015 +0200 description: Makefile: Added run-test.sh to be cleaned by distclean diffstat: Makefile.am | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diffs (13 lines): diff -r 8d95e460094d -r 98eeca072492 Makefile.am --- a/Makefile.am Thu Feb 19 13:05:49 2015 +0200 +++ b/Makefile.am Thu Feb 19 13:45:23 2015 +0200 @@ -83,7 +83,8 @@ DISTCLEANFILES = \ $(top_builddir)/dovecot-version.h \ - $(top_builddir)/dovecot-config + $(top_builddir)/dovecot-config \ + $(top_builddir)/run-test.sh distcheck-hook: if which scan-build > /dev/null; then \ From dovecot at dovecot.org Thu Feb 19 17:32:15 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 19 Feb 2015 17:32:15 +0000 Subject: dovecot-2.2: lib-storage: Index rebuilding (for [sm]dbox) caused... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/4f3db8aec0a1 changeset: 18269:4f3db8aec0a1 user: Timo Sirainen date: Thu Feb 19 19:31:49 2015 +0200 description: lib-storage: Index rebuilding (for [sm]dbox) caused it to reset dovecot.index.cache file Although in some situations this might be wanted, usually it's not needed and simply makes the performance worse. If caching is explicitly unwanted the dovecot.index.cache file can be deleted manually for now. Perhaps there could be a separate doveadm force-resync parameter to do it as well. diffstat: src/lib-storage/index/index-rebuild.c | 5 +---- 1 files changed, 1 insertions(+), 4 deletions(-) diffs (16 lines): diff -r 98eeca072492 -r 4f3db8aec0a1 src/lib-storage/index/index-rebuild.c --- a/src/lib-storage/index/index-rebuild.c Thu Feb 19 13:45:23 2015 +0200 +++ b/src/lib-storage/index/index-rebuild.c Thu Feb 19 19:31:49 2015 +0200 @@ -160,11 +160,8 @@ index_mailbox_reset_uidvalidity(box); (void)mail_index_ext_lookup(box->index, "cache", &ctx->cache_ext_id); - /* open cache and read the caching decisions. we'll reset the cache in - case it contains any invalid data, but we want to preserve the - decisions. */ + /* open cache and read the caching decisions. */ (void)mail_cache_open_and_verify(ctx->box->cache); - mail_cache_reset(box->cache); /* if backup index file exists, try to use it */ index_dir = mailbox_get_index_path(box); From dovecot at dovecot.org Fri Feb 20 10:57:09 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 20 Feb 2015 10:57:09 +0000 Subject: dovecot-2.2: dsync: Added more debug output for mailbox renaming. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/52016e4cb3db changeset: 18270:52016e4cb3db user: Timo Sirainen date: Fri Feb 20 12:56:43 2015 +0200 description: dsync: Added more debug output for mailbox renaming. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 152 +++++++++++++++++++++------ 1 files changed, 119 insertions(+), 33 deletions(-) diffs (truncated from 377 to 300 lines): diff -r 4f3db8aec0a1 -r 52016e4cb3db src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Thu Feb 19 19:31:49 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Fri Feb 20 12:56:43 2015 +0200 @@ -377,7 +377,8 @@ sync_rename_node_to_temp(struct dsync_mailbox_tree_sync_ctx *ctx, struct dsync_mailbox_tree *tree, struct dsync_mailbox_node *node, - struct dsync_mailbox_node *new_parent) + struct dsync_mailbox_node *new_parent, + const char **reason_r) { struct dsync_mailbox_tree_sync_change *change; const char *old_name, *new_name, *p; @@ -411,6 +412,7 @@ old_name = tree != ctx->local_tree ? NULL : dsync_mailbox_node_get_full_name(tree, node); + *reason_r = t_strdup_printf("Renamed '%s' to '%s'", node->name, str_c(&buf)); node->name = p_strdup(tree->pool, str_c(&buf)); node->sync_temporary_name = TRUE; node->last_renamed_or_created = 0; @@ -445,7 +447,8 @@ struct dsync_mailbox_tree *tree, struct dsync_mailbox_node *temp_node, struct dsync_mailbox_node *node, - const struct dsync_mailbox_node *other_node) + const struct dsync_mailbox_node *other_node, + const char **reason_r) { struct dsync_mailbox_tree_sync_change *change; struct dsync_mailbox_tree *other_tree; @@ -467,15 +470,18 @@ if (node_has_parent(parent, node)) { /* don't introduce a loop. temporarily rename node under root. */ - sync_rename_node_to_temp(ctx, tree, node, &tree->root); + sync_rename_node_to_temp(ctx, tree, node, &tree->root, reason_r); + *reason_r = t_strconcat(*reason_r, " (Don't introduce loop)", NULL); return; } - sync_rename_node_to_temp(ctx, tree, temp_node, temp_node->parent); + sync_rename_node_to_temp(ctx, tree, temp_node, temp_node->parent, reason_r); /* get the old name before it's modified */ name = dsync_mailbox_node_get_full_name(tree, node); /* set the new name */ + *reason_r = t_strdup_printf("%s + Renamed '%s' to '%s'", + *reason_r, node->name, other_node->name); node->name = p_strdup(tree->pool, other_node->name); node->sync_temporary_name = other_node->sync_temporary_name; node->last_renamed_or_created = other_node->last_renamed_or_created; @@ -584,7 +590,8 @@ struct dsync_mailbox_node *local_node1, struct dsync_mailbox_node *remote_node1, struct dsync_mailbox_node *local_node2, - struct dsync_mailbox_node *remote_node2) + struct dsync_mailbox_node *remote_node2, + const char **reason_r) { time_t local_ts, remote_ts; @@ -627,11 +634,13 @@ /* local : 1A remote: 1B, 2A -> 2A-temp, 1A */ sync_rename_node(ctx, ctx->remote_tree, remote_node2, - remote_node1, local_node1); + remote_node1, local_node1, reason_r); + *reason_r = t_strconcat(*reason_r, "(local: local_node2=NULL)", NULL); return TRUE; } else if (remote_node1 == remote_node2) { /* FIXME: this fixes an infinite loop when in rename2 test, think it through why :) */ + *reason_r = "local: remote_node1=remote_node2"; } else if (remote_node1 != NULL) { /* a) local_node1->parent == local_node2->parent @@ -651,52 +660,65 @@ always renaming 2 to a temporary name and handling it when we reach B handling. */ sync_rename_node(ctx, ctx->remote_tree, remote_node2, - remote_node1, local_node1); + remote_node1, local_node1, reason_r); + *reason_r = t_strconcat(*reason_r, "(local: remote_node1=NULL)", NULL); return TRUE; } else if (node_has_parent(local_node1, local_node2)) { /* node2 is a parent of node1, but it should be vice versa */ sync_rename_node_to_temp(ctx, ctx->local_tree, - local_node1, local_node2->parent); + local_node1, local_node2->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(local: node2 parent of node1)", NULL); return TRUE; } else if (node_has_parent(local_node2, local_node1)) { /* node1 is a parent of node2, but it should be vice versa */ sync_rename_node_to_temp(ctx, ctx->local_tree, - local_node2, local_node1->parent); + local_node2, local_node1->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(local: node1 parent of node2)", NULL); return TRUE; } else if (local_node1->existence == DSYNC_MAILBOX_NODE_EXISTS) { sync_rename_node_to_temp(ctx, ctx->remote_tree, - remote_node2, remote_node2->parent); + remote_node2, remote_node2->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(local: local_node1 exists)", NULL); return TRUE; } else { /* local : 1A, 2B remote: 2A -> (2B) remote: 2A, 3B -> (3B-temp, 2B) */ + *reason_r = "local: unchanged"; } } else { /* remote nodes have a higher timestamp */ if (remote_node1 == NULL) { sync_rename_node(ctx, ctx->local_tree, local_node1, - local_node2, remote_node2); + local_node2, remote_node2, reason_r); + *reason_r = t_strconcat(*reason_r, "(remote: remote_node1=NULL)", NULL); return TRUE; } else if (local_node2 == local_node1) { + *reason_r = "remote: remote_node2=remote_node1"; } else if (local_node2 != NULL) { sync_rename_node(ctx, ctx->local_tree, local_node1, - local_node2, remote_node2); + local_node2, remote_node2, reason_r); + *reason_r = t_strconcat(*reason_r, "(remote: local_node2=NULL)", NULL); return TRUE; } else if (node_has_parent(remote_node1, remote_node2)) { sync_rename_node_to_temp(ctx, ctx->remote_tree, - remote_node1, remote_node2->parent); + remote_node1, remote_node2->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(remote: node2 parent of node1)", NULL); return TRUE; } else if (node_has_parent(remote_node2, remote_node1)) { sync_rename_node_to_temp(ctx, ctx->remote_tree, - remote_node2, remote_node1->parent); + remote_node2, remote_node1->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(remote: node1 parent of node2)", NULL); return TRUE; } else if (remote_node2->existence == DSYNC_MAILBOX_NODE_EXISTS) { sync_rename_node_to_temp(ctx, ctx->local_tree, - local_node1, local_node1->parent); + local_node1, local_node1->parent, reason_r); + *reason_r = t_strconcat(*reason_r, "(remote: remote_node2 exists)", NULL); return TRUE; + } else { + *reason_r = "remote: unchanged"; } } return FALSE; @@ -704,10 +726,12 @@ static bool sync_rename_conflict(struct dsync_mailbox_tree_sync_ctx *ctx, struct dsync_mailbox_node *local_node1, - struct dsync_mailbox_node *remote_node2) + struct dsync_mailbox_node *remote_node2, + const char **reason_r) { struct dsync_mailbox_node *local_node2, *remote_node1; const uint8_t *guid_p; + bool ret; guid_p = local_node1->mailbox_guid; remote_node1 = hash_table_lookup(ctx->remote_tree->guid_hash, guid_p); @@ -717,13 +741,16 @@ if ((remote_node1 != NULL && remote_node1->existence == DSYNC_MAILBOX_NODE_EXISTS) || (local_node2 != NULL && local_node2->existence == DSYNC_MAILBOX_NODE_EXISTS)) { /* conflicting name, rename the one with lower timestamp */ - return sync_rename_lower_ts(ctx, local_node1, remote_node1, - local_node2, remote_node2); + ret = sync_rename_lower_ts(ctx, local_node1, remote_node1, + local_node2, remote_node2, reason_r); + *reason_r = t_strconcat("conflicting name: ", *reason_r, NULL); + return ret; } else if (dsync_mailbox_node_is_dir(local_node1) || dsync_mailbox_node_is_dir(remote_node2)) { /* one of the nodes is a directory, and the other is a mailbox that doesn't exist on the other side. there is no conflict, we'll just need to create the mailbox later. */ + *reason_r = "mailbox not selectable yet"; return FALSE; } else { /* both nodes are mailboxes that don't exist on the other side. @@ -731,6 +758,7 @@ GUIDs and UIDVALIDITYs to be the same */ local_node1->sync_delayed_guid_change = TRUE; remote_node2->sync_delayed_guid_change = TRUE; + *reason_r = "GUIDs conflict - will be merged later"; return FALSE; } } @@ -769,7 +797,8 @@ static bool sync_rename_directory(struct dsync_mailbox_tree_sync_ctx *ctx, struct dsync_mailbox_node *local_node1, - struct dsync_mailbox_node *remote_node2) + struct dsync_mailbox_node *remote_node2, + const char **reason_r) { struct dsync_mailbox_node *remote_node1, *local_node2; @@ -780,16 +809,24 @@ ctx->remote_tree, local_node1); local_node2 = sync_find_branch(ctx->remote_tree, ctx->local_tree, remote_node2); - if (remote_node1 == NULL || local_node2 == NULL || - node_names_equal(remote_node1, local_node2) || - sync_node_is_namespace_root(ctx->remote_tree, remote_node1) || + if (remote_node1 == NULL || local_node2 == NULL) { + *reason_r = "Directory rename branch not found"; + return FALSE; + } + if (node_names_equal(remote_node1, local_node2)) { + *reason_r = "Directory name paths are equal"; + return FALSE; + } + if (sync_node_is_namespace_root(ctx->remote_tree, remote_node1) || sync_node_is_namespace_root(ctx->remote_tree, remote_node2) || sync_node_is_namespace_root(ctx->local_tree, local_node1) || - sync_node_is_namespace_root(ctx->local_tree, local_node2)) + sync_node_is_namespace_root(ctx->local_tree, local_node2)) { + *reason_r = "Directory is part of namespace prefix"; return FALSE; + } return sync_rename_lower_ts(ctx, local_node1, remote_node1, - local_node2, remote_node2); + local_node2, remote_node2, reason_r); } static bool sync_rename_mailboxes(struct dsync_mailbox_tree_sync_ctx *ctx, @@ -799,8 +836,13 @@ struct dsync_mailbox_node **local_nodep = &local_parent->first_child; struct dsync_mailbox_node **remote_nodep = &remote_parent->first_child; struct dsync_mailbox_node *local_node, *remote_node; + const char *reason; + string_t *debug = NULL; bool changed; + if ((ctx->sync_flags & DSYNC_MAILBOX_TREES_SYNC_FLAG_DEBUG) != 0) + debug = t_str_new(128); + /* the nodes are sorted by their names. */ while (*local_nodep != NULL || *remote_nodep != NULL) { local_node = *local_nodep; @@ -820,28 +862,44 @@ remote_parent, local_node); } i_assert(strcmp(local_node->name, remote_node->name) == 0); + if (debug != NULL) { + str_truncate(debug, 0); + str_printfa(debug, "Mailbox %s: local=%s/%ld/%d, remote=%s/%ld/%d", + local_node->name, + guid_128_to_string(local_node->mailbox_guid), + (long)local_node->last_renamed_or_created, + local_node->existence, + guid_128_to_string(remote_node->mailbox_guid), + (long)remote_node->last_renamed_or_created, + remote_node->existence); + } if (dsync_mailbox_node_is_dir(local_node) && dsync_mailbox_node_is_dir(remote_node)) { /* both nodes are directories (or other side is nonexistent). see if we can match them by their child mailboxes */ - if (sync_rename_directory(ctx, local_node, remote_node)) - return TRUE; + changed = sync_rename_directory(ctx, local_node, remote_node, &reason); } else if (dsync_mailbox_node_guids_equal(local_node, remote_node)) { /* mailboxes are equal, no need to rename */ + reason = "Mailboxes are equal"; + changed = FALSE; } else { /* mailbox naming conflict */ T_BEGIN { changed = sync_rename_conflict(ctx, local_node, - remote_node); + remote_node, &reason); } T_END; - if (changed) - return TRUE; } /* handle children, if there are any */ - T_BEGIN { + if (debug != NULL) { + i_debug("brain %c: %s: %s", + (ctx->sync_flags & DSYNC_MAILBOX_TREES_SYNC_FLAG_MASTER_BRAIN) != 0 ? 'M' : 'S', + str_c(debug), reason); + } + + if (!changed) T_BEGIN { changed = sync_rename_mailboxes(ctx, local_node, remote_node); } T_END; @@ -911,7 +969,8 @@ From dovecot at dovecot.org Fri Feb 20 12:04:54 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 20 Feb 2015 12:04:54 +0000 Subject: dovecot-2.2: LAYOUT=index: Reversed d977a746819d - use storage's... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/18c24e201697 changeset: 18271:18c24e201697 user: Timo Sirainen date: Fri Feb 20 14:04:28 2015 +0200 description: LAYOUT=index: Reversed d977a746819d - use storage's list_index_*() callbacks after all. Just because we have all the mailbox information in the mailbox list index doesn't mean that we necessarily trust the folder mail counts and such to be correct. Setting mailbox_list_index_very_dirty_syncs=yes pretty much reverts to the earlier behavior. diffstat: src/lib-storage/list/mailbox-list-index-backend.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) diffs (11 lines): diff -r 52016e4cb3db -r 18c24e201697 src/lib-storage/list/mailbox-list-index-backend.c --- a/src/lib-storage/list/mailbox-list-index-backend.c Fri Feb 20 12:56:43 2015 +0200 +++ b/src/lib-storage/list/mailbox-list-index-backend.c Fri Feb 20 14:04:28 2015 +0200 @@ -717,7 +717,4 @@ box->v.create_box = index_list_mailbox_create; box->v.update_box = index_list_mailbox_update; box->v.exists = index_list_mailbox_exists; - - box->v.list_index_has_changed = NULL; - box->v.list_index_update_sync = NULL; } From dovecot at dovecot.org Sat Feb 21 09:41:08 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Sat, 21 Feb 2015 09:41:08 +0000 Subject: dovecot-2.2: dsync: Debug logging was attempting to read already... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/466596200825 changeset: 18272:466596200825 user: Timo Sirainen date: Sat Feb 21 11:40:40 2015 +0200 description: dsync: Debug logging was attempting to read already freed memory. diffstat: src/doveadm/dsync/dsync-mailbox-tree-sync.c | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diffs (16 lines): diff -r 18c24e201697 -r 466596200825 src/doveadm/dsync/dsync-mailbox-tree-sync.c --- a/src/doveadm/dsync/dsync-mailbox-tree-sync.c Fri Feb 20 14:04:28 2015 +0200 +++ b/src/doveadm/dsync/dsync-mailbox-tree-sync.c Sat Feb 21 11:40:40 2015 +0200 @@ -887,10 +887,8 @@ changed = FALSE; } else { /* mailbox naming conflict */ - T_BEGIN { - changed = sync_rename_conflict(ctx, local_node, - remote_node, &reason); - } T_END; + changed = sync_rename_conflict(ctx, local_node, + remote_node, &reason); } /* handle children, if there are any */ if (debug != NULL) { From pigeonhole at rename-it.nl Sat Feb 21 17:33:58 2015 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Sat, 21 Feb 2015 18:33:58 +0100 Subject: dovecot-2.2-pigeonhole: doveadm sieve plugin: Implemented comman... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/7432a425ac63 changeset: 1989:7432a425ac63 user: Stephan Bosch date: Sat Feb 21 17:28:22 2015 +0100 description: doveadm sieve plugin: Implemented commands for managing Sieve scripts. This allows performing ManageSieve actions at command line or through doveadm server. diffstat: .hgignore | 1 + doc/man/Makefile.am | 30 +- doc/man/doveadm-sieve.1.in | 122 ++ doc/man/global-options-formatter.inc | 46 + doc/man/global-options.inc | 21 + doc/man/option-A.inc | 27 + doc/man/option-S-socket.inc | 10 + doc/man/option-u-user.inc | 20 + doc/man/pigeonhole.7.in | 16 +- doc/man/reporting-bugs.inc | 2 +- doc/man/sed.sh | 27 +- src/lib-sieve/sieve-script.c | 6 +- src/lib-sieve/sieve-script.h | 2 +- src/managesieve/cmd-deletescript.c | 8 +- src/plugins/doveadm-sieve/Makefile.am | 15 +- src/plugins/doveadm-sieve/doveadm-sieve-cmd-activate.c | 136 +++ src/plugins/doveadm-sieve/doveadm-sieve-cmd-delete.c | 115 ++ src/plugins/doveadm-sieve/doveadm-sieve-cmd-get.c | 75 + src/plugins/doveadm-sieve/doveadm-sieve-cmd-list.c | 74 + src/plugins/doveadm-sieve/doveadm-sieve-cmd-put.c | 180 ++++ src/plugins/doveadm-sieve/doveadm-sieve-cmd-rename.c | 77 + src/plugins/doveadm-sieve/doveadm-sieve-cmd.c | 181 ++++ src/plugins/doveadm-sieve/doveadm-sieve-cmd.h | 48 + src/plugins/doveadm-sieve/doveadm-sieve-plugin.c | 720 +---------------- src/plugins/doveadm-sieve/doveadm-sieve-plugin.h | 20 + src/plugins/doveadm-sieve/doveadm-sieve-sync.c | 727 +++++++++++++++++ 26 files changed, 1955 insertions(+), 751 deletions(-) diffs (truncated from 2940 to 300 lines): diff -r 281a90a91f95 -r 7432a425ac63 .hgignore --- a/.hgignore Thu Feb 19 01:00:19 2015 +0100 +++ b/.hgignore Sat Feb 21 17:28:22 2015 +0100 @@ -56,6 +56,7 @@ **/.libs **/.deps +doc/man/doveadm-sieve.1 doc/man/sievec.1 doc/man/sieve-dump.1 doc/man/sieve-test.1 diff -r 281a90a91f95 -r 7432a425ac63 doc/man/Makefile.am --- a/doc/man/Makefile.am Thu Feb 19 01:00:19 2015 +0100 +++ b/doc/man/Makefile.am Sat Feb 21 17:28:22 2015 +0100 @@ -1,10 +1,13 @@ pkgsysconfdir = $(sysconfdir)/dovecot rundir = ${prefix}/var/run/dovecot +SUFFIXES = .1.in .1 .7.in .7 + dist_man1_MANS = \ sieved.1 nodist_man1_MANS = \ + doveadm-sieve.1 \ sievec.1 \ sieve-dump.1 \ sieve-test.1 \ @@ -14,9 +17,15 @@ pigeonhole.7 man_includefiles = \ + $(srcdir)/global-options-formatter.inc \ + $(srcdir)/global-options.inc \ + $(srcdir)/option-A.inc \ + $(srcdir)/option-S-socket.inc \ + $(srcdir)/option-u-user.inc \ $(srcdir)/reporting-bugs.inc EXTRA_DIST = \ + doveadm-sieve.1.in \ sievec.1.in \ sieve-dump.1.in \ sieve-test.1.in \ @@ -27,22 +36,11 @@ CLEANFILES = $(nodist_man1_MANS) $(nodist_man7_MANS) -sievec.1: $(srcdir)/sievec.1.in $(man_includefiles) Makefile +.1.in.1: $(man_includefiles) Makefile $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ - < $(srcdir)/sievec.1.in > sievec.1 + $(pkglibexecdir) < $< > $@ +.7.in.7: $(man_includefiles) Makefile + $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ + $(pkglibexecdir) < $< > $@ -sieve-dump.1: $(srcdir)/sieve-dump.1.in $(man_includefiles) Makefile - $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ - < $(srcdir)/sieve-dump.1.in > sieve-dump.1 -sieve-filter.1: $(srcdir)/sieve-filter.1.in $(man_includefiles) Makefile - $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ - < $(srcdir)/sieve-filter.1.in > sieve-filter.1 - -sieve-test.1: $(srcdir)/sieve-test.1.in $(man_includefiles) Makefile - $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ - < $(srcdir)/sieve-test.1.in > sieve-test.1 - -pigeonhole.7: $(srcdir)/pigeonhole.7.in $(man_includefiles) Makefile - $(SHELL) $(srcdir)/sed.sh $(srcdir) $(rundir) $(pkgsysconfdir) \ - < $(srcdir)/pigeonhole.7.in > pigeonhole.7 diff -r 281a90a91f95 -r 7432a425ac63 doc/man/doveadm-sieve.1.in --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/man/doveadm-sieve.1.in Sat Feb 21 17:28:22 2015 +0100 @@ -0,0 +1,122 @@ +.\" Copyright (c) 2010-2015 Pigeonhole authors, see the included COPYING file +.TH DOVEADM\-SIEVE 1 "2015-02-21" "Pigeonhole v0.4 for Dovecot v2.2" "Pigeonhole" +.SH NAME +doveadm\-sieve \- Commands related to handling Sieve scripts +.\"------------------------------------------------------------------------ +.SH SYNOPSIS +.BR doveadm " [" \-Dv "] [" \-f +.IR formatter "] " sieve_cmd " [" options "] [" arguments ] +.\"------------------------------------------------------------------------ +.SH DESCRIPTION +.PP +The +.B doveadm sieve +commands are part of the Pigeonhole Project (\fBpigeonhole\fR(7)), which adds +Sieve (RFC 5228) and ManageSieve (RFC 5804) support to the Dovecot secure IMAP +and POP3 server (\fBdovecot\fR(1)). The +.B doveadm sieve +commands can be used to manage Sieve filtering. +.\"------------------------------------------------------------------------ + at INCLUDE:global-options-formatter@ +.\" --- command specific options --- "/. +.PP +Command specific +.IR options : +.\"------------------------------------- + at INCLUDE:option-A@ +.\"------------------------------------- + at INCLUDE:option-S-socket@ +.\"------------------------------------- + at INCLUDE:option-u-user@ +.\"------------------------------------------------------------------------ +.SH ARGUMENTS +.TP +.I scriptname +Is the name of a +.IR Sieve\ script , +as visible to ManageSieve clients. +.\"------------------------------------------------------------------------ +.SH COMMANDS +.SS sieve put +.B doveadm sieve put +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.RB [ \-a ] +.IR scriptname +.PP +This command puts one new Sieve script in the script storage. The script +is read from standard input. If the script compiles successfully, it is stored +under the provided +.IR scriptname\ . +If the +.B \-a +option is present, the Sieve script is subsequently marked as the active script +for execution at delivery. +.\"------------------------------------------------------------------------ +.SS sieve get +.B doveadm sieve get +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.I scriptname +.PP +This command retrieves the Sieve script named +.IR scriptname . +.\"------------------------------------------------------------------------ +.SS sieve delete +.B doveadm sieve delete +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.RB [ \-a ] +.IR scriptname\ ... +.PP +This command deletes one or more Sieve scripts. The deleted script may not be the +active script, unless the +.B \-a +option is present. +.\"------------------------------------------------------------------------ +.SS sieve list +.B doveadm sieve list +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.I scriptname +.PP +Use this command to get an overview of existing Sieve scripts. +.\"------------------------------------------------------------------------ +.SS sieve rename +.B doveadm sieve rename +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.I old_name +.I new_name +.PP +The +.B sieve rename +command is used to rename the Sieve script +.I old_name +to +.IR new_name . +.\"------------------------------------------------------------------------ +.SS sieve activate +.B doveadm sieve activate +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.IR scriptname +.PP +This command marks the Sieve script named +.I scriptname +as the active script for execution at delivery. +.\"------------------------------------------------------------------------ +.SS sieve deactivate +.B doveadm sieve deactivate +[\fB\-A\fP|\fB\-u\fP \fIuser\fP] +[\fB\-S\fP \fIsocket_path\fP] +.I scriptname +.PP +This command deactivates Sieve processing. +.\"------------------------------------------------------------------------ + at INCLUDE:reporting-bugs@ +.\"------------------------------------------------------------------------ +.SH SEE ALSO +.BR doveadm (1) +.BR dovecot\-lda (1), +.BR pigeonhole (7) diff -r 281a90a91f95 -r 7432a425ac63 doc/man/global-options-formatter.inc --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/man/global-options-formatter.inc Sat Feb 21 17:28:22 2015 +0100 @@ -0,0 +1,46 @@ +.SH OPTIONS +Global +.BR doveadm (1) +.IR options : +.TP +.B \-D +Enables verbosity and debug messages. +.TP +.BI \-f\ formatter +Specifies the +.I formatter +for formatting the output. +Supported formatters are: +.RS +.TP +.B flow +prints each line with +.IB key = value +pairs. +.TP +.B pager +prints each +.IR key :\ value +pair on its own line and separates records with form feed character +.RB ( ^L ). +.TP +.B tab +prints a table header followed by tab separated value lines. +.TP +.B table +prints a table header followed by adjusted value lines. +.RE +.TP +.BI \-o\ setting = value +Overrides the configuration +.I setting +from +.I @pkgsysconfdir@/dovecot.conf +and from the userdb with the given +.IR value . +In order to override multiple settings, the +.B \-o +option may be specified multiple times. +.TP +.B \-v +Enables verbosity, including progress counter. diff -r 281a90a91f95 -r 7432a425ac63 doc/man/global-options.inc --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/man/global-options.inc Sat Feb 21 17:28:22 2015 +0100 @@ -0,0 +1,21 @@ +.SH OPTIONS +Global +.BR doveadm (1) +.IR options : +.TP +.B \-D +Enables verbosity and debug messages. +.TP +.BI \-o\ setting = value +Overrides the configuration +.I setting +from +.I @pkgsysconfdir@/dovecot.conf +and from the userdb with the given +.IR value . +In order to override multiple settings, the +.B \-o +option may be specified multiple times. +.TP +.B \-v +Enables verbosity, including progress counter. diff -r 281a90a91f95 -r 7432a425ac63 doc/man/option-A.inc --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/doc/man/option-A.inc Sat Feb 21 17:28:22 2015 +0100 @@ -0,0 +1,27 @@ +.TP +.B \-A +If the +.B \-A +option is present, the +.I command +will be performed for all users. +Using this option in combination with system users from +.B userdb { driver = passwd } +is not recommended, because it contains also users with a lower UID than +the one configured with the +.I first_valid_uid +setting. +.sp +When the SQL userdb module is used make sure that the +.I iterate_query +setting in +.I @pkgsysconfdir@/dovecot\-sql.conf.ext +matches your database layout. +When using the LDAP userdb module, make sure that the +.IR iterate_attrs " and " iterate_filter +settings in +.I @pkgsysconfdir@/dovecot-ldap.conf.ext From pigeonhole at rename-it.nl Sat Feb 21 17:44:37 2015 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Sat, 21 Feb 2015 18:44:37 +0100 Subject: dovecot-2.2-pigeonhole: doveadm sieve plugin: Forgot to change o... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/e5cb9d585bc1 changeset: 1990:e5cb9d585bc1 user: Stephan Bosch date: Sat Feb 21 18:44:30 2015 +0100 description: doveadm sieve plugin: Forgot to change one sieve_script_delete() invocation in previous change. diffstat: src/plugins/doveadm-sieve/doveadm-sieve-sync.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diffs (18 lines): diff -r 7432a425ac63 -r e5cb9d585bc1 src/plugins/doveadm-sieve/doveadm-sieve-sync.c --- a/src/plugins/doveadm-sieve/doveadm-sieve-sync.c Sat Feb 21 17:28:22 2015 +0100 +++ b/src/plugins/doveadm-sieve/doveadm-sieve-sync.c Sat Feb 21 18:44:30 2015 +0100 @@ -115,7 +115,13 @@ int ret = 0; script = sieve_storage_open_script(svstorage, scriptname, NULL); - ret = script == NULL ? -1 : sieve_script_delete(&script); + if (script == NULL) { + ret = -1; + } else { + ret = sieve_script_delete(script); + sieve_script_unref(&script); + } + if (ret < 0) { errstr = sieve_storage_get_last_error(svstorage, &error); if (error == SIEVE_ERROR_NOT_FOUND) { From pigeonhole at rename-it.nl Mon Feb 23 08:21:19 2015 From: pigeonhole at rename-it.nl (pigeonhole at rename-it.nl) Date: Mon, 23 Feb 2015 09:21:19 +0100 Subject: dovecot-2.2-pigeonhole: lib-sieve: Fixed bug in `:matches' match... Message-ID: details: http://hg.rename-it.nl/dovecot-2.2-pigeonhole/rev/c8edece267cd changeset: 1991:c8edece267cd user: Stephan Bosch date: Mon Feb 23 09:21:10 2015 +0100 description: lib-sieve: Fixed bug in `:matches' match-type that made a pattern without wildcards match as if there were a '*' at the beginning. diffstat: src/lib-sieve/mcht-matches.c | 65 +++++++++++++++++++++++---------------- tests/match-types/matches.svtest | 28 +++++++++++++++++ 2 files changed, 66 insertions(+), 27 deletions(-) diffs (120 lines): diff -r e5cb9d585bc1 -r c8edece267cd src/lib-sieve/mcht-matches.c --- a/src/lib-sieve/mcht-matches.c Sat Feb 21 18:44:30 2015 +0100 +++ b/src/lib-sieve/mcht-matches.c Mon Feb 23 09:21:10 2015 +0100 @@ -183,42 +183,53 @@ pvp = vp; if ( next_wcard == '\0' ) { - /* No more wildcards; find the needle substring at the end of string */ + if ( wcard == '\0' ) { + /* No current wildcard; match needs to happen right at the beginning */ + debug_printf("next_wcard = NULL && wcard = NUL; needle should be equal to value.\n"); - const char *qp, *qend; + if ( (vend - vp) != (nend - needle) || + !cmp->def->char_match(cmp, &vp, vend, &needle, nend) ) { + debug_printf(" key not equal to value\n"); + break; + } - debug_printf("next_wcard = NUL; must find needle at end\n"); + } else { + const char *qp, *qend; - /* Check if the value is still large enough */ - if ( vend - str_len(section) < vp ) { - debug_printf(" wont match: value is too short\n"); - break; - } + /* No more wildcards; find the needle substring at the end of string */ + debug_printf("next_wcard = NUL; must find needle at end\n"); - /* Move value pointer to where the needle should be */ - vp = PTR_OFFSET(vend, -str_len(section)); + /* Check if the value is still large enough */ + if ( vend - str_len(section) < vp ) { + debug_printf(" wont match: value is too short\n"); + break; + } - /* Record match values */ - qend = vp; - qp = vp - key_offset; + /* Move value pointer to where the needle should be */ + vp = PTR_OFFSET(vend, -str_len(section)); - if ( mvalues != NULL ) - str_append_n(mvalue, pvp, qp-pvp); + /* Record match values */ + qend = vp; + qp = vp - key_offset; - /* Compare needle to end of value string */ - if ( !cmp->def->char_match(cmp, &vp, vend, &needle, nend) ) { - debug_printf(" match at end failed\n"); - break; - } + if ( mvalues != NULL ) + str_append_n(mvalue, pvp, qp-pvp); - /* Add match values */ - if ( mvalues != NULL ) { - /* Append '*' match value */ - sieve_match_values_add(mvalues, mvalue); + /* Compare needle to end of value string */ + if ( !cmp->def->char_match(cmp, &vp, vend, &needle, nend) ) { + debug_printf(" match at end failed\n"); + break; + } - /* Append any initial '?' match values */ - for ( ; qp < qend; qp++ ) - sieve_match_values_add_char(mvalues, *qp); + /* Add match values */ + if ( mvalues != NULL ) { + /* Append '*' match value */ + sieve_match_values_add(mvalues, mvalue); + + /* Append any initial '?' match values */ + for ( ; qp < qend; qp++ ) + sieve_match_values_add_char(mvalues, *qp); + } } /* Finish match */ diff -r e5cb9d585bc1 -r c8edece267cd tests/match-types/matches.svtest --- a/tests/match-types/matches.svtest Sat Feb 21 18:44:30 2015 +0100 +++ b/tests/match-types/matches.svtest Mon Feb 23 09:21:10 2015 +0100 @@ -210,4 +210,32 @@ } } +test "Fixed beginning" { + if not header :matches "subject" "make your *" { + test_fail "should have matched"; + } +} +test "Fixed end" { + if not header :matches "subject" "* very fast!!!" { + test_fail "should have matched"; + } + + if header :matches "subject" "* very fast!!" { + test_fail "should not have matched"; + } +} + +test "Fixed string" { + if not address :matches "to" "sirius at example.org" { + test_fail "should have matched"; + } + + if address :matches "to" "example.org" { + test_fail "should not have matched"; + } + + if address :matches "to" "sirius" { + test_fail "should not have matched"; + } +} From dovecot at dovecot.org Tue Feb 24 16:43:24 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Tue, 24 Feb 2015 16:43:24 +0000 Subject: dovecot-2.2: expire plugin: If expire_cache=yes, cache the dict ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/c2e46b4df488 changeset: 18273:c2e46b4df488 user: Timo Sirainen date: Tue Feb 24 18:42:43 2015 +0200 description: expire plugin: If expire_cache=yes, cache the dict db value in dovecot.index file. This avoids all of the dict lookups when mails are being saved to a mailbox that is tracked for expiring. The only downside is that if the dict is externally modified, the changes won't reflect the cached value. This isn't normally a problem (except maybe in initial testing stages). diffstat: src/plugins/expire/expire-plugin.c | 161 +++++++++++++++++++++++++++--------- 1 files changed, 119 insertions(+), 42 deletions(-) diffs (247 lines): diff -r 466596200825 -r c2e46b4df488 src/plugins/expire/expire-plugin.c --- a/src/plugins/expire/expire-plugin.c Sat Feb 21 11:40:40 2015 +0200 +++ b/src/plugins/expire/expire-plugin.c Tue Feb 24 18:42:43 2015 +0200 @@ -1,5 +1,10 @@ /* Copyright (c) 2006-2015 Dovecot authors, see the included COPYING file */ +/* There are several race conditions in this plugin, but they should be + happening pretty rarely and usually it's not a big problem if the results + are temporarily wrong. Fixing the races would likely be a lot of work, + so it's not really worth it. */ + #include "lib.h" #include "ioloop.h" #include "array.h" @@ -21,15 +26,21 @@ #define EXPIRE_USER_CONTEXT(obj) \ MODULE_CONTEXT(obj, expire_mail_user_module) +struct expire_mail_index_header { + uint32_t timestamp; +}; + struct expire_mail_user { union mail_user_module_context module_ctx; struct dict *db; struct expire_set *set; + bool expire_cache; }; struct expire_mailbox { union mailbox_module_context module_ctx; + uint32_t expire_ext_id; }; struct expire_transaction_context { @@ -62,6 +73,108 @@ return t; } +static void first_save_timestamp(struct mailbox *box, time_t *stamp_r) +{ + struct mailbox_transaction_context *t; + const struct mail_index_header *hdr; + struct mail *mail; + + *stamp_r = ioloop_time; + + t = mailbox_transaction_begin(box, 0); + mail = mail_alloc(t, 0, NULL); + + /* find the first non-expunged mail. we're here because the first + mail was expunged, so don't bother checking it. */ + hdr = mail_index_get_header(box->view); + if (hdr->messages_count > 0) { + mail_set_seq(mail, 1); + (void)mail_get_save_date(mail, stamp_r); + } + mail_free(&mail); + (void)mailbox_transaction_commit(&t); +} + +static uint32_t expire_get_ext_id(struct mailbox *box) +{ + struct expire_mailbox *xpr_box = EXPIRE_CONTEXT(box); + + if (xpr_box->expire_ext_id != (uint32_t)-1) + return xpr_box->expire_ext_id; + + xpr_box->expire_ext_id = + mail_index_ext_register(box->index, "expire", + sizeof(struct expire_mail_index_header), 0, 0); + return xpr_box->expire_ext_id; +} + +static int expire_lookup(struct mailbox *box, const char *key, + time_t *new_stamp_r) +{ + struct expire_mail_user *euser = + EXPIRE_USER_CONTEXT(box->storage->user); + const struct expire_mail_index_header *hdr; + const void *data; + size_t data_size; + const char *value; + int ret; + + /* default to ioloop_time for newly saved mails. it may not be exactly + the first message's save time, but a few seconds difference doesn't + matter */ + *new_stamp_r = ioloop_time; + + if (euser->expire_cache) { + mail_index_get_header_ext(box->view, expire_get_ext_id(box), + &data, &data_size); + if (data_size == sizeof(*hdr)) { + hdr = data; + if (hdr->timestamp == 0) + return 0; + /* preserve the original timestamp */ + *new_stamp_r = hdr->timestamp; + return 1; + } + /* cache doesn't exist yet */ + } + + ret = dict_lookup(euser->db, pool_datastack_create(), + key, &value); + if (ret <= 0) { + if (ret < 0) + return -1; + first_save_timestamp(box, new_stamp_r); + return 0; + } + return strcmp(value, "0") != 0 ? 1 : 0; +} + +static void +expire_update(struct mailbox *box, const char *key, time_t timestamp) +{ + struct expire_mail_user *euser = + EXPIRE_USER_CONTEXT(box->storage->user); + struct dict_transaction_context *dctx; + struct mail_index_transaction *trans; + struct expire_mail_index_header hdr; + + dctx = dict_transaction_begin(euser->db); + dict_set(dctx, key, dec2str(timestamp)); + if (dict_transaction_commit(&dctx) < 0) + i_error("expire: dict commit failed"); + else if (euser->expire_cache) { + memset(&hdr, 0, sizeof(hdr)); + hdr.timestamp = timestamp; + + trans = mail_index_transaction_begin(box->view, + MAIL_INDEX_TRANSACTION_FLAG_EXTERNAL); + mail_index_update_header_ext(trans, expire_get_ext_id(box), + 0, &hdr, sizeof(hdr)); + if (mail_index_transaction_commit(&trans) < 0) + i_error("expire: index transaction commit failed"); + } +} + static void first_nonexpunged_timestamp(struct mailbox_transaction_context *t, time_t *stamp_r) { @@ -90,34 +203,11 @@ } } -static void first_save_timestamp(struct mailbox *box, time_t *stamp_r) -{ - struct mailbox_transaction_context *t; - const struct mail_index_header *hdr; - struct mail *mail; - - *stamp_r = ioloop_time; - - t = mailbox_transaction_begin(box, 0); - mail = mail_alloc(t, 0, NULL); - - /* find the first non-expunged mail. we're here because the first - mail was expunged, so don't bother checking it. */ - hdr = mail_index_get_header(box->view); - if (hdr->messages_count > 0) { - mail_set_seq(mail, 1); - (void)mail_get_save_date(mail, stamp_r); - } - mail_free(&mail); - (void)mailbox_transaction_commit(&t); -} - static int expire_mailbox_transaction_commit(struct mailbox_transaction_context *t, struct mail_transaction_commit_changes *changes_r) { struct mail_user *user = t->box->storage->user; - struct expire_mail_user *euser = EXPIRE_USER_CONTEXT(user); struct expire_mailbox *xpr_box = EXPIRE_CONTEXT(t->box); struct expire_transaction_context *xt = EXPIRE_CONTEXT(t); struct mailbox *box = t->box; @@ -149,7 +239,7 @@ t = NULL; if (xt->first_expunged || xt->saves) T_BEGIN { - const char *key, *value; + const char *key; key = t_strconcat(DICT_EXPIRE_PREFIX, box->storage->user->username, "/", @@ -160,8 +250,7 @@ i_assert(xt->saves); /* saved new mails. dict needs to be updated only if this is the first mail in the database */ - ret = dict_lookup(euser->db, pool_datastack_create(), - key, &value); + ret = expire_lookup(box, key, &new_stamp); if (ret <= 0) { /* first time saving here with expire enabled. also handle lookup errors by just assuming @@ -170,14 +259,6 @@ i_warning("expire: dict lookup failed, " "assuming update is needed"); } - first_save_timestamp(box, &new_stamp); - update_dict = TRUE; - } else if (strcmp(value, "0") == 0) { - /* we're saving the first mail to this mailbox. - ioloop_time may not be exactly the first - message's save time, but a few seconds - difference doesn't matter */ - new_stamp = ioloop_time; update_dict = TRUE; } else { /* already exists */ @@ -189,14 +270,8 @@ } } - if (update_dict) { - struct dict_transaction_context *dctx; - - dctx = dict_transaction_begin(euser->db); - dict_set(dctx, key, dec2str(new_stamp)); - if (dict_transaction_commit(&dctx) < 0) - i_error("expire: dict commit failed"); - } + if (update_dict) + expire_update(box, key, new_stamp); } T_END; i_free(xt); return 0; @@ -273,6 +348,7 @@ xpr_box = p_new(box->pool, struct expire_mailbox, 1); xpr_box->module_ctx.super = *v; box->vlast = &xpr_box->module_ctx.super; + xpr_box->expire_ext_id = (uint32_t)-1; v->transaction_begin = expire_mailbox_transaction_begin; v->transaction_commit = expire_mailbox_transaction_commit; @@ -356,6 +432,7 @@ euser->db = db; euser->set = expire_set_init(expire_get_patterns(user)); + euser->expire_cache = mail_user_plugin_getenv(user, "expire_cache") != NULL; MODULE_CONTEXT_SET(user, expire_mail_user_module, euser); } From dovecot at dovecot.org Wed Feb 25 13:10:18 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 25 Feb 2015 13:10:18 +0000 Subject: dovecot-2.2: lib-auth: auth_master_pass/user_lookup() now return... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/bcad7792a0fc changeset: 18274:bcad7792a0fc user: Timo Sirainen date: Wed Feb 25 15:05:22 2015 +0200 description: lib-auth: auth_master_pass/user_lookup() now returns -2 for user-specific errors. Compared to -1 which are about a more generic error with auth process communication. If -2 is returned the lookup could still succeed for another user. diffstat: src/doveadm/doveadm-auth.c | 1 + src/lib-auth/auth-master.c | 4 ++-- src/lib-auth/auth-master.h | 6 +++--- 3 files changed, 6 insertions(+), 5 deletions(-) diffs (48 lines): diff -r c2e46b4df488 -r bcad7792a0fc src/doveadm/doveadm-auth.c --- a/src/doveadm/doveadm-auth.c Tue Feb 24 18:42:43 2015 +0200 +++ b/src/doveadm/doveadm-auth.c Wed Feb 25 15:05:22 2015 +0200 @@ -67,6 +67,7 @@ i_error("%s failed for %s: %s", lookup_name, input->username, fields[0]); } + ret = -1; } else if (ret == 0) { fprintf(show_field == NULL ? stdout : stderr, "%s: user %s doesn't exist\n", lookup_name, diff -r c2e46b4df488 -r bcad7792a0fc src/lib-auth/auth-master.c --- a/src/lib-auth/auth-master.c Tue Feb 24 18:42:43 2015 +0200 +++ b/src/lib-auth/auth-master.c Wed Feb 25 15:05:22 2015 +0200 @@ -155,7 +155,7 @@ i_debug("user %s: Auth %s lookup returned temporary failure: %s", user, expected_reply, *args); } - return -1; + return -2; } i_error("Unknown reply: %s", cmd); return -1; @@ -511,7 +511,7 @@ p_new(pool, const char *, 1); if (ctx.return_value > 0) { i_error("Userdb lookup didn't return username"); - ctx.return_value = -1; + ctx.return_value = -2; } } else { *username_r = ctx.fields[0]; diff -r c2e46b4df488 -r bcad7792a0fc src/lib-auth/auth-master.h --- a/src/lib-auth/auth-master.h Tue Feb 24 18:42:43 2015 +0200 +++ b/src/lib-auth/auth-master.h Wed Feb 25 15:05:22 2015 +0200 @@ -31,9 +31,9 @@ /* Returns the auth_socket_path */ const char *auth_master_get_socket_path(struct auth_master_connection *conn); -/* Do a USER lookup. Returns -1 = error, 0 = user not found, 1 = ok. - When returning -1 and fields[0] isn't NULL, it contains an error message - that should be shown to user. */ +/* Do a USER lookup. Returns -2 = user-specific error, -1 = internal error, + 0 = user not found, 1 = ok. When returning -1 and fields[0] isn't NULL, it + contains an error message that should be shown to user. */ int auth_master_user_lookup(struct auth_master_connection *conn, const char *user, const struct auth_user_info *info, pool_t pool, const char **username_r, From dovecot at dovecot.org Wed Feb 25 13:10:19 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 25 Feb 2015 13:10:19 +0000 Subject: dovecot-2.2: doveadm: If command going through multiple users fa... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/c398321df8f5 changeset: 18275:c398321df8f5 user: Timo Sirainen date: Wed Feb 25 15:06:29 2015 +0200 description: doveadm: If command going through multiple users fails with user-specific error, don't stop. diffstat: src/doveadm/doveadm-mail-server.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diffs (12 lines): diff -r bcad7792a0fc -r c398321df8f5 src/doveadm/doveadm-mail-server.c --- a/src/doveadm/doveadm-mail-server.c Wed Feb 25 15:05:22 2015 +0200 +++ b/src/doveadm/doveadm-mail-server.c Wed Feb 25 15:06:29 2015 +0200 @@ -266,7 +266,7 @@ ret = doveadm_mail_server_user_get_host(ctx, input, &user, &host, error_r); if (ret < 0) - return -1; + return ret; if (ret == 0 && (ctx->set->doveadm_worker_count == 0 || doveadm_server)) { /* run it ourself */ From dovecot at dovecot.org Wed Feb 25 16:59:33 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Wed, 25 Feb 2015 16:59:33 +0000 Subject: dovecot-2.2: lib-index: Added an assert. Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/e7c65fa3ffa9 changeset: 18276:e7c65fa3ffa9 user: Timo Sirainen date: Wed Feb 25 18:58:50 2015 +0200 description: lib-index: Added an assert. It seems to be triggering later on, but not sure how it's happening. diffstat: src/lib-index/mail-transaction-log-file.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-) diffs (34 lines): diff -r c398321df8f5 -r e7c65fa3ffa9 src/lib-index/mail-transaction-log-file.c --- a/src/lib-index/mail-transaction-log-file.c Wed Feb 25 15:06:29 2015 +0200 +++ b/src/lib-index/mail-transaction-log-file.c Wed Feb 25 18:58:50 2015 +0200 @@ -674,7 +674,9 @@ const char *path2; buffer_t *writebuf; int fd, ret; - bool rename_existing; + bool rename_existing, need_lock; + + need_lock = file->log->head != NULL && file->log->head->locked; if (fcntl(new_fd, F_SETFL, O_APPEND) < 0) { log_file_set_syscall_error(file, "fcntl(O_APPEND)"); @@ -774,7 +776,7 @@ file->fd = new_fd; ret = mail_transaction_log_file_stat(file, FALSE); - if (file->log->head != NULL && file->log->head->locked) { + if (need_lock) { /* we'll need to preserve the lock */ if (mail_transaction_log_file_lock(file) < 0) ret = -1; @@ -816,7 +818,9 @@ /* success */ file->fd = new_fd; - mail_transaction_log_file_add_to_list(file); + mail_transaction_log_file_add_to_list(file); + + i_assert(!need_lock || file->locked); return 1; } From dovecot at dovecot.org Thu Feb 26 18:25:51 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Thu, 26 Feb 2015 18:25:51 +0000 Subject: dovecot-2.2: dsync: If we get disconnected from remoset server, ... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/6d788397444e changeset: 18277:6d788397444e user: Timo Sirainen date: Thu Feb 26 20:24:16 2015 +0200 description: dsync: If we get disconnected from remoset server, exit with EX_TEMPFAIL Instead of 1000, which gets truncated to 232. diffstat: src/doveadm/doveadm-dsync.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diffs (11 lines): diff -r e7c65fa3ffa9 -r 6d788397444e src/doveadm/doveadm-dsync.c --- a/src/doveadm/doveadm-dsync.c Wed Feb 25 18:58:50 2015 +0200 +++ b/src/doveadm/doveadm-dsync.c Thu Feb 26 20:24:16 2015 +0200 @@ -686,6 +686,7 @@ &ctx->output, &ctx->ssl_iostream); break; case SERVER_EXIT_CODE_DISCONNECTED: + ctx->ctx.exit_code = EX_TEMPFAIL; ctx->error = p_strdup_printf(ctx->ctx.pool, "Disconnected from remote: %s", error); break; From dovecot at dovecot.org Fri Feb 27 12:29:48 2015 From: dovecot at dovecot.org (dovecot at dovecot.org) Date: Fri, 27 Feb 2015 12:29:48 +0000 Subject: dovecot-2.2: fts-lucene: Fixed lookups from virtual mailboxes wi... Message-ID: details: http://hg.dovecot.org/dovecot-2.2/rev/7f67e5c86af9 changeset: 18278:7f67e5c86af9 user: Timo Sirainen date: Fri Feb 27 14:29:10 2015 +0200 description: fts-lucene: Fixed lookups from virtual mailboxes with over 32 physical mailboxes. diffstat: src/plugins/fts-lucene/fts-backend-lucene.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diffs (26 lines): diff -r 6d788397444e -r 7f67e5c86af9 src/plugins/fts-lucene/fts-backend-lucene.c --- a/src/plugins/fts-lucene/fts-backend-lucene.c Thu Feb 26 20:24:16 2015 +0200 +++ b/src/plugins/fts-lucene/fts-backend-lucene.c Fri Feb 27 14:29:10 2015 +0200 @@ -538,6 +538,12 @@ unsigned int i, j; p_array_init(&box_results, result->pool, 32); + /* first create the box_results - we'll be using pointers to them + later on and appending to the array changes the pointers */ + for (i = 0; boxes[i] != NULL; i++) { + box_result = array_append_space(&box_results); + box_result->box = boxes[i]; + } for (i = 0; boxes[i] != NULL; i++) { if (fts_mailbox_get_guid(boxes[i], &guid) < 0) return -1; @@ -547,8 +553,7 @@ for (j = 0; j < MAILBOX_GUID_HEX_LENGTH; j++) guid_dup[j] = guid[j]; - box_result = array_append_space(&box_results); - box_result->box = boxes[i]; + box_result = array_idx_modifiable(&box_results, i); hash_table_insert(guids, guid_dup, box_result); }