doveadm index crash/assert
I got the following crash whilst looking at my logs and seeing some indexer-worker crashes.
doveadm(ler): Panic: file http-client-request.c: line 1108 (http_client_request_send_more): assertion failed: (req->payload_input != NULL) Abort trap (core dumped)
bt full at: https://www.lerctr.org/~ler/doveadm.index.crash.txt
doveconf -n: https://www.lerctr.org/~ler/doveconf.n.txt
2.3.4 from FreeBSD ports.
What else do you need?
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Op 01/01/2019 om 02:12 schreef Larry Rosenman:
This may have something to do with the fact that you're using both Tika and Solr. Is this something you can reproduce? It would be helpful to have a debug log for the Tika and Solr HTTP exchange before the panic. This can be obtained by having mail_debug=yes.
Regards,
Stephan.
created with: doveadm -D -o mail_debug=yes index 2>&1 | tee https://www.lerctr.org/~ler/dovecot/doveadm.index.debug.out
bt full for the above: https://www.lerctr.org/~ler/dovecot/doveadm.index.crash.bt.txt
On Wed, Jan 2, 2019 at 2:11 AM Stephan Bosch <stephan@rename-it.nl> wrote:
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Oh, d'oh. I was looking for some solr debug logging in your output, but I didn't see much of it. Then I noticed that for some reason fts-solr has its own debug setting , while fts-tika uses mail_debug. Can you enable that as well?
Regards,
Stephan.
Op 02/01/2019 om 14:41 schreef Larry Rosenman:
https://www.lerctr.org/~ler/dovecot/doveadm-index-fts-debug.txt
https://www.lerctr.org/~ler/dovecot/doveadm-index-fts-bt.txt
I wish there was a way to set plugins {fts_solr = <blah>} from the command line :(
but I turned it on globally for that run.
On Wed, Jan 2, 2019 at 3:40 PM Stephan Bosch <stephan@rename-it.nl> wrote:
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
I honestly didn't try that :(
And I just did, and yes, that works. I didn't realize that was the case.
On Wed, Jan 2, 2019 at 3:56 PM Stephan Bosch <stephan@rename-it.nl> wrote:
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Op 02/01/2019 om 22:51 schreef Larry Rosenman:
Yeah, it is caused by the interaction between the solr and tika HTTP clients in this case. The bug itself will require some care and attention, but a quick fix should be provided by: diff --git a/src/plugins/fts-solr/solr-connection.c b/src/plugins/fts-solr/solr-connection.c index 947ea76af..3389729d1 100644 --- a/src/plugins/fts-solr/solr-connection.c +++ b/src/plugins/fts-solr/solr-connection.c @@ -156,7 +156,7 @@ int solr_connection_init(const char *url, http_set.request_timeout_msecs = 60*1000; http_set.ssl = ssl_client_set; http_set.debug = debug; - solr_http_client = http_client_init(&http_set); + solr_http_client = http_client_init_private(&http_set); } conn->xml_parser = XML_ParserCreate("UTF-8"); diff --git a/src/plugins/fts/fts-parser-tika.c b/src/plugins/fts/fts-parser-tika.c index a4b8b5c30..b8b57a350 100644 --- a/src/plugins/fts/fts-parser-tika.c +++ b/src/plugins/fts/fts-parser-tika.c @@ -77,7 +77,7 @@ tika_get_http_client_url(struct mail_user *user, struct http_url **http_url_r) http_set.request_timeout_msecs = 60*1000; http_set.ssl = &ssl_set; http_set.debug = user->mail_debug; - tika_http_client = http_client_init(&http_set); + tika_http_client = http_client_init_private(&http_set); } *http_url_r = tuser->http_url; return 0; This will bluntly prevent sharing any state between Solr and Tika HTTP clients, also across users (if there is more than one per mail process). Regards, Stephan.
Thanks, Stephan. I've added the workaround patch to the FreeBSD port. I went through the mailbox trees that I was seeing errors on, and no crashes with the patches applied. Let me know if there is anything I can do to help or provide access to. On Wed, Jan 2, 2019 at 4:05 PM Stephan Bosch <stephan@rename-it.nl> wrote:
-- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: larryrtx@gmail.com US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
participants (2)
-
Larry Rosenman
-
Stephan Bosch