Not able to combine java stack trace with the previous line as single event in elastic search

[harpritc]

Problem

I am using fluentd to send spring boot logs to elastic search, I am not able to combine java stack trace with the previous line as single event in elastic search,
I tried using concat plugin as well as detect_exception plugin
...

Steps to replicate

input:

2019-09-27 11:34:06.208 +0000 [http-nio-exec-9 ] [ERROR] c.l.u.e.LcValidationErrorHandler clientIP=xxxx- Unknown Exception
org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:278)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:2

fluentd conf:

<source>
@type tail
path /error.log
pos_file /error.pos
tag elastic.access
<parse>
@type grok
        <grok>
                pattern %{TIMESTAMP_ISO8601:timestamp} %{DATA:thread}\[%{LOGLEVEL:level}\] %{NOTSPACE:provider} sourceId=%{GREEDYDATA:sourceId} corrId=%{GREEDYDATA:corrId} clientIP=%{IPORHOST:clientIP} %{GREEDYDATA:msg}
                time_format %{TIMESTAMP_ISO8601}
        </grok>
        <grok>
                pattern %{GREEDYDATA:msg}
        </grok>
</parse>
@label @CONCAT
</source>
<label @CONCAT>
 <filter **>
  @type concat
  key msg
  multiline_start_regexp /Unknown Exception/
  continuous_line_regexp /^(\s+at|Caused by:|\s+...|org.|com.)/
  flush_interval 3s
  timeout_label @NORMAL
 </filter>
 <match **>
        @type aws-elasticsearch-service
        type_name "access_log"
        logstash_format true
        include_tag_key true
        tag_key "@log_name"
        flush_interval 1s
        <buffer>
                flush_thread_count 4
        </buffer>

        <endpoint>
                url https://xxx.es.amazonaws.com
                region us-west-2
        </endpoint>
 </match>
</label>

output

msg:- Unknown Exception org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:278)

Expected Behavior

msg:- Unknown Exception timestamp:8237-09-27 11:34:06.208 thread:+0000 [http-nio-exec-9 ] level:ERROR provider:c.l.u.e.LcValidationErrorHandler  clientIP:xxx @timestamp:September 30th 2019, 21:39:48.023 @log_name:access _id:WyOehW0B3OvYvndt8-Ek _type:access_log _index:logstash-2019.10.01
org.springframework.dao.DataIntegrityViolationException: could not execute statement; SQL [n/a]; constraint [null]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:278)

Your environment

• OS version
  CentOS Linux 7 (Core)
• paste result of fluentd --version or td-agent --version
  fluentd 1.7.1
• plugin version
  fluent-plugin-concat' version '2.4.0'
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock