Skip to content
This repository has been archived by the owner on Dec 18, 2019. It is now read-only.

logstash-forwarder stop sending logs after a logstash restart #4

Open
sbarrier opened this issue Aug 20, 2015 · 0 comments
Open

logstash-forwarder stop sending logs after a logstash restart #4

sbarrier opened this issue Aug 20, 2015 · 0 comments

Comments

@sbarrier
Copy link

I noticed after a logstash restart using logstash-1.5.3 on linux, logstash is temporary refusing new connection which is to avoid oom in logstash, but on logstash-forwarder side it didn't notice it and stop sending new data, a kill -HUP generate a "watcher saw error: read: interrupted system call" and restart of logstash-forwarder is needed to recover.

  • logstash logs :

{:timestamp=>"2015-08-20T02:51:19.072000-0700", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-08-20T02:51:19.075000-0700", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-08-20T02:51:19.078000-0700", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-08-20T02:51:19.411000-0700", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
...

  • logstash-forwarder logs :

2015/08/20 02:50:51.123661 registrar received 512 events. /var/log/tomcat/localhost_access_log.txt: 497, /var/log/xxx-app/xxx.log: 15
2015/08/20 02:59:02.638187 watcher saw error: read: interrupted system call
2015/08/20 03:05:20.449806 lumberjack starting
2015/08/20 03:05:20.451258 resume tracking /var/log/tomcatlocalhost_access_log.txt
2015/08/20 03:05:20.451308 resume tracking /var/log/xxx/xxx.log
2015/08/20 03:05:20.451375 TLS config: &{ [] map[] 0xc20803ced0 [] 0 true [] false false [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 0 0 [] {{0 0} 0}}
...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant