Fluentd vs Logstash
Logstash is successful enough that Elasticsearch, Logstash, and Kibana are known as the ELK stack. At Panda Strike, we use the ELK stack and have several Elasticsearch clusters. But we take the L out, and use Fluentd instead, even though “EFK stack” sounds more awkward.
Here’s why.
First, both Fluentd and Logstash provide both log forwarders and log shippers. According to Wikipedia, log shipping “is the process of automating the backup of a database and transaction log files on a primary (production) database server, and then restoring them onto a standby server,” while a log forwarder “is a forestry vehicle that carries big felled logs from the stump to a roadside landing.” This is why it’s perfectly reasonable to run a serious, production application using nothing but Wikipedia definitions.
Nonetheless, we’ve chosen to use alternate definitions. Log shippers are an essential component in modern devops, because logs are streams, not files. Log forwarders send logging events to log shippers. A forwarder’s goal is to send those events upstream as quickly as possible. Log shippers make delivery/routing decisions based upon the log event stream. A shipper may aggregate events, and/or send them to remote storage or analysis tools.
These are both functionalities you can get from either Fluentd or Logstash. Some more similarities between Fluentd and Logstash:
- Both can be configured to function in a multi-tier setup
- In either case, forwarders can detect failure of a shipper and switch to another active shipper
- Both deliver log events to many different receivers, for instance readily supporting SQS, Elasticsearch, and S3
- Both handle JSON log events natively
- Both are under active development, with corporate sponsorship
- Both use a plugin architecture and have their own plugin manager tools
- Both support log event filtering, transformation, and have a similar breadth of plugins available
Fluentd’s written mostly in Ruby, with performance-sensitive parts written in C, and with a more convenient, pre-compiled stable version available. Fluentd also has a forwarder written in Go. Logstash’s forwarder is in Go, while its shipper runs on JRuby, which requires the JVM. We felt this was serious overkill for log shipping.
They also differ in these aspects:
- reliability, durability, and failure recovery
- tunability, and the ability to flush intervals, workers, and threads
- memory and CPU requirements
For us, Fluentd wins hands-down in this last category. Here’s an elided and abridged set of sample output:
# LOGSTASH FORWARDER
$ ps fauxwww --sort -rss |awk '/(USER|logstash-forwarder)/ { print }'
%CPU %MEM
0.0 0.7
# LOGSTASH SHIPPER
ps fauxwww --sort -rss |awk '/(USER|logstash)/ { print }'
%CPU %MEM
4.8 20.9
# FLUENTD FORWARDER
ps fauxwww --sort -rss |awk '$1 ~ /(USER|td-agent)/ { print }'
%CPU %MEM
0.0 2.3
0.1 4.5
# FLUENTD SHIPPER
ps fauxwww --sort -rss |awk '$1 ~ /(USER|td-agent)/ { print }'
%CPU %MEM
0.0 2.3
0.2 7.3
These numbers represent our experience in production and are not an attempt at creating perfect, objective benchmarks. And Logstash’s forwarder outdid Fluentd’s forwarder here. But Fluentd’s shipper did much, much better than Logstash’s log shipper. And, after a disappointing experience with LogstashCrashes were sufficiently frequent and unpredictable that we had to create a watchdog process., Fluentd was remarkably reliable and easy to get running.