Fluentd vs Logstash

Logstash is successful enough that Elasticsearch, Logstash, and Kibana are known as the ELK stack. At Panda Strike, we use the ELK stack and have several Elasticsearch clusters. But we take the L out, and use Fluentd instead, even though “EFK stack” sounds more awkward.

Here’s why.

First, both Fluentd and Logstash provide both log forwarders and log shippers. According to Wikipedia, log shipping “is the process of automating the backup of a database and transaction log files on a primary (production) database server, and then restoring them onto a standby server,” while a log forwarder “is a forestry vehicle that carries big felled logs from the stump to a roadside landing.” This is why it’s perfectly reasonable to run a serious, production application using nothing but Wikipedia definitions.

Nonetheless, we’ve chosen to use alternate definitions. Log shippers are an essential component in modern devops, because logs are streams, not files. Log forwarders send logging events to log shippers. A forwarder’s goal is to send those events upstream as quickly as possible. Log shippers make delivery/routing decisions based upon the log event stream. A shipper may aggregate events, and/or send them to remote storage or analysis tools.

These are both functionalities you can get from either Fluentd or Logstash. Some more similarities between Fluentd and Logstash:

Fluentd’s written mostly in Ruby, with performance-sensitive parts written in C, and with a more convenient, pre-compiled stable version available. Fluentd also has a forwarder written in Go. Logstash’s forwarder is in Go, while its shipper runs on JRuby, which requires the JVM. We felt this was serious overkill for log shipping.

They also differ in these aspects:

For us, Fluentd wins hands-down in this last category. Here’s an elided and abridged set of sample output:

$ ps fauxwww --sort -rss |awk '/(USER|logstash-forwarder)/ { print }'
 0.0  0.7

ps fauxwww --sort -rss |awk '/(USER|logstash)/ { print }'
 4.8 20.9

ps fauxwww --sort -rss |awk '$1 ~ /(USER|td-agent)/ { print }'
 0.0  2.3
 0.1  4.5

ps fauxwww --sort -rss |awk '$1 ~ /(USER|td-agent)/ { print }'
 0.0  2.3
 0.2  7.3

These numbers represent our experience in production and are not an attempt at creating perfect, objective benchmarks. And Logstash’s forwarder outdid Fluentd’s forwarder here. But Fluentd’s shipper did much, much better than Logstash’s log shipper. And, after a disappointing experience with LogstashCrashes were sufficiently frequent and unpredictable that we had to create a watchdog process., Fluentd was remarkably reliable and easy to get running.