This is not a scenario that any real organisation or enterprise would be likely to face, but I wanted to find a way to use a Raspberry Pi as a log forwarder for Azure Sentinel. This will help me collect logs from the home network, especially from my Ubiquiti EdgeRouter firewall.
Since neither of the usual suspects for syslog forwarding - Logstash and Log Analytics Gateway - support the Pi or any ARM based Linux, I needed to find something new for this case. Luckily there is a third option that we can use: Fluentd.
Fluentd is a modular data collector that has input, output and filtering plugins for many different use cases, so in many ways it is similar to Logstash. And like Logstash, Fluentd has syslog input and Log Analytics output plugins and thus is capable of acting as a log forwarder to Sentinel.
Fluentd is also very simple to install and configure on a Pi running Raspbian OS. So far I’ve had no performance issues or problems with it. Below is a short description of the install and configuration process.
As a bonus there is some info here on parsing Syslog data from the EdgeRouter, in case someone is interested in that as well.
Installing Fluentd
With the following commands we install Ruby, latest version of Fluentd and the Log Analytics output plugin:
$ sudo apt install ruby-full
$ sudo gem install fluentd -v "~> 0.13.0"
$ sudo fluent-gem install fluent-plugin-azure-loganalytics
Configuring Fluentd for Syslog input and Log Analytics output
You can configure Fluentd by creating a new file at /etc/fluentd.conf.
Below is my example configuration, containing a Syslog listener, filterer and parser for the EdgeRouter logs and a forwarder to push the messages to Log Analytics workspace and Sentinel.
As you can see, the message handling in Fluentd is based on tags and a top-down process.
# Syslog listener for EdgeRouter hardcoded Syslog port, udp 514:
<source>
@type syslog
port 514
bind 0.0.0.0
tag edgerouter
</source>
# This filter makes sure we only process EdgeRouter messages that contain network traffic:
<filter edgerouter.**>
@type grep
<regexp>
key message
pattern /PROTO=/
</regexp>
</filter>
# This filter parses the EdgeRouter log message data to Log Analytics columns:
<filter edgerouter.**>
@type parser
key_name message
<parse>
@type regexp
expression /.*?IN=(?<inIf>[^ ]*) .*?OUT=(?<outIf>[^ ]*) .*?SRC=(?<srcIP>[^ ]*) .*?DST=(?<dstIP>[^ ]*) .*?PROTO=(?<proto>[^ ]*) .*?SPT=(?<srcPort>[^ ]*) .*?DPT=(?<dstPort>[^ ]*)/
</parse>
</filter>
# This segment forwards the messages to two destinations:
# 1) stdout for debugging (can also be written to a file as shown later)
# 2) Log Analytics and Azure Sentinel (custom table named UbiEdgeRouter_CL)
<match edgerouter.**>
@type copy
<store>
@type stdout
</store>
<store>
@type azure-loganalytics
customer_id [Log Analytics Workspace ID]
shared_key [Log Analytics Workspace Key]
log_type UbiEdgeRouter
</store>
</match>
Getting the logs flowing
Starting Fluentd, with parameters to select our config file and log destination:
# /usr/local/bin/fluentd -c /etc/fluent.conf -o /var/log/fluent.log
You can follow the log file to make sure things are working as intended. The log destination config and startup options can be removed later when things run smoothly, so the logs will not fill up the Pi SD card.
As seen below, we now have the EdgeRouter logs forwarded to Sentinel:
