Tips & Tricks

Output to Elasticsearch with Rsyslog in Logstash format

This post is just a reminder of how to ship logs directly from rsyslog to Elasticsearch so we have to go trough Logstash.

We got allot of help from the people on the rsyslog mailing list and for me the following is working pretty good so far.

Our config:

module(load="imfile")
module(load="mmjsonparse")
module(load="omelasticsearch")

template(name="logstash-index"
  type="list") {
    constant(value="logstash-")
    property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="4")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="6" position.to="7")
    constant(value=".")
    property(name="timereported" dateFormat="rfc3339" position.from="9" position.to="10")
}

input(type="imfile" File="/var/log/nginx/access.json"
      Tag="nginxulyaoth"
      PersistStateInterval="10000"
      StateFile="nginxulyaoth"
      Severity="info"
      MaxSubmitAtOnce="20000"
      Facility="user"
      Ruleset="nginxrule")

template(name="uly-nginx" type="list") {
  constant(value="{")
  constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
  constant(value="\",\"host\":\"")        property(name="hostname")
  constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
  constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
  constant(value="\",\"tag\":\"")         property(name="syslogtag" format="json")
  constant(value="\",\"message\":\"")     property(name="msg" format="json")
  constant(value="\"}")
}

ruleset(name="nginxrule") {
  action(type="mmjsonparse" name="jsonparse")
  action(type="omelasticsearch"
  server="loghost.ulyaoth.net"
  serverport="9200"
  template="uly-nginx"
  searchIndex="logstash-index"
  dynSearchIndex="on"
  errorFile="/var/log/rsyslog/ES-error.log")
}

We are still working on getting the json log split into variables so it will show more nice in Kibana.

Related posts

AWS Cli Commands to remember

Sjir Bagmeijer

Rack::OnePiece

Sjir Bagmeijer

How to fix the “Pane is dead” error during the installation of Fedora on a HP Proliant server

Sjir Bagmeijer

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More