Tutorials

logstash-forwarder and grok examples

Just adding some logstash-forwarder examples so I have them as a reminder for the future and perhaps it will help someone else also.

You can test your groks at this website:
https://grokdebug.herokuapp.com/
This is very useful to see if your grok does actually work correctly.

I will provide one full example for a Nginx “access.log” and then afterwards I provide simply the configs for additional programs that I find interesting.
Example: Nginx access.log
Step 1: Create the patterns directory
$ sudo mkdir -p /opt/logstash/patterns

Step 2: Set the correct user and group on patterns directory
$ sudo chown logstash:logstash /opt/logstash/patterns

Step 3: Create a log_format for nginx
$ sudo vi /etc/nginx/nginx.conf
You add the following piece of code inside the “http” area this code below is default nginx so should be there already.

log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';

Step 4: Add a filter inside your logstash config
$ sudo vi /etc/logstash/logstash.conf

Then add the following filter:
filter {
if [type] == "nginx-access" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
}

Step 5: Add the log file and field in logstash-forwarder config
$ sudo vi /opt/logstash-forwarder/conf/logstash-forwarder.conf

Then inside your “files” bracket add the following:

    {
      "paths": [
        "/var/log/nginx/kibana/access.log"
      ],

      "fields": { "type": "nginx-access" }
    },

Step 6: Create the nginx pattern for logstash
$ sudo vi /opt/logstash/patterns/nginx

And add the following:
NGINXACCESS %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

Save the file and give the correct permissions to the file:
$ sudo chown logstash:logstash /opt/logstash/patterns/nginx

If you did it correctly then it should look like this:

As you can see the log file is now fully split up and you can use the different fields for better graphing.

Related posts

How to install Tomcat 8 on Windows Server 2012 R2.

Sjir Bagmeijer

Install WordPress on Windows Server 2012 R2 with MariaDB 10.0 in replication.

Sjir Bagmeijer

VirtualBox USB fix for non-root users in Linux

Sjir Bagmeijer

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More