Tutorials

logstash-forwarder and grok examples

Just adding some logstash-forwarder examples so I have them as a reminder for the future and perhaps it will help someone else also.

You can test your groks at this website:
https://grokdebug.herokuapp.com/
This is very useful to see if your grok does actually work correctly.

I will provide one full example for a Nginx “access.log” and then afterwards I provide simply the configs for additional programs that I find interesting.
Example: Nginx access.log
Step 1: Create the patterns directory
$ sudo mkdir -p /opt/logstash/patterns

Step 2: Set the correct user and group on patterns directory
$ sudo chown logstash:logstash /opt/logstash/patterns

Step 3: Create a log_format for nginx
$ sudo vi /etc/nginx/nginx.conf
You add the following piece of code inside the “http” area this code below is default nginx so should be there already.

log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"';

Step 4: Add a filter inside your logstash config
$ sudo vi /etc/logstash/logstash.conf

Then add the following filter:
filter {
if [type] == "nginx-access" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
}

Step 5: Add the log file and field in logstash-forwarder config
$ sudo vi /opt/logstash-forwarder/conf/logstash-forwarder.conf

Then inside your “files” bracket add the following:

    {
      "paths": [
        "/var/log/nginx/kibana/access.log"
      ],

      "fields": { "type": "nginx-access" }
    },

Step 6: Create the nginx pattern for logstash
$ sudo vi /opt/logstash/patterns/nginx

And add the following:
NGINXACCESS %{IPORHOST:remote_addr} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{INT:status} %{INT:body_bytes_sent} %{QS:http_referer} %{QS:http_user_agent}

Save the file and give the correct permissions to the file:
$ sudo chown logstash:logstash /opt/logstash/patterns/nginx

If you did it correctly then it should look like this:

As you can see the log file is now fully split up and you can use the different fields for better graphing.

Related posts

How to set up suEXEC in Apache

Sjir Bagmeijer

How to add Instana to Xenforo 2.0

Sjir Bagmeijer

How to flash your Samsung Galaxy Series phone

Sjir Bagmeijer

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More