Analyze Nginx logs with ELK stack




 

NGINX is an open source, high performance HTTP server and reverse proxy.
Although NGINX is far behind Apache, from the features perspective, its low memory usage and high performance are enough to keep NGINX competitive.

A real common scenario is that NGINX logging system can generate high number of access/error logs (at /var/log/nginx by default) and therefore could be hard to read and fetch valuable information.

The best way to solve this problem and efficiently analyse your NGINX logs is with the help of The ELK stack (Elasticsearch – Logstash – Kibana).

NGINX Logging System

NGINX has 2 types of logs, access and error logs.

Default log file location:
Access logs: /var/log/nginx/access.log
Error logs: /var/log/nginx/error.log

The access logs store all the requests processed by apache.
The error logs store diagnostic information and any error encountered while processing requests.

How to ship your NGINX Logs to ELK stack

Requirements

  • Apache installed
  • Filebeat installed
  • ELK stack or an active account in Logstail.com – the secure and scalable Log Management platform, offering the ELK stack as a service.

 

  • Filebeat configuration

Add the following configuration to filebeat.yml

  •  Ship logs to Logstail.com

Make sure to replace USER_TOKEN with your token provided in your account

  • Ship logs directly to ELK stack

For this example we use the port 5000 for the logstash port.

 

Save filebeat.yml and restart Filebeat to apply the changes.

Filebeat is now sending your NGINX logs to your local ELK stack. For remote installation replace hosts with your logstash remote ip and port and make the appropriate configurations to your firewall to allow connections.

NGINX Log Parsing

NGINX logs are stored in plain text format, therefore we need to apply some Logstash filters in order to separate each log into specific fields before storing them into Elasticsearch.

Access logs parsing

 

Visualizing NGINX Logs

By visualising our NGINX logs to Kibana – the amazing visualisation tool, we can clearly analyze valuable information which would possibly take hours of manual reading the access log file.

Here are some example graphs that will help us fetch some essential information:

 

 

 

 

 

For more NGINX Visualisations and Dashboards start now your free trial with us.

Analyze Apache logs with ELK stack<< >>Monitor Mikrotik with Logstail.com hosted ELK

Leave a Reply

Your email address will not be published.