f

Nginx is an extremely popular open-source web server serving millions of applications around the world. Nginx’s owes its popularity as a web server (it can also serve as a reverse proxy, HTTP cache, and load balancer) to the way it efficiently serves static content and overall performance. From an operational and security perspective, Nginx sits at a critical juncture within an application’s architecture and requires close monitoring at all times. The ELK Stack (Elasticsearch, Logstash, Kibana and Beats) is the world’s most popular open-source log management and log analysis platform, and offers engineers with an extremely easy and effective way of monitoring Nginx. To complete the steps shown in this guide you will need an Nginx web server installed and active, and either of the following:

  1. ELK Stack installed and configured by you
  2. Or just a Logstail.com account!

The basics of Nginx logs

Nginx provides users with various lodging options, including logging to file, conditional logging and syslog logging. The Nginx web server provides two log types that can be used for monitoring your installation:

  1. access logs
  2. error logs

Both logs are located, by default, under /var/log/nginx

1) Access logs

Access logs contain information on all the requests being sent to and served by, Nginx. As such, they are a valuable resource to use for performance monitoring but also security. The default format for Nginx access logs is the combined format but this may change from distribution to distribution. As with error logs, you can use the access_log directive to set the log file path and log format. Example log:

2) Error logs

Error logs contain diagnostic information that can be used for troubleshooting operational issues. The Nginx error_log directive can be used to specify the log file path and severity and can be used in the main, http, mail, stream, server, location context (in that order). Example log:

How to ship your logs to ELK Stack (or Logstail.com)

The most user-friendly way to ship your Nginx logs into the ELK Stack (or Logstail.com) is with the use of Filebeat. Filebeat belongs to the family of Beats which are open source data shippers that you install as agents on your server to send operational data to Elasticsearch. In essence, there is no real need to add Logstash to handle the processing, which makes the set-up of the pipeline much simpler. Instead of ELK Stack you can choose Logstail.com to ship your logs directly without the need to use Filebeat Nginx module (this is even more easier!).

1) Installing Filebeat to the Nginx server

First, add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):

Next, add the repository definition to your system

Update and install Filebeat with:

2) Enabling the Nginx Filebeat module (not necessary in Logstail.com case)

Another option to ship your logs to ELK Stack is by enabling the Nginx Filebeat module (not needed in Logstail.com case). To do this, enter:

Next, use the following setup command to load a recommended index template and deploy sample dashboards for visualizing the data in Kibana:

And last but not least, start Filebeat with:

It’s time to verify that your pipeline working as expected. First, cURL the Elasticsearch to verify a “filebeat-*” index has indeed been created:

Next, open Kibana at http://localhost:5601— the index will be defined and loaded automatically and the data visible on the Discover page:

Steps to ship Nginx logs to Logstail.com

To make things even easier, Logstail.com gives you the opportunity to automatically parse the Nginx logs, without the need of Logstash or Filebeat’s Nginx module. In order to operationally use your logs, all you have to do is install Filebeat to Nginx Server and modify the Filebeat.yml configuration file on your Logstail.com:

1) Download the SSL certificate

Firstly, to secure ship your logs to Logstail.com (encrypted), you have to download our public SSL certificate:

2) Editing Filebeat

Open the Filebeat.yml configuration file with your favorite editor (vim or nano):

Paste the following configuration (for Debian, taken from shippers page):

Save the file and restart Filebeat with:

Finally, check that Nginx data is received from your Filebeat.

How to analyze Nginx logs

Now you can query your logs with the help of Kibana. Kibana gives you many query options and features like auto-suggest and auto-complete makes searching much easier. For example, you can search with free text. Just enter your search query in the search field as follows (search word: athens):

 

 

Another option is to conduct field-level searches if you want to be more specific. For example, you can search for any Nginx error log for the last two (2) days using this search query:  

The query options are actually extremely varied depending on your actual needs, which can be analytics, troubleshooting, security and many more.

 

Start Free

How to visualize them

With Kibana you can instantly visualize your data with dashboards in many different ways. The most frequent uses cases of visualizing Nginx logs using Kibana are:

1) Request map

For Nginx access logs and any other type of logs recording traffic, the usual place to start is a map of the different locations submitting requests. This helps you monitor regular behavior and identify suspicious traffic. Logstail.com automatically geo enriches the IP fields within the Nginx access logs so you can use a Coordinate Map visualization to map the requests as shown below:  

2) Responses code over time

Another common visualization used for Nginx access logs monitors response codes over time. Again, this gives you a good picture of normal behavior and can help you detect a sudden spike in error response codes. You can use Bar Chart, Line Chart or Area Chart visualizations for this:

 

3) Data Volume

Data volume visualizations are a great way to be aware of the timeframe between your server is exchanging data with the clients:

 

4) Errors over time

The Nginx server error logs give us a simple indication of the number of errors reported:

 

Apps2Go

Based on your needs you can customize the visualizations in Kibana, and these were just some simple examples of the tool’s capabilities. The creation of a dashboard is the finishing touch after the visualizations are ready. With a comprehensive dashboard, we have an operational overview of the Nginx web server. To make things easier we provide Apps2Go which is a library of ready dashboards for Nginx.

Alerting

Alerting is an extremely useful feature provided by Logstail.com. We provide you a mechanism to receive when certain indicators exceed the thresholds been defined. Now you have the ability to immediately realize performance-related or other issues and take the appropriate measures to mitigate the problem. This functionality is a must when you want to have real-time operational awareness of your systems. Email and Slack are currently supported. You can find a thorough article about how our alerting mechanism works, here.

 

Conclusion

Logstail.com with its advanced features brings the functionality of ELK Stack to your hands. You don’t have to be an engineer in order to set up and use Elasticsearch anymore. Now you can convert your data into actionable insights with just some tweaks. You can maximize the performance of your infrastructure or be notified of potential problems and take the appropriate actions. Sign-up for a free demo in order to realize yourself the power of Logstail.com.

 

Start Free

0 0 vote
Article Rating