Nginx is an extremely popular open-source web server serving millions of applications around the world. Nginx’s owes its popularity as a web server (it can also serve as a reverse proxy, HTTP cache, and load balancer) to the way it efficiently serves static content and overall performance. From an operational and security perspective, Nginx sits at a critical juncture within an application’s architecture and requires close monitoring at all times. The ELK Stack (Elasticsearch, Logstash, Kibana and Beats) is the world’s most popular open-source log management and log analysis platform, and offers engineers with an extremely easy and effective way of monitoring Nginx. To complete the steps shown in this guide you will need an Nginx web server installed and active, and either of the following:
- ELK Stack installed and configured by you
- Or just a Logstail.com account!
The basics of Nginx logs
Nginx provides users with various lodging options, including logging to file, conditional logging and syslog logging. The Nginx web server provides two log types that can be used for monitoring your installation:
- access logs
- error logs
Both logs are located, by default, under /var/log/nginx
1) Access logs
Access logs contain information on all the requests being sent to and served by, Nginx. As such, they are a valuable resource to use for performance monitoring but also security. The default format for Nginx access logs is the combined format but this may change from distribution to distribution. As with error logs, you can use the access_log directive to set the log file path and log format. Example log:
|
1 |
199.203.204.57 - - [19/Jan/2020:06:35:54 +0000] "GET /hello.html HTTP/1.1" 200 63 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36" |
2) Error logs
Error logs contain diagnostic information that can be used for troubleshooting operational issues. The Nginx error_log directive can be used to specify the log file path and severity and can be used in the main, http, mail, stream, server, location context (in that order). Example log:
|
1 |
2020/01/19 06:41:46 [emerg ] 12233#12233: directive “http” has no opening “{” in /etc/nginx/nginx.conf:17 |
How to ship your logs to ELK Stack (or Logstail.com)
The most user-friendly way to ship your Nginx logs into the ELK Stack (or Logstail.com) is with the use of Filebeat. Filebeat belongs to the family of Beats which are open source data shippers that you install as agents on your server to send operational data to Elasticsearch. In essence, there is no real need to add Logstash to handle the processing, which makes the set-up of the pipeline much simpler. Instead of ELK Stack you can choose Logstail.com to ship your logs directly without the need to use Filebeat Nginx module (this is even more easier!).
1) Installing Filebeat to the Nginx server
First, add Elastic’s signing key so that the downloaded package can be verified (skip this step if you’ve already installed packages from Elastic):
|
1 |
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - |
Next, add the repository definition to your system
|
1 |
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list |
Update and install Filebeat with:
|
1 |
sudo apt-get update && sudo apt-get install filebeat |
2) Enabling the Nginx Filebeat module (not necessary in Logstail.com case)
Another option to ship your logs to ELK Stack is by enabling the Nginx Filebeat module (not needed in Logstail.com case). To do this, enter:
|
1 |
sudo filebeat modules enable nginx |
Next, use the following setup command to load a recommended index template and deploy sample dashboards for visualizing the data in Kibana:
|
1 |
sudo filebeat setup -e |
And last but not least, start Filebeat with:
|
1 |
sudo service filebeat start |
It’s time to verify that your pipeline working as expected. First, cURL the Elasticsearch to verify a “filebeat-*” index has indeed been created:
|
1 |
curl -X GET "localhost:9200/_cat/indices?v" |
Next, open Kibana at http://localhost:5601— the index will be defined and loaded automatically and the data visible on the Discover page:
Steps to ship Nginx logs to Logstail.com
To make things even easier, Logstail.com gives you the opportunity to automatically parse the Nginx logs, without the need of Logstash or Filebeat’s Nginx module. In order to operationally use your logs, all you have to do is install Filebeat to Nginx Server and modify the Filebeat.yml configuration file on your Logstail.com:
1) Download the SSL certificate
Firstly, to secure ship your logs to Logstail.com (encrypted), you have to download our public SSL certificate:
|
1 |
demo@debian:~$ sudo wget https://raw.githubusercontent.com/logstail/public-certs/master/SectigoRSADomainValidationSecureServerCA.crt -P /etc/certs/logstail/ |
2) Editing Filebeat
Open the Filebeat.yml configuration file with your favorite editor (vim or nano):
|
1 |
sudo vim /etc/filebeat/filebeat.yml |
Paste the following configuration (for Debian, taken from shippers page):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
filebeat.inputs: # Filebeat configuration for Nginx (access and error) - type: log paths: - /var/log/nginx/access.log* exclude_files: [".gz$"] fields: codec: plain token: USER_TOKEN type: nginx_access fields_under_root: true - type: log paths: - /var/log/nginx/error.log* exclude_files: [".gz$"] fields: codec: plain token: USER_TOKEN type: nginx_error fields_under_root: true # end of Nginx configuration #************************************************************************ #Add here configuration for other services (if any) like Apache etc. # ************************************************************************ output.logstash: hosts: ["apps.logstail.com:35723"] ssl.certificate_authorities: ["/etc/certs/logstail/SectigoRSADomainValidationSecureServerCA.crt"] |
Save the file and restart Filebeat with:
|
1 |
sudo service filebeat restart |
Finally, check that Nginx data is received from your Filebeat.
How to analyze Nginx logs
Now you can query your logs with the help of Kibana. Kibana gives you many query options and features like auto-suggest and auto-complete makes searching much easier. For example, you can search with free text. Just enter your search query in the search field as follows (search word: athens):
Another option is to conduct field-level searches if you want to be more specific. For example, you can search for any Nginx error log for the last two (2) days using this search query:
The query options are actually extremely varied depending on your actual needs, which can be analytics, troubleshooting, security and many more.
How to visualize them
With Kibana you can instantly visualize your data with dashboards in many different ways. The most frequent uses cases of visualizing Nginx logs using Kibana are:
1) Request map
For Nginx access logs and any other type of logs recording traffic, the usual place to start is a map of the different locations submitting requests. This helps you monitor regular behavior and identify suspicious traffic. Logstail.com automatically geo enriches the IP fields within the Nginx access logs so you can use a Coordinate Map visualization to map the requests as shown below:
2) Responses code over time
Another common visualization used for Nginx access logs monitors response codes over time. Again, this gives you a good picture of normal behavior and can help you detect a sudden spike in error response codes. You can use Bar Chart, Line Chart or Area Chart visualizations for this:
3) Data Volume
Data volume visualizations are a great way to be aware of the timeframe between your server is exchanging data with the clients:
4) Errors over time
The Nginx server error logs give us a simple indication of the number of errors reported:
Apps
Based on your needs you can customize the visualizations in Kibana, and these were just some simple examples of the tool’s capabilities. The creation of a dashboard is the finishing touch after the visualizations are ready. With a comprehensive dashboard, we have an operational overview of the Nginx web server. To make things easier we provide Apps which is a library of ready dashboards for every service we offer!
Alerting
Alerting is an extremely useful feature provided by Logstail. We provide you a mechanism to receive when certain indicators exceed the thresholds been defined. Now you have the ability to immediately realize performance-related or other issues and take the appropriate measures to mitigate the problem. This functionality is a must when you want to have real-time operational awareness of your systems. Email and Slack are currently supported. You can find a thorough article about how our alerting mechanism works, here.
Conclusion
Logstail with its advanced features brings the functionality of ELK Stack to your hands. You don’t have to be an engineer in order to set up and use Elasticsearch anymore. Now you can convert your data into actionable insights with just some tweaks. You can maximize the performance of your infrastructure or be notified of potential problems and take the appropriate actions. Sign-up for a free demo in order to realize yourself the power of Logstail.
Contact Our Expertsor Sign Up for Free
