Analyze Apache logs with ELK stack




Apache is the most widely used open source web server. It can be used for all kinds of websites and is quite easy to setup. Apache comes with an integrated log system which stores access and error logs (at /var/log/apache2 by default).

In production environments where traffic is high, the log files are too large to read and fetch valuable information.

The ELK stack is the best monitoring and analytics tool that will help you solve this problem and analyse your logs with efficiency.

Apache Logging System

Apache has 2 types of logs, access and error logs.

Default log file location:

  • Access logs: /var/log/apache2/access.log
  • Error logs: /var/log/apache2/error.log

The access logs store all the requests processed by apache.

The error logs store diagnostic information and any error encountered while processing requests.

How to ship your Apache Logs to ELK stack

Requirements

  • Apache installed
  • Filebeat installed
  • ELK stack or an active account in Logstail.com – the secure and scalable Log Management platform, offering the ELK stack as a service.

 

  • Filebeat configuration

Add the following configuration to filebeat.yml

  •  Ship logs to Logstail.com

Make sure to replace USER_TOKEN with your token provided in your account

 

  • Ship logs directly to ELK stack

For this example we use the port 5000 for the logstash port.

Save filebeat.yml and restart Filebeat to apply the changes.

Filebeat is now sending your logs to your local ELK stack. For remote installation replace hosts with your logstash remote ip and port and make the appropriate configurations to your firewall to allow connections.

Apache Log Parsing

Apache logs are stored in plain text format, therefore we need to apply some Logstash filters in order to separate each log into specific fields before storing them into Elasticsearch.

Access logs parsing

Error logs parsing

Analyzing and Visualizing Apache Logs

By visualising our Apache logs to Kibana – the amazing visualisation tool, we can clearly analyze valuable information which would possibly take hours of manual reading the access log file.

Here are some example graphs which will help us fetch some essential information:

 

 

 

For more Apache Visualisations and Dashboards start now your free trial to logstail.com and enjoy your journey to hidden knowledge discovery.

How to Monitor HAProxy with ELK Stack<< >>Analyze Nginx logs with ELK stack

Leave a Reply

Your email address will not be published.