Alerting is an extremely useful feature provided by Logstail. We provide you a mechanism to receive alerts when certain indicators exceed the thresholds been defined. But what is alerting and how useful is for your daily operations? In this article, we will explain the basic terms our platform is using and how you can make the most out of it!
An important part of any log management system is alerting. Rather than searching log data for specific events, our platform offers the alerting functionality. Logstail has the ability to save your searches on a schedule that is determined bu you and send messages or emails when alerts are being triggered.
Now you have the ability to be immediately notified for performance or other issues and take the appropriate measures to mitigate the problem. This functionality is a must when you want to have real-time operational awareness of your systems.
In this article, we will demonstrate how the alerting mechanism is functioning, but before doing we will explain the basic terms, which are:
- Destinations
- Monitors
- Triggers
- Actions
- Alerts
- Dashboards
Basic Terms Explanation
1) Destinations
Destinations are the reusable locations for an action, such as Slack, Amazon Chime, or a webhook URL.
2) Monitors
Monitors are jobs running on a defined schedule and query Logstail.com. The results of these queries are then used as input for one or more triggers.
3) Triggers
Triggers are conditions that, when met, generate alerts and can perform some action.
4) Actions
Actions are the piece of information that you want the monitor to send out after being triggered. They have a destination, a message subject, and a message body.
5) Alerts
Alert is a notification that a monitor’s trigger condition has been met.
6) Dashboards
The dashboard gives you an overall picture of your alerting state.
How to create alerts
The general steps to create and manage alerts are shown in the image below
1) Create Destinations
The first step is to create a Destination. Choose Alerting -> Destinations -> Add destination. Specify a name for the destination so that you can identify it later. For Type, choose Slack, Amazon Chime, or custom webhook. Take note that these requests use the HTTP POST method.
For this use case, we will show how to create a webhook in Slack which seems to be extremely useful and many teams are utilizing it.
First of all, we must create an account in Slack and even better download the app for iOS or Android. First, sign in to your account.
Next step is to search for the term “Incoming webhooks”
The first result that will be shown is the Incoming Webhooks Add-on
Press the button “Add to Slack” to add this functionality to your Slack and choose the channel you want the send the notification.
If you want, you can create a new channel that will be used only for notifications.
2) Create Monitors
The second step is to create a Monitor by choosing Alerting -> Monitors -> Create monitor.
Specify a name ( in “Monitor name”) and schedule (how often do you want to run) for the monitor.
Choose one or more indices. You can also use * as a wildcard to specify an index pattern.
Define the monitor in one of three ways: visually, using a query, or using an anomaly detector.
- Visual graph definition works well for monitors that you can define as “some value is above or below some threshold for some amount of time”.To define a monitor visually, choose Define using visual graph. Then choose an aggregation, a set of documents, and a timeframe. Visual definition is the most common choice.
- Query definition gives you flexibility in terms of what you query for (using the Elasticsearch query DSL) and how you evaluate the results of that query (Painless scripting). To use a query, choose Define using extraction query, add your query (using the Elasticsearch query DSL), and test it using the Run button.
- To use an anomaly detector, choose Define using Anomaly detector and select your Detector.
Finally, choose Create.
3) Create Triggers
The third step is to create a trigger. These steps differ depending on whether you chose Define using visual graph, Define using extraction query or Define using Anomaly detector when you created the monitor.
Either way, you begin by specifying a name and severity level for the trigger which helps you to manage alerts. A trigger with a high severity level (e.g. 1) might inform a specific individual, whereas a trigger with a low severity level might message the whole team.
- Visual graph For Trigger condition, specify a threshold for the aggregation and timeframe you chose earlier, such as “is below 1,200” or “is exactly 1200.” The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.
- Extraction query For Trigger condition, specify a script that returns true or false. A return value of true means the trigger condition has been met, and the trigger should execute its actions. Test your script using the Run button.
- Anomaly detector For Trigger type, choose Anomaly detector grade and confidence. Specify the Anomaly grade condition for the aggregation and timeframe you chose earlier, “IS ABOVE 0.8” or “IS EXACTLY 0.6.” The anomaly grade is a number between 0 and 1 that indicates the level of severity of how anomalous a data point is. Specify the Anomaly confidence condition for the aggregation and timeframe you chose earlier, “IS ABOVE 0.8” or “IS EXACTLY 0.6.” The anomaly confidence is an estimate of the probability that the reported anomaly grade matches the expected anomaly grade. The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.
4) Add Actions
The final step is to create an action. Actions send notifications when trigger conditions are met and support Slack, Amazon Chime, and webhooks. If you don’t want to receive notifications for alerts, you don’t have to add actions to your triggers. Instead, you can periodically check Kibana.
Specify a name for the action and choose a destination.
Add a subject and body for the message. You can add variables to your messages using Mustache templates. You can send a test message to verify that everything is according to your settings.
Finally, choose Create.
When you complete the above steps and the conditions for the alert are met, you will be notified with a message like this:
PS. We recently added the mail option to send your alerts. Simply when you create a destination, choose the Email option.
Conclusion
Logstail with its advanced features brings the functionality of ELK Stack to your hands. When we combine this functionality with a robust alerting mechanism, we can both maximize the performance of our infrastructure and be notified of potential problems in order to timely take the appropriate actions. Sign-up for a free demo in order to realize the power of Logstail.
Contact Our Expertsor Sign Up for Free
