The general steps to create and manage alerts are shown in the image below.
- Create Destinations
The first step is to create a Destination. Choose Alerting -> Destinations -> Add destination. Specify a name for the destination so that you can identify it later. For Type, choose Slack, Amazon Chime, or custom webhook. Take note that these requests use the HTTP POST method.
For this use case, we will show how to create a webhook in Slack which seems to be extremely useful and many teams are utilizing it.
First of all, we must create an account in Slack and even better download the app for iOS or Android. First, sign in to your account.
Next step is to search for the term “Incoming webhooks”.
The first result that will be shown is the Incoming Webhooks Add-on.
Press the button “Add to Slack” to add this functionality to your Slack and choose the channel you want the send the notification.
If you want, you can create a new channel that will be used only for notifications.
2. Create Monitors
The second step is to create a Monitor by choosing Alerting -> Monitors -> Create monitor.
Specify a name ( in “Monitor name”) and schedule (how often do you want to run) for the monitor.
Choose one or more indices. You can also use * as a wildcard to specify an index pattern.
Define the monitor in one of three ways: visually, using a query, or using an anomaly detector.
- Visual graph definition works well for monitors that you can define as “some value is above or below some threshold for some amount of time”.To define a monitor visually, choose Define using visual graph. Then choose an aggregation, a set of documents, and a timeframe. Visual definition is the most common choice.
- Query definition gives you flexibility in terms of what you query for (using the Elasticsearch query DSL) and how you evaluate the results of that query (Painless scripting). To use a query, choose Define using extraction query, add your query (using the Elasticsearch query DSL), and test it using the Run button.
- To use an anomaly detector, choose Define using Anomaly detector and select your Detector.
Finally, choose Create.
3. Create triggers
The third step is to create a trigger. These steps differ depending on whether you chose Define using visual graph, Define using extraction query or Define using Anomaly detector when you created the monitor.
Either way, you begin by specifying a name and severity level for the trigger which helps you to manage alerts. A trigger with a high severity level (e.g. 1) might inform a specific individual, whereas a trigger with a low severity level might message the whole team.
- Visual graph For Trigger condition, specify a threshold for the aggregation and timeframe you chose earlier, such as “is below 1,200” or “is exactly 1200.” The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.
- Extraction query For Trigger condition, specify a script that returns true or false. A return value of true means the trigger condition has been met, and the trigger should execute its actions. Test your script using the Run button.
- Anomaly detector For Trigger type, choose Anomaly detector grade and confidence. Specify the Anomaly grade condition for the aggregation and timeframe you chose earlier, “IS ABOVE 0.8” or “IS EXACTLY 0.6.” The anomaly grade is a number between 0 and 1 that indicates the level of severity of how anomalous a data point is. Specify the Anomaly confidence condition for the aggregation and timeframe you chose earlier, “IS ABOVE 0.8” or “IS EXACTLY 0.6.” The anomaly confidence is an estimate of the probability that the reported anomaly grade matches the expected anomaly grade. The line moves up and down as you increase and decrease the threshold. Once this line is crossed, the trigger evaluates to true.
4. Add Actions
The final step is to create an action. Actions send notifications when trigger conditions are met and support Slack, Amazon Chime, and webhooks. If you don’t want to receive notifications for alerts, you don’t have to add actions to your triggers. Instead, you can periodically check Kibana.
Specify a name for the action and choose a destination.
Add a subject and body for the message. You can add variables to your messages using Mustache templates. You can send a test message to verify that everything is according to your settings.
Finally, choose Create.
When you complete the above steps and the conditions for the alert are met, you will be notified with a message like this: