SENTINL
Proof-of-Concept / Beginners Tutorial
This tutorial will illustrate a working example of SENTINL for alerting
WARNING: This guide is a work-in-progress and should not be used as-is in production!
Requirements
- Elasticsearch 2.4.x
- Kibi or Kibana 4.5+
- shell + curl to execute commands
Setup
Before starting, download and install the latest dev version of the plugin:
git clone https://github.com/sirensolutions/sentinl
cd sentinl && npm install && npm run package
/opt/kibana/bin/kibana plugin --install kaae -u file://`pwd`/sentinl-latest.tar.gz
Dataset
To illustrate the logic and elements involved with sentinl we will generate some random data and insert it to Elasticsearch.
Our sample JSON object will report a UTC @timestamp
and mos
value per each interval:
{"mos":4,"@timestamp":"2016-07-17T15:56:02.890"}
The following BASH script will produce our entries for a realistic example:
#!/bin/bash
INDEX=`date +"%Y.%m.%d"`
SERVER="http://127.0.0.1:9200/mos-$INDEX/mos/"
echo "Press [CTRL+C] to stop.."
while :
do
header="Content-Type: application/json"
timestamp=`TZ=UTC date +"%Y-%m-%dT%T.%3N"`
mos=$(( ( RANDOM % 5 ) + 1 ))
mystring="{\"mos\":${mos},\"@timestamp\":\"${timestamp}\"}"
echo $mystring;
curl -sS -i -XPOST -H "$header" -d "$mystring" "$SERVER"
sleep 5
done
- Save the file as
elasticgen.sh
and execute it for a few minutes
Watcher rule
To illustrate the trigger logic, we will create an alert for an aggregation against the data we just created.
The basic sentinl example will use simple parameters: Run each 60 seconds Target the daily mos- index with query aggregation Trip condition when aggregations.avg.value < 3 * Email action with details
curl -XPUT http://127.0.0.1:9200/watcher/watch/mos -d'
{
"trigger": {
"schedule" : { "later" : "every 1 minute" }
},
"input" : {
"search" : {
"request" : {
"indices" : [ "<mos-{now/d}>", "<mos-{now/d-1d}>" ],
"body" : {
"query" : {
"filtered" : {
"query": {
"query_string": {
"query": "mos:*",
"analyze_wildcard": true
}
},
"filter" : { "range" : { "@timestamp" : { "from" : "now-5m" } } }
}
},
"aggs": {
"avg": {
"avg": {
"field": "mos"
}
}
}
}
}
}
},
"condition" : {
"script" : {
"script" : "payload.aggregations.avg.value < 3"
}
},
"transform" : {},
"actions" : {
"email_admin" : {
"throttle_period" : "15m",
"email" : {
"to" : "mos@qxip.net",
"from" : "sentinl@qxip.net",
"subject" : "Low MOS Detected: {{payload.aggregations.avg.value}} ",
"priority" : "high",
"body" : "Low MOS Detected:\n {{payload.aggregations.avg.value}} average with {{payload.aggregations.count.value}} measurements in 5 minutes"
}
}
}
}'
Extending Logic
The basic Watcher can be extended and improved following the same logic used with the stock _Watcher, for example by using transform
to insert detections back in ES. An interesting set of examples is available here
Alarm Triggering
SENTINL will automatically fetch and schedule jobs, executing the watcher queries according to the trigger.schedule
parameter, validating their results according to the provided condition.script
Check output
Assuming all data and scripts are correctly executed, you should start seeing output in your application logs.
Positive Match (Kibana/UI)
Positive Match (console)
Jul 17 17:55:00 es2pcap kibana[44702]: SENTINL Payload: { took: 21, Jul 17 17:55:00 es2pcap kibana[44702]: timed_out: false, Jul 17 17:55:00 es2pcap kibana[44702]: _shards: { total: 186, successful: 186, failed: 0 }, Jul 17 17:55:00 es2pcap kibana[44702]: aggregations: { avg: { value: 2.9069767441860463 } } } Jul 17 17:55:00 es2pcap kibana[44702]: SENTINL Condition: payload.aggregations.avg.value < 3 Jul 17 17:55:00 es2pcap kibana[44702]: Low MOS Detected: 2.9069767441860463 Low MOS Detected: Jul 17 17:55:00 es2pcap kibana[44702]: 2.9069767441860463 average with measurements in 5 minutes