Elastic Stack
Overview
ELK: Elasticsearch, Logtash, Kibana
Elasticsearch: Distributed, fast, highly scalable document database
Logstash:
Aggregates, filters, and supplements log data
Forwards altered logs to Elasticsearch
Sending logs directly to Elasticsearch without Logstash can lead to inconsistent data
Kibana:
web-based front-end, written in Node.js
Works easily with Elasticsearch for charts, graphs, and visualizing data
Free from Elastic Company
Beats:
Small, lightweight utilities for reading logs from a variety of sources. Usually sends data to Logstash
Filebeat: Text Log files
Metricbeat: OS and applications
Packetbeat: Network Monitoring
Winlogbeat: Windows Event Log
Libbeat: Write your own
Alerting
Help track conditions based on Elasticsearch data
Continually monitors log data for pre-configured conditions
Send notification to email, Slack, Hipchat, and PagerDuty out of the box
Installation
Elastic Search
On Linux
Install java jvm: apt install openjdk-8-jre-headless
Check version: java -version
Download and instsall: https://www.elastic.co/downloads/elasticsearch
Windows
Install Java SDK
Create environment variable: JAVA_HOME=$path/to/java/bin
Download and extract
Change jvm.options under elasticsearch folder/config
Install
Docker
Install on Docker: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Logstash
Linux
Install java jvm: apt install openjdk-8-jre
Add Key and source list, then apt update && apt install logstash
/usr/share/logstash/bin/logstash -e 'input {stdin {}} output { elasticsearch {host => ["192.168.0.12:9200"]}}'
Verify: curl http://192.168.0.12:9200/logstash-*/_search
systemctl enable logstash
service logstash start
Kibana
Linux
Add source public key and source list
apt update && apt install kibana
Configuration
ElasticSearch
Configuration: /etc/elasticsearch/elasticsearch.yml
sysctl -w vm.max_map_count=262144: change max memory
service elasticsearch start
access: https://$ip:9200
Logstash
Input: where is data coming from? Logs? Beats?
Filter: How should we parse the data? Ignore some? Modify any?
Some Logstash filters: grok filter, geoip filter
Output: Where should we store the logs? Back end? Elasticsearch?
Kibana
/etc/kibana/kibana.yml
service kibana start
access: http://192.168.0.15:5601
Instrumenting Network Traffic
Use packetbeat
Installation
Windows
Install WinPCap
Download packetbeat and unzip
cd to folder
Configuration: packetbeat.yml
Install template file: IWR -method PUT -infile .\packetbeat.template.json -uri http://192.168.0.12:9200/_template/packetbeat
Configuration
Log into Kibana: http://192.168.0.15:5601
Add Index pattern: Management > Index Patterns > Add New: packetbeat-*
Discover
Alerting with Watcher
Commercial plugin, not free
Installation
Stop elasticsearch: service elasticsearch stop
Install plugin: ./elasticsearch-plugin install x-pack
Configure SMTP in elasticsearch: /etc/elasticsearch/elasticsearch.yml
Kibana Query Language - ELK Stack
Rserved Characters
+, -, =, &&, ||, &, |, !
To use reserved character in query, use \ before it such as +
Wildcards
*: any number of characters
?: single character
Range
response_time_seconds >=100 AND response_time_seconds < 300
incident_date < "2022-12-01": before 01/12/2022
Fuzzy Search
incident_comment:true~1: misspelling the word true
Proximity Searches
log_message:"server error"~1: search for message with server and error that has 1 word or less between them
log_message:"server error"~1 OR "login server"~1
Regular Expression
Event_Type:/(S|M).*/: only the entries that start with s and M
Reference
https://www.neteye-blog.com/2017/10/sending-cisco-syslogs-to-elasticsearch-a-simple-guide/
ELK in the cloud: https://github.com/strandjs/IntroLabs/blob/master/IntroClassFiles/navigation.md
Last updated