The path to a great syslog! My first full-fledged syslog deployment should be interesting, and an educational read for anyone who’s on a similar path.
**sips coffee**
Ok, off to the races!
Identify what your business needs are.
- Why do you need a syslog server?
- What’s your budget?
- Can your existing network meet the demand of a syslog?
- Number of devices logging and messages per second
- What is your desired retention period?
- Understand the topology of your network and where you may need forwarders setup.
- What are you looking for as sources? See my SOURCES list for an example.
Top Syslog options (in my preferred order):
- Splunk (paid – free up to 500mb per day)
- Graylog (opensource)
- Rsyslog (opensource)
- ELK (Elasticsearch, Logstash, Kibana) (opensource)
- Logz.io (paid)
- Elsa (opensource)1)https://github.com/mcholste/elsa
- SolarWinds LEM (paid)
- Kiwi (paid)
- AlertLogic (paid)
Out of all the options, I ended up choosing Graylog. The deciding factor for me was cost. Graylog is a capable solution, especially Graylog2. From my research there were quite a few issues with the server falling over after 2k+ messages per second. However that was addressed with Graylog2.x. I’m anticipating throwing a RSYSLOG server into the mix for long-term archival of logs while the Graylog server will handle recent events (<3 months).
Once you’ve decided which stack you’re moving forward with install it! The remainder of this document will be encompassing ELK/Graylog and my experiences with it. During demoing the many products, Splunk was my favorite. The best thing about it was that it had pre-built apps that would organize my data into something meaningful. With Graylog, or other opensource solutions, you’ll have to manually create those visuals and streams.
When I first installed Graylog OVA for VMware it didn’t go well. My experience was a very negative one, nothing seemed to be functioning properly. The largest issue was that elasticsearch wasn’t able to start. There was also a number of libraries missing on the distro. I took the night off and attempted again a few days later with a freshly downloaded OVA. Spun it up and everything worked per the getting started guide.2)http://docs.graylog.org/en/2.1/pages/getting_started/web_console.html
Note: if you change IP (I went from DHCP to static) you need to run “sudo graylog-ctl reconfigure”
Once I wrapped up the getting started everything ‘just worked’. I created a few inputs and pointed my equipment accordingly. Data was showing up! Success! I ended up creating a few dashboards, mostly just to make sure everything was working.
Tips!
- Make sure you change your default linux user password and web user admin password!3)http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html?highlight=admin%20password
- Follow the hardening guides! Hardedning4)http://docs.graylog.org/en/2.1/pages/installation/virtual_machine_appliances.html#vmware-tools and Securing5)http://docs.graylog.org/en/2.1/pages/securing.html.
- Back your machine up! MongoDB/Elastic backup6)http://docs.graylog.org/en/2.1/pages/configuration/backup.html
As I find more time I’ll continue to add more to this article outlining additional tips.
Syslog sources:
- Windows event logs
- Firewalls
- WAPs
- Switches
- Antivirus
- VMWARE
- Cameras and DVR
- Linux Servers
- Software
- Exchange
- ADSU
- DNS
- DHCP
- Backup
- Citrix
OVA storage tips and advice:
The default OVA installation of Graylog has a caveat, IDE storage. I installed the default 20GB storage for my graylog testing and found that it quickly filled up (within two weeks of logging 5 devices). I attempted to extend the disk, but with vmware, you cannot extend IDE disks, only SCSI. I ended up having to add an additional disk (SCSI so I can extend) to the VM. To extend your disk after it is low on space run the command:
|
1 2 3 4 5 |
ubuntu@graylog:~$ sudo resize2fs /dev/sdb resize2fs 1.42.9 (4-Feb-2014) Filesystem at /dev/sdb is mounted on /elasticsearch_data; on-line resizing required old_desc_blocks = 2, new_desc_blocks = 7 The filesystem on /dev/sdb is now 26214400 blocks long. |
This added an additional 80GB to my 20GB elasticsearch_data partition /dev/sdb/. If you find your state is red and you have unassigned shards you may want to follow a great guide7)https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html to resolve.
|
1 2 3 4 5 |
Check status: curl -XGET http://localhost:9200/_cluster/health?pretty Get shard assignment: curl -XGET http://localhost:9200/_cat/shards |
When running the script that I linked to T37 above, I ran into issues. I had to enclose the $shard in ”. I also didn’t realize my node name was ‘Bling’ which I believe is a default. Below is the script I ran:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "graylog_0", "shard" : '$shard', "node" : "Bling", "allow_primary" : true } } ] }' sleep 5 done |
Example procedure for an OVA appliance on VMWare:
| Action | Explanation |
|---|---|
| shutdown the VM | Preparation for creating a consistend snapshot |
| take a snapshot through VMWare | Use the VMWare GUI to create a snapshot of the VM in case something goes wrong |
| attach an additional hard drive | Use the VMWare GUI to attach another harddrive suitable for the amount of logs you want to store |
| start the VM again and follow these steps: | |
sudo graylog-ctl stop |
Stop all running services to prevent disk access |
sudo lshw -class disk |
Check for the logical name of the new hard drive. Usually this is /dev/sdb |
sudo parted -a optimal /dev/sdb mklabel gpt(A reboot may be necessary at this point)
sudo parted -a optimal -- /dev/sdb unit \\compact mkpart primary ext3 "1" "-1"sudo mkfs.ext4 /dev/sdb1 |
Partition and format new disk |
sudo mkdir /mnt/tmpsudo mount /dev/sdb1 /mnt/tmp |
Mount disk to temporary mount point |
cd /var/opt/graylog/datasudo cp -ax * /mnt/tmp/ |
Copy current data to new disk |
sudo diff -qr --suppress-common-lines \\/var/opt/graylog/data /mnt/tmp |
Compare both folders. Output should be: Only in /mnt/tmp: lost+found |
sudo rm -rf /var/opt/graylog/data/* |
Delete old data |
sudo umount /mnt/tmpsudo mount /dev/sdb1 /var/opt/graylog/data |
Mount new disk over data folder |
echo "/dev/sdb1 /var/opt/graylog/data ext4 \\defaults 0 0" \| sudo tee -a /etc/fstabsudo shutdown -r now |
Make change permanent |
Once the storage has been added you need to modify the configuration file path of elasticsearch to reflect the new storage. VIM the config file at /opt/graylog/elasticsearch/config/elasticsearch.yml. Look for the ‘path.data’ attribute in the document and add the new mount path. Keep in mind that if you’re replacing the data folder you copy all the contents of the existing data folder to the new path. NOTE: Before you modify the elasticsearch config file run ‘sudo graylog-ctl stop’ to stop the processes. Modify the file and run ‘sudo graylog-ctl start’ to start everything back up. I had a second session open tailing the elasticsearch log file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
root@graylog:/var/log/graylog/elasticsearch# tail -f graylog.log [2016-11-29 15:44:21,872][INFO ][cluster.service ] [Shrunken Bones] new_master {Shrunken Bones}{FVTu19qzQlq4W4Z3dRvuHg}{10.13.37.22}{10.13.37.22:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2016-11-29 15:44:21,900][INFO ][http ] [Shrunken Bones] publish_address {10.13.37.22:9200}, bound_addresses {10.13.37.22:9200} [2016-11-29 15:44:21,901][INFO ][node ] [Shrunken Bones] started [2016-11-29 15:44:22,100][INFO ][gateway ] [Shrunken Bones] recovered [1] indices into cluster_state [2016-11-29 15:44:29,704][INFO ][cluster.service ] [Shrunken Bones] added {{graylog-f3016b49-5173-4ce5-bad3-fd88b88b2f53}{B90Y0lr-Qjin_Q24nER_gw}{10.13.37.22}{10.13.37.22:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-f3016b49-5173-4ce5-bad3-fd88b88b2f53}{B90Y0lr-Qjin_Q24nER_gw}{10.13.37.22}{10.13.37.22:9350}{client=true, data=false, master=false}]) [2016-11-29 15:45:14,043][INFO ][cluster.routing.allocation] [Shrunken Bones] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_1][3]] ...]). [2016-11-29 15:46:42,385][INFO ][node ] [Shrunken Bones] stopping ... [2016-11-29 15:46:45,545][INFO ][node ] [Shrunken Bones] stopped [2016-11-29 15:46:45,552][INFO ][node ] [Shrunken Bones] closing ... [2016-11-29 15:46:45,606][INFO ][node ] [Shrunken Bones] closed [2016-11-29 15:56:32,569][INFO ][node ] [Darkoth] version[2.3.1], pid[11395], build[bd98092/2016-04-04T12:25:05Z] [2016-11-29 15:56:32,574][INFO ][node ] [Darkoth] initializing ... [2016-11-29 15:56:33,618][INFO ][plugins ] [Darkoth] modules [reindex, lang-expression, lang-groovy], plugins [kopf], sites [kopf] [2016-11-29 15:56:33,687][INFO ][env ] [Darkoth] using [1] data paths, mounts [[/elasticsearch_data (/dev/sdb)]], net usable_space [9.4gb], net total_space [19.5gb], spins? [possibly], types [ext4] [2016-11-29 15:56:33,688][INFO ][env ] [Darkoth] heap size [2.3gb], compressed ordinary object pointers [true] [2016-11-29 15:56:33,688][WARN ][env ] [Darkoth] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536] [2016-11-29 15:56:37,394][INFO ][node ] [Darkoth] initialized [2016-11-29 15:56:37,394][INFO ][node ] [Darkoth] starting ... [2016-11-29 15:56:37,584][INFO ][transport ] [Darkoth] publish_address {10.13.37.22:9300}, bound_addresses {10.13.37.22:9300} [2016-11-29 15:56:37,597][INFO ][discovery ] [Darkoth] graylog/kRTWCFAUSAyfWd-FRj5TWA [2016-11-29 15:56:47,686][INFO ][cluster.service ] [Darkoth] new_master {Darkoth}{kRTWCFAUSAyfWd-FRj5TWA}{10.13.37.22}{10.13.37.22:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2016-11-29 15:56:47,770][INFO ][http ] [Darkoth] publish_address {10.13.37.22:9200}, bound_addresses {10.13.37.22:9200} [2016-11-29 15:56:47,770][INFO ][node ] [Darkoth] started [2016-11-29 15:56:47,796][INFO ][gateway ] [Darkoth] recovered [1] indices into cluster_state [2016-11-29 15:56:48,494][INFO ][cluster.service ] [Darkoth] added {{graylog-f3016b49-5173-4ce5-bad3-fd88b88b2f53}{vOBRN3MKT9eLs-N1-lbdsw}{10.13.37.22}{10.13.37.22:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-f3016b49-5173-4ce5-bad3-fd88b88b2f53}{vOBRN3MKT9eLs-N1-lbdsw}{10.13.37.22}{10.13.37.22:9350}{client=true, data=false, master=false}]) [2016-11-29 15:56:50,269][INFO ][cluster.routing.allocation] [Darkoth] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][3], [graylog_0][1], [graylog_0][2], [graylog_0][1], [graylog_0][2], [graylog_0][3]] ...]). |
Windows Active Directory setup:
To setup Windows reporting (AD and event logs) I found the following add-on8)https://marketplace.graylog.org/addons/750b88ea-67f7-47b1-9a6c-cbbc828d9e25. Read the readme and requirements. I installed the NXLog9)https://nxlog.co/products/nxlog-community-edition/download agent onto my 2012r2 domain controller. I was able to use the default nxlog configuration below (different from the nxlog installed default config).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Input in> # For windows vista/2008 and above use: Module im_msvistalog # For windows 2003 and earlier use the following: # Module im_mseventlog </Input> <Output out> Module om_udp Host graylog.server.com Port 5141 OutputType GELF </Output> <Route 1> Path in => out </Route> |
As you can see under <Output out> in the above context you need to create a input on your Graylog server for GELF UDP port 5141. If you’re having any issues with data showing up check out the logs of nxlog: C:\Program Files (x86)\nxlog\data\nxlog.log. To install the plugin I navigated to the upload at http://yoursyslogserver.com/system/contentpacks/.
Here is the .conf for Windows eventlogs10)https://gist.github.com/tom-henderson/dc9572a973003db18019#file-nxlog-conf:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
define ROOT C:\Program Files (x86)\nxlog Moduledir %ROOT%\modules CacheDir %ROOT%\data Pidfile %ROOT%\data\nxlog.pid SpoolDir %ROOT%\data LogFile %ROOT%\data\nxlog.log <Extension gelf> Module xm_gelf </Extension> <Input in> # Use 'im_mseventlog' for Windows XP and 2003 Module im_msvistalog Exec if ($EventID == 4202 or $EventID == 4208 or $EventID == 4302 or $EventID == 4304 or $EventID == 5004) drop();\ else{\ if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\ if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\ if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\ } </Input> <Output out> Module om_udp Host syslog.yourwebsite.com Port 5414 OutputType GELF </Output> <Route 1> Path in => out </Route> |
- Installing ssacli – view raid ESXI 6.5 - December 6, 2017
- Installing hpacucli on Proxmox - December 5, 2017
- Finding your node name Graylog 2.2.x - May 22, 2017
Notes & References
| 1. | ↑ | https://github.com/mcholste/elsa |
| 2. | ↑ | http://docs.graylog.org/en/2.1/pages/getting_started/web_console.html |
| 3. | ↑ | http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html?highlight=admin%20password |
| 4. | ↑ | http://docs.graylog.org/en/2.1/pages/installation/virtual_machine_appliances.html#vmware-tools |
| 5. | ↑ | http://docs.graylog.org/en/2.1/pages/securing.html |
| 6. | ↑ | http://docs.graylog.org/en/2.1/pages/configuration/backup.html |
| 7. | ↑ | https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html |
| 8. | ↑ | https://marketplace.graylog.org/addons/750b88ea-67f7-47b1-9a6c-cbbc828d9e25 |
| 9. | ↑ | https://nxlog.co/products/nxlog-community-edition/download |
| 10. | ↑ | https://gist.github.com/tom-henderson/dc9572a973003db18019#file-nxlog-conf |