The incomplete path to a syslog

The path to a great syslog! My first full-fledged syslog deployment should be interesting, and an educational read for anyone who’s on a similar path.

**sips coffee**

Ok, off to the races!

Identify what your business needs are.

  • Why do you need a syslog server?
  • What’s your budget?
  • Can your existing network meet the demand of a syslog?
  • Number of devices logging and messages per second
  • What is your desired retention period?
  • Understand the topology of your network and where you may need forwarders setup.
  • What are you looking for as sources? See my SOURCES list for an example.

Top Syslog options (in my preferred order):

  • Splunk (paid – free up to 500mb per day)
  • Graylog (opensource)
  • Rsyslog (opensource)
  • ELK (Elasticsearch, Logstash, Kibana) (opensource)
  • Logz.io (paid)
  • Elsa (opensource)1)https://github.com/mcholste/elsa
  • SolarWinds LEM (paid)
  • Kiwi (paid)
  • AlertLogic (paid)

Out of all the options, I ended up choosing Graylog. The deciding factor for me was cost. Graylog is a capable solution, especially Graylog2. From my research there were quite a few issues with the server falling over after 2k+ messages per second. However that was addressed with Graylog2.x. I’m anticipating  throwing a RSYSLOG server into the mix for long-term archival of logs while the Graylog server will handle recent events (<3 months).

Once you’ve decided which stack you’re moving forward with install it! The remainder of this document will be encompassing ELK/Graylog and my experiences with it. During demoing the many products, Splunk was my favorite. The best thing about it was that it had pre-built apps that would organize my data into something meaningful. With Graylog, or other opensource solutions, you’ll have to manually create those visuals and streams.

When I first installed Graylog OVA for VMware it didn’t go well. My experience was a very negative one, nothing seemed to be functioning properly. The largest issue was that elasticsearch wasn’t able to start. There was also a number of libraries missing on the distro. I took the night off and attempted again a few days later with a freshly downloaded OVA. Spun it up and everything worked per the getting started guide.2)http://docs.graylog.org/en/2.1/pages/getting_started/web_console.html

Note: if you change IP (I went from DHCP to static) you need to run “sudo graylog-ctl reconfigure”

Once I wrapped up the getting started everything ‘just worked’. I created a few inputs and pointed my equipment accordingly. Data was showing up! Success! I ended up creating a few dashboards, mostly just to make sure everything was working.

Graylog_dashboard

Tips!

As I find more time I’ll continue to add more to this article outlining additional tips.

Syslog sources:

  • Windows event logs
  • Firewalls
  • WAPs
  • Switches
  • Antivirus
  • VMWARE
  • Cameras and DVR
  • Linux Servers
  • Software
    • Exchange
    • ADSU
    • DNS
    • DHCP
    • Backup
    • Citrix

OVA storage tips and advice:

The default OVA installation of Graylog has a caveat, IDE storage. I installed the default 20GB storage for my graylog testing and found that it quickly filled up (within two weeks of logging 5 devices). I attempted to extend the disk, but with vmware, you cannot extend IDE disks, only SCSI. I ended up having to add an additional disk (SCSI so I can extend) to the VM. To extend your disk after it is low on space run the command:

This added an additional 80GB to my 20GB elasticsearch_data partition /dev/sdb/. If you find your state is red and you have unassigned shards you may want to follow a great guide7)https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html to resolve.

When running the script that I linked to T37 above, I ran into issues. I had to enclose the $shard in ”. I also didn’t realize my node name was ‘Bling’ which I believe is a default. Below is the script I ran:

 

Example procedure for an OVA appliance on VMWare:

Action Explanation
shutdown the VM Preparation for creating a consistend snapshot
take a snapshot through VMWare Use the VMWare GUI to create a snapshot of the VM in case something goes wrong
attach an additional hard drive Use the VMWare GUI to attach another harddrive suitable for the amount of logs you want to store
start the VM again and follow these steps:
sudo graylog-ctl stop
Stop all running services to prevent disk access
sudo lshw -class disk
Check for the logical name of the new hard drive. Usually this is /dev/sdb
sudo parted -a optimal /dev/sdb mklabel gpt
(A reboot may be necessary at this point)
sudo parted -a optimal -- /dev/sdb unit \\
compact mkpart primary ext3 "1" "-1"
sudo mkfs.ext4 /dev/sdb1
Partition and format new disk
sudo mkdir /mnt/tmp
sudo mount /dev/sdb1 /mnt/tmp
Mount disk to temporary mount point
cd /var/opt/graylog/data
sudo cp -ax * /mnt/tmp/
Copy current data to new disk
sudo diff -qr --suppress-common-lines \\
/var/opt/graylog/data /mnt/tmp
Compare both folders. Output should be: Only in /mnt/tmp: lost+found
sudo rm -rf /var/opt/graylog/data/*
Delete old data
sudo umount /mnt/tmp
sudo mount /dev/sdb1 /var/opt/graylog/data
Mount new disk over data folder
echo "/dev/sdb1 /var/opt/graylog/data ext4 \\
defaults 0 0" \| sudo tee -a /etc/fstab
sudo shutdown -r now
Make change permanent

Once the storage has been added you need to modify the configuration file path of elasticsearch to reflect the new storage. VIM the config file at /opt/graylog/elasticsearch/config/elasticsearch.yml. Look for the ‘path.data’ attribute in the document and add the new mount path. Keep in mind that if you’re replacing the data folder you copy all the contents of the existing data folder to the new path. NOTE: Before you modify the elasticsearch config file run ‘sudo graylog-ctl stop’ to stop the processes. Modify the file and run ‘sudo graylog-ctl start’ to start everything back up. I had a second session open tailing the elasticsearch log file:

 

Windows Active Directory setup:

To setup Windows reporting (AD and event logs) I found the following add-on8)https://marketplace.graylog.org/addons/750b88ea-67f7-47b1-9a6c-cbbc828d9e25. Read the readme and requirements. I installed the NXLog9)https://nxlog.co/products/nxlog-community-edition/download agent onto my 2012r2 domain controller. I was able to use the default nxlog configuration below (different from the nxlog installed default config).

As you can see under <Output out> in the above context you need to create a input on your Graylog server for GELF UDP port 5141. If you’re having any issues with data showing up check out the logs of nxlog: C:\Program Files (x86)\nxlog\data\nxlog.log. To install the plugin I navigated to the upload at http://yoursyslogserver.com/system/contentpacks/.

graylog-web-interface

graylog-web-interface

graylog-web-interface

Here is the .conf for Windows eventlogs10)https://gist.github.com/tom-henderson/dc9572a973003db18019#file-nxlog-conf:

 

Mr-Moo

IT Director/System Engineer at 127.0.0.1
Currently residing in Southwest Florida, though Indiana native. I've always enjoyed technology and it's been a passion my entire live.

Latest posts by Mr-Moo (see all)

Notes & References   [ + ]

1. https://github.com/mcholste/elsa
2. http://docs.graylog.org/en/2.1/pages/getting_started/web_console.html
3. http://docs.graylog.org/en/2.1/pages/configuration/graylog_ctl.html?highlight=admin%20password
4. http://docs.graylog.org/en/2.1/pages/installation/virtual_machine_appliances.html#vmware-tools
5. http://docs.graylog.org/en/2.1/pages/securing.html
6. http://docs.graylog.org/en/2.1/pages/configuration/backup.html
7. https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html
8. https://marketplace.graylog.org/addons/750b88ea-67f7-47b1-9a6c-cbbc828d9e25
9. https://nxlog.co/products/nxlog-community-edition/download
10. https://gist.github.com/tom-henderson/dc9572a973003db18019#file-nxlog-conf

Leave a Reply