Teemu Turpeinen

Teemu Turpeinen

Author works as a SecOps Specialist for Mint Security. He has over two decades of experience with Linux systems, over a decade with Identity and Access Management and has been using Splunk since around 2010. RHCSA & RHCE Certified.
Share on facebook
Share on twitter
Share on linkedin
Share on xing
Share on whatsapp

Clustered Splunk Enterprise installations are mainly managed by the related management nodes. Indexer Cluster with the Cluster Master and Search Head Cluster with the Search Head Cluster Deployer. This kind of architecture already provides a certain level of centralized configuration management, but it does not provide any version control. Without an additional process of change management, there is no way to easily track down what has been changed, by who and when. But there are multiple options to achieve this goal. Easily.

Server management with Spacewalk

Centralized server configuration and software management is a must even in smaller Linux environments. Depending on the used Linux distribution, there are different options available, but for a CentOS 7 based environment, using Spacewalk is highly recommended. It provides a centralized repository of all distribution related packages and also custom apps, like, Splunk Enterprise or Splunk Universal Forwarder.

Using Spacewalk provides a detailed overview of your whole Splunk Enterprise installation from the OS point of view. Like

  • Are there any critical patches missing from some servers
  • What is the installed Splunk software version and are there any updates
  • Which configuration files are not in line with the baseline

With Spacewalk, there is no need to check all this information individually from each server, but all the data can be easily seen from a web based user interface.

Spacewalk overview
Spacewalk overview

Deploying new Splunk instances and automated scaling

Spacewalk does not limit itself to just managing already installed servers. By using registration keys, the default software and configurations of a Splunk server can be installed automatically when the server is first registered to Spacewalk. For example, a new server registration can trigger Splunk Universal Forwarder installation and registration to Deployment Server which will then automatically install all the required Splunk apps to the server and start monitoring logs automatically.

This method can also be extended to installing new Splunk Enterprise nodes. Registration to Spacewalk with a specific key can automatically add the server to a Splunk Indexer Cluster or a Splunk Search Head Cluster, allowing almost automated scaling of the cluster installation. This can be achieved by managing the Splunk Enterprise configuration files with Spacewalk.

View list of managed configurations
Audit file modifications
Edit configurations
Manage server specific settings with custom keys

Managing Splunk Enterprise Configuration

Spacewalk provides a feature called Configuration Channels. Each Configuration Channel may contain file and folder definitions and they can be mapped to multiple servers. What we normally do, is we create different channels for

  • cluster master
  • cluster slave
  • search head
  • deployer
  • deployment server

Each of these can be thought of as a role of the target server. A newly installed Splunk Enterprise is registered with the selected Configuration Channel and can pull all required configuration automatically.

You may ask, “But aren’t the configurations on each server then identical?”. Well yes. And no. Spacewalk includes a concept of macros and custom keys. By using macros and custom keys, the configuration files can be identical but server specific. Any server specific information is stored as a custom key value to the managed server and the configuration just references this data, or sets a default value if a custom key value is not found. In addition to custom keys, there are certain standard macros, that provide for example the server hostname or the ip address of a certain network interface.

By using custom keys, we normally manage things like

  • Cluster wide secret
  • Splunk role of the server
  • Cluster master information (address and label)
  • License master information
  • Default server name
  • SSL Certificate and key location
  • Indexer discover

When additional tuning is required for different clusters, we can clone a configuration channel and subscribe that the required servers and then create the changes there. This way we only need to manage single configurations for a single cluster. From a single location.

Splunk Managed Delivery

Role specific configuration management

As Splunk Enterprise nodes in a cluster needs to be managed by the cluster management node, the cluster specific configuration is included in the channel targeted for the cluster master or the deployer. So for example in an indexer cluster, indexes.conf will always be deployed to the master, which will then deploy the configuration to the indexers.

The same applies to forwarder management: all app configurations managed by Spacewalk are always deployed to Deployment Server, which will then push the configurations to the Forwarders when they check in.

Local Splunk configurations under /opt/splunk/etc/system/local are always deployed directly to the target servers.

Wrap up

To efficiently manage a complex Splunk environment, using a central configuration management system with version control is highly recommended.

When you add additional Spacewalk features, such as software channels, you can have a single location for fully managing your Splunk software versions as well and do all Splunk configuration and upgrades centrally.

Obviously Spacewalk is not specific for Splunk – it just happens to work perfectly for the purpose. In addition, you will be able to manage all kinds of server parameters and software settings.

Teemu Turpeinen

Teemu Turpeinen

Author works as a SecOps Specialist for Mint Security. He has over two decades of experience with Linux systems, over a decade with Identity and Access Management and has been using Splunk since around 2010. RHCSA & RHCE Certified.
Share on facebook
Share on twitter
Share on linkedin
Share on xing
Share on whatsapp

contact us

Please do contact us. We most likely respond faster than you thought,