Clustered Splunk Enterprise installations are mainly managed by the related management nodes. Indexer Cluster with the Cluster Master and Search Head Cluster with the Search Head Cluster Deployer. This kind of architecture already provides a certain level of centralized configuration management, but it does not provide any version control. Without an additional process of change management, there is no way to easily track down what has been changed, by who and when. But there are multiple options to achieve this goal. Easily.
Server management with Spacewalk
Centralized server configuration and software management is a must even in smaller Linux environments. Depending on the used Linux distribution, there are different options available, but for a CentOS 7 based environment, using Spacewalk is highly recommended. It provides a centralized repository of all distribution related packages and also custom apps, like, Splunk Enterprise or Splunk Universal Forwarder.
Using Spacewalk provides a detailed overview of your whole Splunk Enterprise installation from the OS point of view. Like
- Are there any critical patches missing from some servers
- What is the installed Splunk software version and are there any updates
- Which configuration files are not in line with the baseline
With Spacewalk, there is no need to check all this information individually from each server, but all the data can be easily seen from a web based user interface.
Deploying new Splunk instances and automated scaling
Spacewalk does not limit itself to just managing already installed servers. By using registration keys, the default software and configurations of a Splunk server can be installed automatically when the server is first registered to Spacewalk. For example, a new server registration can trigger Splunk Universal Forwarder installation and registration to Deployment Server which will then automatically install all the required Splunk apps to the server and start monitoring logs automatically.
This method can also be extended to installing new Splunk Enterprise nodes. Registration to Spacewalk with a specific key can automatically add the server to a Splunk Indexer Cluster or a Splunk Search Head Cluster, allowing almost automated scaling of the cluster installation. This can be achieved by managing the Splunk Enterprise configuration files with Spacewalk.
Managing Splunk Enterprise Configuration
Spacewalk provides a feature called Configuration Channels. Each Configuration Channel may contain file and folder definitions and they can be mapped to multiple servers. What we normally do, is we create different channels for
- cluster master
- cluster slave
- search head
- deployer
- deployment server
Each of these can be thought of as a role of the target server. A newly installed Splunk Enterprise is registered with the selected Configuration Channel and can pull all required configuration automatically.
You may ask, “But aren’t the configurations on each server then identical?”. Well yes. And no. Spacewalk includes a concept of macros and custom keys. By using macros and custom keys, the configuration files can be identical but server specific. Any server specific information is stored as a custom key value to the managed server and the configuration just references this data, or sets a default value if a custom key value is not found. In addition to custom keys, there are certain standard macros, that provide for example the server hostname or the ip address of a certain network interface.
By using custom keys, we normally manage things like
- Cluster wide secret
- Splunk role of the server
- Cluster master information (address and label)
- License master information
- Default server name
- SSL Certificate and key location
- Indexer discover
When additional tuning is required for different clusters, we can clone a configuration channel and subscribe that the required servers and then create the changes there. This way we only need to manage single configurations for a single cluster. From a single location.
Role specific configuration management
As Splunk Enterprise nodes in a cluster needs to be managed by the cluster management node, the cluster specific configuration is included in the channel targeted for the cluster master or the deployer. So for example in an indexer cluster, indexes.conf will always be deployed to the master, which will then deploy the configuration to the indexers.
The same applies to forwarder management: all app configurations managed by Spacewalk are always deployed to Deployment Server, which will then push the configurations to the Forwarders when they check in.
Local Splunk configurations under /opt/splunk/etc/system/local are always deployed directly to the target servers.
Wrap up
To efficiently manage a complex Splunk environment, using a central configuration management system with version control is highly recommended.
When you add additional Spacewalk features, such as software channels, you can have a single location for fully managing your Splunk software versions as well and do all Splunk configuration and upgrades centrally.
Obviously Spacewalk is not specific for Splunk – it just happens to work perfectly for the purpose. In addition, you will be able to manage all kinds of server parameters and software settings.
Splunk delivery models
Mint Security has a set of predefined delivery models to choose from. These are based on best practices and experience.
