Search heads

After configuring the licensing and forwarding the internal logs, you can configure a node to be a member of a search head cluster, either by executing a CLI command, or editing the server.conf file directly—there is no provision to make all these settings in Splunk Web. Let's try the CLI command splunk init shcluster-config first, providing all the needed elements in the arguments:

• auth: The admin-level username and password.
• mgmt_uri: scheme:ipaddress:port of the node you're configuring.
• replication_port: A port you've selected for the search heads to replicate knowledge artifacts with each other.
• replication_factor: The number of searchable copies of data (index files) to be retained on the indexers (default is 2).
• conf_deploy_fetch_url: The scheme:ipaddress:port of the deployer – this is so that the search head knows who to contact to get updates if it was previously down.
• secret: The same pass4SymmKey password you configured on the deployer - use this same secret key on all the search heads, too. If you use certain non-text characters in your password, you may need to wrap this entry in single quotes.
• shcluster_label: The same cluster label you configured in the deployer.
• Here's what the command looks like fully filled out, with entries for our example deployment:

./splunk init shcluster-config -auth admin:Splunk1t2me -mgmt_uri https://172.31.28.137:8089 -replication_port 8090 -replication_factor 2 -conf_deploy_fetch_url https://172.31.18.102:8089 -secret '!Sp1unkSH!' -shcluster_label DevTestSearchHeads

The preceding command creates the following entries in the /opt/splunk/etc/system/local/
server.conf file. Note that the disabled = 0 entry is needed because search head clustering is disabled by default in /opt/splunk/etc/system/default/server.conf:

[replication_port://8090]

[shclustering]
conf_deploy_fetch_url = https://172.31.28.225:8089
mgmt_uri = https://172.31.28.137:8089

pass4SymmKey = $1$q3Fg5DtBkC6yGZA=
replication_factor = 2
disabled = 0
shcluster_label = DevTestSearchHeads

Next, you need to configure the search heads to communicate with the cluster master, and thus join them to the indexing cluster. The cluster master provides a list of the search peers (indexers) to the search heads so that they can contact the active indexers for search requests. Here's the CLI command to run on each search head:

./splunk edit cluster-config -mode searchhead -master_uri https://172.31.18.102:8089 -secret '!Sp1unkCM!'

Note that you're specifying the cluster master's URI, and providing the index cluster secret key. Running this command will add the following entries to the server.conf file:

[clustering]
master_uri = https://172.31.18.102:8089
mode = searchhead
pass4SymmKey = $1$q3Fg5DtBkG23GZA=

You can configure nodes to be a clustered search head by directly editing the server.conf file with the correct entries; sometimes, this is easier if you're sure of the entries to be made. After restarting Splunk on these servers, you will have a search head cluster!