Search head clusters

A search head cluster is a group of Splunk Enterprise search heads that share configurations, search job scheduling, and search artifacts, which are the results and associated metadata from a completed search job. You will want to utilize a search head cluster in a distributed Splunk deployment to handle more users and concurrent searches, and to provide multiple search heads so that search capability is not lost if one or more search members goes down. There are a few design-related factors about search head clusters you need to be aware of while architecting and especially administering your Splunk solution, which we'll cover here.

In a search head cluster, one of the members has the role of captain, whose duties include, among others:

• Coordinating job scheduling: Scheduled search jobs (reports, alerts, and so on) are assigned to one of the search members by the captain when it is time for them to be run.
• Coordinating replication activities across the members: That changes to saved searches, lookup tables, dashboards, and so on made by a user on one search member are replicated to all the other members so that all of them are working with the same knowledge objects.
• Sending knowledge bundles to the search peers: A knowledge bundle is a set of knowledge objects residing on the search head that is sent to the search peers so that they can properly process search requests.

The captain will also handle its own replication tasks and assign search jobs to itself unless purposely configured not to, so it shares the search head loads just like any of the other members.

Any search member can perform the role of captain, but there is only one captain at any given time, and the search head member that is to perform the role of captain is selected by an election process between the members. If a failure occurs, the election process may be repeated and the captain can change, although a given member will usually remain the captain for quite some time unless that member itself goes down. The election process involves a member winning a majority vote of all members, and involves a random timer on each member – the member whose timer runs out first stands for the election; the other members usually comply and that member becomes the new captain. Note that Splunk uses the Raft Consensus Protocol from Stanford to manage leader election and auto-failover.

Because of how the majority vote process works, a search head cluster should consist of a minimum of three members. If either member of a two-member cluster fails, a captain cannot be elected and you lose the benefits of the captain's role in a search head cluster. I've introduced this topic here because you need to be aware of this constraint, and why we need three search heads to build a minimal cluster.

If you employ a search head cluster, you will also need to provide a deployer that distributes Splunk apps and other configurations to the search cluster members. You may also want to place the search cluster members behind a load balancer so that users can access the search heads through a single URL, without the need to specify a specific search head.

Splunk provides a document entitled Search head clustering architecture that you can read if you want to gain a more through understanding of clustered search head functions and considerations—it is available here: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/SHCarchitecture.