Splunk directory structure

 Splunk's directory structure is identical, whether you're running on Windows or Linux, once you get past the initial installation path - C:\Splunk for Windows (if you took the advice on shortening the path), and /opt/splunk on Linux. Beyond that, there are a number of directories, as follows (Linux and Windows follow the exact same directory layout):

bin/        binaries, python and shell scripts
etc/        numerous - see below
include/     python include file
lib/         various links to libraries
openssl/     openssl files
share/       most of the Splunk Web UI code resides here
var/         default indexes and Splunk logs 

Nearly all of the configuration work you will do as you administer Splunk on a day-to-day basis will be on .conf files (variously named files with a .conf extension) residing in several directories under the $SPLUNK_HOME/etc directory.

Let's explore the most prominent of these locations, as you'll want to get familiar with them as soon as possible. We will be working in these directories for configuring Splunk components and all kinds of administration tasks:

• $SPLUNK_HOME/etc/system/: This is where system-wide configuration files reside; they control what specific function a Splunk server performs (indexer, search head, and so on) and a number of other important system settings. Note that changes you make to Splunk's configuration using Splunk Web or the CLI will be stored in various .conf files located in the /opt/splunk/etc
  /system/local directory.
• $SPLUNK_HOME/etc/apps/: This is where Splunk apps reside, including default apps like the search function, apps you might install from Splunkbase, or apps you create yourself. A location you might want to peruse to get familiar with the contents of these directories is the search app: 
  $SPLUNK_HOME/etc/apps/search.

Note that under the /system and /apps directories, there is always a /default directory, and usually a /local directory. Splunk places all of its default operational settings in the .conf files in the /default directory upon installation – you should never alter any file in the default directory. Configuration changes made by an administrator or user – either by using Splunk Web, the CLI, or by editing the .conf files directly – should be done within .conf files located in the /local directory (create one if it doesn't already exist). Splunk then merges the contents of the files in /default and /local using precedence, which we'll cover in the next section.

• $SPLUNK_HOME/etc/auth/: Location of security certificates
• $SPLUNK_HOME/etc/users: User-specific configurations

I recommend that you explore these locations – open and read through some of the files to start getting a feel for what they contain. As you do, note that in each .conf file, the convention is to use stanzas such as [settings] and attributes entries under each stanza for settings pertaining to that stanza, such as this example from a $SPLUNK_HOME/etc/system/local/web.conf file:

[settings]
enableSplunkWebSSL = 1
httpport = 8000

Also, there is the following:

$SPLUNK_HOME/var/log/splunk/: This is where splunk creates and updates its own internal logs. You'll want to be familiar with the logs that are most useful for troubleshooting: splunkd.log, metrics.log, audit.log, and various access logs.

$SPLUNK_HOME/var/log/introspection/: This is where Splunk creates logs related to its kvstore, splunk_disk_objects (indexes and search-related artifacts), and splunk_resource_usage, which is a record of usage and performance-related metrics.