上QQ阅读APP看书，第一时间看更新

Disk-sizing calculations

From the information you gathered from your line of business users, you hopefully have a general idea of how much ingestion volume your Splunk deployment will need to accommodate, and have rounded that up to some 250 GB/day increment. You also know what the retention periods will be for all the data inputs; these are typically 7-14 days for dev/test data, and 30 days or more for production data, depending on the business case. Finally, you have decided on a replication factor and search factor—we'll assume that you have selected an RF of 2 and an SF of 2. Assuming you have a list of sizing options for a suitable disk subsystem solution that meets the required IOPS and RAID specification (800-1,200) for the indexers, you can now perform the needed calculations and trade-offs to determine how many indexers you will need.

There are a number of indexer disk space calculators available on the web, but here we will again employ a spreadsheet and do the calculations ourselves. The significant factors and formulas for calculating the disk space usage for one group of data inputs with the same retention period (14 days) is as follows:

RF = 2
SF = 2
GB/day ingestion rate = 125
Retention in days = 14

Splunk documentation suggests that a ballpark figure for calculating the actual disk space occupied by compressed and indexed data is 50%:

15% for the raw data file
35% for index files

The formulas for calculating disk space occupancy, in progressive order and with values rounded up, are as follows:

Base size of raw data = GB/day * .15 * Retention in days: 125 * .15 * 14 = 263 GB
Base size of index data = GB/day * .35 * Retention in days: 125 * .35 * 14 = 613 GB
Replicated size on disk = RF * Raw data + SF * Index data: 2 * 263 + 2 * 613 = 1,752 GB

Repeat the preceding calculations for each input group with a different retention period, then add them together to obtain the total disk occupancy. You can then divide the total disk storage needs by the per-indexer disk array size (after accounting for any RAID level reductions) to obtain the number of indexers needed to store the required volume.

A fairly fancy version of a disk-sizing calculator, which I used for calculating this example, is depicted in the Fig 2.3. Assuming a 500 GB/day total ingestion volume from several data input groups with differing retention periods, the ability to handle the loss of one indexer without exceeding our disk storage capability after all the missing buckets are redistributed, and with the availability of a 12-TB disk array on each indexer, we will need at least four indexers to store the 500 GB/day of incoming data for the desired retention periods.

Fig 2.3: Splunk indexer disk-sizing calculator

Note that in the preceding example, the per-indexer ingestion volume is only 125 GB/day, which is roughly half (or less) of what a typical reference server indexer can handle, depending on its specifications—it is common for disk storage, and not indexer processing capabilities, to be the limiting factor that drives indexer counts up. But there is a solution — the preceding spreadsheet depicts the calculations of hot and warm versus cold storage sizes (leveraging the number of days with a hot/warm value of 14), which reveals that by utilizing some slower, cheaper, off-indexer disk storage for cold data, you can reduce your per-indexer storage needs from 12 TB to ~5.25 TB, which would allow for a smaller number of indexers and/or increased indexer utilization. The search performance for events older than 14 days would be slower, but the cost savings in terms of both storage and indexer hardware would very likely warrant the performance hit.

Once you've made your final hardware selections, and completed your disk size and indexer count calculations, you're ready to procure your Splunk servers.