Merging Indexes

The process of combining data from multiple indexes into a single index is called merging. Merging results in getting rid of some redundant data and also freeing resources. Queries are also resolved faster with fewer indexes. There are three types of merges:

Shadow Merge
Master Merge
Annealing Merge

After a merge is completed, the multiple source indexes are replaced by a single target index.

Shadow Merge

Combining multiple word lists and shadow indexes into a single shadow index is called a shadow merge. A shadow merge is performed to free memory used by word lists and also to make the filtered data persistent; it is usually a quick operation.

In the most common case, the source indexes for a shadow merge are word lists. However, if the total number of shadow indexes exceeds MaxIndexes, some of the shadow indexes are also used as source indexes. Shadow indexes are also used as source indexes during an annealing merge.

A shadow merge is triggered by one of the following conditions:

The number of word lists exceeds MaxWordLists.
The combined size of WordLists exceeds MinSizeMergeWordLists.
As a precursor to a master merge. Before starting a master merge, a shadow merge is performed to merge all existing word lists into a shadow index.
For an annealing merge.

Master Merge

For a master merge, the source indexes are all of the existing shadow indexes and the current master index (if any). At the end of a master merge, all the source indexes are replaced by a single target master index. Although the master merge itself is a very resource-intensive (both for CPU and disk space) operation, after the completion of a master merge, resources are freed. Redundant data is deleted and queries run faster.

Depending upon the size of the source indexes, a master merge can be a very long-running operation. However, it is fully restartable after failures and shutdowns. A master merge will continue from where it left off.

Whenever a master merge is started, restarted, or paused, an event is written to the event log. There are several reasons for starting a master merge. Some reasons for starting a master merge follow.

Nightly maintenance master merge. This can be done at a specified time every night. The registry value MasterMergeTime is the number of minutes after midnight when the merge should happen. By default, the nightly master merge happens at midnight. This value should be adjusted to reflect the time when the load on the server is lowest.
When the number of changed documents since the last master merge exceed MaxFreshCount, a master merge is performed to reduce the number of changed documents. If the number of changed documents is too high, it puts an extra load on memory usage. A master merge reduces the FreshCount to zero.
When the disk space remaining on the catalog drive is less than MinDiskFreeForceMerge and the cumulative space occupied by shadow indexes exceeds MaxShadowFreeForceMerge, a master merge is started to combine the shadow indexes and free up disk space.
When the total disk space occupied by shadow indexes exceeds MaxShadowIndexSize, a master merge is started to combine the shadow indexes. This condition has higher precedence than the previous condition.
Finally, a master merge can be forced by an administrator by using the administrative Web pages. Because of the fact that a master merge will make queries run faster after it completes, the administrator may want to force a merge even before one of the preceding conditions triggers it.

Annealing Merge

An annealing merge is a special kind of shadow merge performed when the system is idle for a certain length of time and the total number of persistent indexes exceed MaxIdealIndexes. The registry parameter MinMergeIdleTime specifies the percentage of CPU time that must be idle during a time period to trigger an annealing merge. An annealing merge improves query performance and disk space usage by reducing the number of shadow indexes.