Configure Lucene analyzers
Overview
Lucene analyzers are components in Apache Lucene that process text during indexing and searching. Their job is to convert raw text into a stream of tokens (words or terms) that can be efficiently searched and matched.
An analyzer typically performs several steps:
- Tokenization – Splitting text into individual words or tokens.
- Normalization – Converting tokens to a consistent form (e.g., lowercasing).
- Filtering – Removing or transforming tokens.
Sitefinity CMS supports three analyzers: ClassicAnalyzer (the default one), StandardAnalyzer, and SitefinityWhitespaceAnalyzer.
ClassicAnalyzer
The ClassicAnalyzer splits text into words, lowercases all tokens, and removes common English stop words.
For more information, see Classic Analyzer.
EXAMPLE: A headline or a search input of "Product [ABC] review" generates the following tokens: product, abc, review.
StandardAnalyzer
The StandardAnalyzer is a general-purpose text analyzer that provides modern, Unicode-aware tokenization.
It splits text into words using the UAX #29 standard (handling punctuation, symbols, and languages correctly), lowercases all tokens, and removes common English stop words.
Use it when you need accurate, language-neutral text analysis for most search applications. For more information, see Standard Analyzer.
EXAMPLE: A headline or a search input of "Product [ABC] review" generates the following tokens: product, abc, review. For input like "U.S.A.", the ClassicAnalyzer generates a token "u.s.a.", while the StandardAnalyzer generates "u, s, a".
SitefinityWhitespaceAnalyzer
The SitefinityWhitespaceAnalyzer is a simple analyzer that splits text only on whitespace characters (spaces, tabs, newlines) without changing or filtering anything.
It preserves case, punctuation, and symbols, making it useful when you need exact matches or are indexing pre-tokenized data like IDs, codes, or keywords.
The SitefinityWhitespaceAnalyzer extends Lucene’s WhitespaceAnalyzer by adding LowerCaseFilter and StopFilter. This means text is lowercased and common English stop words are removed, while tokens are still split only by whitespace. For more information, see WhitespaceAnalyzer.
EXAMPLE: A headline or a search input of "Product [ABC] review" generates the following tokens: product, [abc], review.
Configure Lucene analyzers
To change the currently used analyzer you can do so by changing the value of the luceneServiceAnalyzerName parameter:
- Navigate to Administration » Settings » Advanced » Search » Search Services » LuceneSearchService » luceneServiceAnalyzerName.
- Change the value to one of the supported analyzers.
- Click Save changes.
- Reindex your search indexes for the changes to take effect.
Register custom analyzers
If you need to register a custom analyzer in Sitefinity CMS, this can be done through the ObjectFactory. Each custom analyzer must be registered with a unique name, as shown in the example below. Once the registration is complete, configure Sitefinity CMS to use this analyzer by navigating to Settings » Advanced » Search » Search Services » LuceneSearchService » Parameters » luceneServiceAnalyzerName and setting the Value to the name used during registration (for example: MyCustomAnalyzer).
This ensures that the LuceneSearchService uses your custom Analyzer for search indexing and querying within Sitefinity CMS.