Sometimes, a search query does not return the expected results. Or displays some entries which should not be returned. Or returns results in a counter-intuitive order. This article shows ways to troubleshoot those issues.
Luke is the tool of choice to understand search results. It is a free Java utility that lets you analyse a Lucene index.
When trying to understand why some items are not returned, Luke allows to know whether the issue is upstream (the information is not indexed) or downstream (a widget is not reading the index correctly).
When trying to understand why some unexpected items are returned, Luke gives more information about said items.
Last but not least, Luke can give precise details about how an item ranking is computed.
You can download this tool from https://luke.googlecode.com/files/lukeall-3.5.0.jar. To run it, type "java -jar lukeall-3.5.0.jar" in a console and open the Sitefinity Lucene index directory, by default located in ~/App_Data/Sitefinity/Search/[index name]
On the default Overview tab, Luke allows to see the various fields indexed as well as the number of terms on the bottom left table:
The bottom right table shows the top terms for the whole index, or just for a particular field. For instance, if you select the ContentType field and click "Show top terms >>", you will see the number of document indexed per type (e.g. telerik.sitefinity.pages.model.pagenode represents the number of Pages being indexed):
You can also perform some searches in the Search tab (in the rest of this article, search queries will be displayed inside square brackets):
The search works using a series of one or more <field name>:<term> predicates - e.g. [Title:hello]. Note that the query is case sensitive: the field name should always use the exact same case than the field name displayed, but the term should always be lowercase, e.g. [Title:hello]. Adding a + before a predicate indicates it is mandatory, e.g. [+Title:hello +Title:world] searches for content whose title contains both "hello" and "world".
Notice how the ContentType field indicates the type of content you're dealing with. When a result shows up when it shouldn't, it is generally because Sitefinity is indexing a similar item of another type. The ContentType combined with the Id (and sometimes the Link field) helps pinpoint the exact item being indexed.
When trying to understand the search results, the first step is often to try to reproduce the issue inside Luke by running a Lucene search query similar with what Sitefinity is running under the hood. Here are the general rules:
As the default behavior, searching for [company] will search for any term starting with "company". This is not achieved by using wildcards (even though they are supported by Lucene), but by rewriting the query internally before sending it to Lucene.
When searching for [company], Sitefinity will look for terms in the Lucene index starting with "company" (e.g. "companyA", "companyB"). If it finds such terms, it will rewrite the query internally to search for company, companyA or companyB.
This behavior can be disabled by going to Administration / Settings / Advanced / Search, and checking "Enable exact match"
Note that Sitefinity does NOT support stemming, e.g. searching for [company] will NOT find occurrences of "companies". However, searching for [compan] would look for occurrences of "company", "companies", "companyA" and "companyB" if such terms are already indexed.
By default, Sitefinity is looking at only two fields when performing a search: Title and Content. You can however index and search in extra fields. In the example below, we add "Symptom" (a field added to a dynamic module) by editing the index and adding the field name under "Additional fields for indexing" in the Advanced section:
After a reindex, we now see a new field:
The last step is to update the Search Results widget (in the Advanced properties) to both search for the Symptom field and to highlight any keyword found in that field:
A common request is to be able to perform a more granular search than a given type, e.g. search for documents inside a particular library. This is achieved by:
Keep in mind that anything that the search widgets process must be in the Lucene index - whether filtering results or displaying extra fields. Before adding any clause that filters on a particular field, make sure that the index contains actual values for that field. Those values can also give you hints about what to filter.
For example, filtering documents stored in a particular library requires to filter for results whose "Link" field begins with, say "~/docs/default-source/sub-library/". A look at the Link top terms in the Overview tab however indicates that this field is broken down by words. In other words, it is not possible to add a Lucene query filter that looks for results whose Link field starts with "~/docs/default-source/sub-library/". It is thus best to filter the elements after the search result. Below is an implementation example:
MySearch : SearchResults
MySearcher : ISearchResultsBuilder
.control = control;
var control =
var service = Telerik.Sitefinity.Services.ServiceBus.ResolveService<ISearchService>();
var queryBuilder = ObjectFactory.Resolve<IQueryBuilder>();
var searchQuery = queryBuilder.BuildQuery(query, control.SearchFields);
searchQuery.IndexName = catalogue;
searchQuery.Skip = skip;
searchQuery.Take = take;
searchQuery.HighlightedFields = control.HighlightedFields;
// Contains the default filter - by current language
var currentFilter = searchQuery.Filter;
var myFilter =
myFilter.Operator = QueryOperator.And;
MySearch myControl = (MySearch)control;
// Persist the language filter, if exists
searchQuery.Filter = myFilter;
IResultSet result = service.Search(searchQuery);
var filtered_result = myControl.Link.IsNullOrEmpty() ?
result.Where(r => r.GetValue(
List<IDocument> documents = filtered_result.SetContentLinks().ToList<IDocument>();
hitCount = documents.Count();
Note that this control defines a Link property which can be accessed when looking at the advanced properties of the widget:
This avoids the need to hard-code the library path in the control itself, making it reusable.
Ranking is always a difficult topic, as there will always be some user who disagree with the ranking.
Nonetheless, Luke can help you understand the rationale behind a particular ranking. In the Search tab, select a result and click on the Explain button.
Lucene relies on three scores to determine ranking:
View all posts from Laurent Poulain on the Progress blog. Connect with us about all things application development and deployment, data integration and digital business.
Subscribe to get all the news, info and tutorials you need to build better business apps and sites
Copyright © 2019 Progress Software Corporation and/or its subsidiaries or affiliates.
All Rights Reserved.
Progress, Telerik, and certain product names used herein are trademarks or registered trademarks of Progress Software Corporation and/or one of its subsidiaries or affiliates in the U.S. and/or other countries. See Trademarks for appropriate markings.