Boost Documents with Freshness Function in Azure Search

In the previous article, we explored how weighted fields can help us boost the documents in Azure search. Here, we will explore the Freshness Function and how it can help us to boost documents.

Before starting through this article, it would be good to have background knowledge of Azure Search. We will use the same examples that we have used in the previous article, so reading the previous article once will be very helpful.

What is the Purpose of the Freshness Function?

The Freshness Function is one more way to change the default search scores of documents. It only works on the DateTimeOffset fields. For example, a document with a more recent date can be ranked higher than older items.

In other words, if a new document is uploaded to the index, that document may be rated higher. Items such as calendar events with future dates closer to the present may be ranked higher than those far ahead in the future.

The freshness function is made up of the following Parameters :

FieldName - Any DateTimeoffset type property within an index definition.
Boost - A value that applies Weight to a DateTimeOffset field.
BoostingDuration - A Timespan object that may also be negative [-].
Interpolation - The scoring slope is constantly falling because the scoring goes from high to low, but the interpolation determines the curve of the downward slope. It's set to linear by default. Refer to this documentation to know more about interpolation.

Note: As per the current service release, one end of the range will be fixed to the current date-time. The other end is a time in the past based on the BoostingDuration property value. To boost a range of times in the future, use a negative value in the BoostingDuration property.

If you've read this far, you would think this can be done with the $orderby clause, so why is the Freshness Function needed? 🤔

Yes, without a doubt, but what if I ask you if there are 100 future events already established, but I only want the ones scheduled within the next 10 days to appear first? Of course, we can achieve it with complex, lengthy logic and many lines of code. If you have ten to twenty scenarios of this type, your code logic becomes much more complicated, whereas Freshness Function would solve that quickly.

How is that going to help?

Based on the examples of events explained in the last post, Freshness boosting is helpful as follows:

Boosting a newly added event could help in selling more tickets to these events sooner.
Inverted-freshness-boosting: Events could be boosted a few days before the actual event date, thus ensuring they will be returned to better positions a few days before the event, even if their actual score result (un-boosted score) isn't too high.

How do I add a Freshness Function?

Using Azure Search Client Library, Freshness boosting is applied by adding a FreshnessFunction to the list of functions within a scoring profile. Considering the previous example, this is done like as shown below:

var eventfreshnessfunction = new FreshnessFunction()
{
    Boost = 5,
    BoostingDuration = new TimeSpan(15,5,0,0,0),
    FieldName = "EventDate",
    Interpolation = InterpolationTypes.Linear
};
sc.Functions = new List() { eventfreshnessfunction }; /* where sc is scoring profile object */

Or, If you would like to update the existing search index, we can add a function directly to an Azure Portal by clicking on Add Function that can update the index definition on its own.

In the above code snippet, a new FreshnessFunction is instantiated with the following properties

Boost applied to any search results that match the keywords is 5.
Boosting is applied to the DateTimeOffset type field named EventDate but only within the range specified in the BoostingDuration field. Other items will be fallbacks as per the default score.
Interpolation set to Linear means items that are within the range, the boost applied to the item will be done in a constantly decreasing amount.

Note: Here, One end is connected to the current date, whereas the other is connected to the past, i.e. (Current date - 15 days and 5 hours). If we want a future date instead of a past date, we can use a negative Timespan object in the BoostingDuration property, such as (- new TimeSpan(15,5,0,0,0)).

Example

Before continuing with the article, if you're unfamiliar with query writing on the Azure portal, please refer to this article.

Consider today is April 15th, 2022, and we need to fetch all of the events in the index, but the ones that are more recent to today's date should be presented first, according to the Freshness Function specified above.

Ex: 1 - Wild Search without the use of a function

"Output": [
        {
            "@search.score": 1,
            "EventName": "Mumbai vs Chennai",
            "EventDate": "2022-04-03T13:56:45.556Z",
        },
        {
            "@search.score": 1,
            "EventName": "Banglore vs Chennai",
            "EventDate": "2022-04-02T13:56:45.556Z",
        },
        {
            "@search.score": 1,
            "EventName": "Mumbai vs Kolkatta",
            "EventDate": "2022-04-05T13:56:45.556Z",
        },
        {
            "@search.score": 1,
            "EventName": "Kolkatta vs Delhi",
            "EventDate": "2022-04-06T13:56:45.556Z",
        },
        {
            "@search.score": 1,
            "EventName": "Chennai vs Hydrebad",
            "EventDate": "2022-04-04T13:56:45.556Z",
        },
        {
            "@search.score": 1,
            "EventName": "Delhi vs Mumbai",
            "EventDate": "2022-04-01T13:56:45.556Z",
        }
    ]

When we look at the result set, we see that the documents are not organized most recently to the current date, so we didn't receive the desired result.

Now, if we use FreshnessFunction to make the same request, the output will show that the order of the documents has changed, with the most recent date event on top.

"Output": [
        {
            "@search.score": 4.138331,
            "EventName": "Kolkatta vs Delhi",
            "EventDate": "2022-04-06T13:56:45.556Z",
        },
        {
            "@search.score": 4.032471,
            "EventName": "Mumbai vs Kolkatta",
            "EventDate": "2022-04-05T13:56:45.556Z",
        },
        {
            "@search.score": 3.9197402,
            "EventName": "Chennai vs Hydrebad",
            "EventDate": "2022-04-04T13:56:45.556Z",
        },
        {
            "@search.score": 3.7991827,
            "EventName": "Mumbai vs Chennai",
            "EventDate": "2022-04-03T13:56:45.556Z",
        },
        {
            "@search.score": 3.6696305,
            "EventName": "Banglore vs Chennai",
            "EventDate": "2022-04-02T13:56:45.556Z",
        },
        {
            "@search.score": 3.5296328,
            "EventName": "Delhi vs Mumbai",
            "EventDate": "2022-04-01T13:56:45.556Z",
        }
    ]

Ex: 2 - Here's an example of a search with a specific keyword without setting up the Freshness Function

If we search by keyword mumbai, we expect the most recent events related to the search keyword to be on top instead of the outdated ones. But if we see the results the past event is on top of the list due to the default score.

Don't be confused if some of the documents don't have the mumbai keyword but still appear in the result set; because I used a $select clause to keep the output short. There may be a mumbai term in the Description or Venue, which you can check without using the $select clause.

"output": [
        {
            "@search.score": 1.3331333,
            "EventName": "Mumbai vs Kolkatta",
            "EventDate": "2022-04-05T13:56:45.556Z"
        },
        {
            "@search.score": 1.1071612,
            "EventName": "Kolkatta vs Delhi",
            "EventDate": "2022-04-06T13:56:45.556Z"
        },
        {
            "@search.score": 0.650463,
            "EventName": "Chennai vs Hydrebad",
            "EventDate": "2022-04-04T13:56:45.556Z"
        },
        {
            "@search.score": 0.5063205,
            "EventName": "Delhi vs Mumbai",
            "EventDate": "2022-04-01T13:56:45.556Z"
        },
        {
            "@search.score": 0.25316024,
            "EventName": "Mumbai vs Chennai",
            "EventDate": "2022-04-03T13:56:45.556Z"
        }
    ]

Now, If we fire the same query by setting up the Function, we get the desired result.

"output": [
        {
            "@search.score": 5.564825,
            "EventName": "Kolkatta vs Delhi",
            "EventDate": "2022-04-06T13:56:45.556Z"
        },
        {
            "@search.score": 4.818283,
            "EventName": "Mumbai vs Kolkatta",
            "EventDate": "2022-04-05T13:56:45.556Z"
        },
        {
            "@search.score": 2.8222241,
            "EventName": "Chennai vs Hydrebad",
            "EventDate": "2022-04-04T13:56:45.556Z"
        },
        {
            "@search.score": 0.96613073,
            "EventName": "Mumbai vs Chennai",
            "EventDate": "2022-04-03T13:56:45.556Z"
        }
        {
            "@search.score": 2.0536695,
            "EventName": "Delhi vs Mumbai",
            "EventDate": "2022-04-01T13:56:45.556Z"
        },
    ]

For a further demonstration, I've added one more document to the index, which has the following information. Now there are a total of 7 documents on the index.

{
            "id": "7",
            "EventName": "Mumbai vs Banglore",
            "EventDesc": "Another exciting cricketing game where Mumbai is unbeatable this season.",
            "EventDate": "2022-04-20T13:56:45.556Z",
            "Venue": "Mumbai"
 }

Ex: 3 - Boost for future date events.

If we search for banglore with the scoring profile set as above, we get the following set of results. However, we wanted the newly added banglore related event, which is on April 20th, to be at the top.

"output": [
        {
            "@search.score": 3.046002,
            "EventName": "Banglore vs Chennai",
            "EventDate": "2022-04-02T13:56:45.556Z"
        },
        {
            "@search.score": 2.1953745,
            "EventName": "Mumbai vs Banglore",
            "EventDate": "2022-04-20T13:56:45.556Z"
        }
    ]

However, as we all know, the freshness function is linked to the current date (15th April) and a previous date (current date - 15), i.e., 1st April. But what if we need to connect another end to a future date? In that scenario, we may use a Timespan object to specify a negative time interval of [-15] or according to your need.

With that said, check what happens if we rerun the query after adjusting the time interval to negative.

"output": [
		{
            "@search.score": 2.1953745,
            "EventName": "Mumbai vs Banglore",
            "EventDate": "2022-04-20T13:56:45.556Z"
        },
        {
            "@search.score": 3.046002,
            "EventName": "Banglore vs Chennai",
            "EventDate": "2022-04-02T13:56:45.556Z"
        }
    ]

Also, keep in mind that all scoring parameters indicated inside the scoring profile must be sent together with the query while using the scoring profile in a search query. There's currently no way of specifying a default scoring parameter value.

Key Takeaways‌

Freshness Function works only on the DateTimeOffset type field.
In one Scoring Profile object we can put many Functions.
We can use a mix of Weighted Field and Function in one Scoring Profile.

References:

How to add a Scoring Profile?
Similarity and Scoring in Azure Cognitive Search

How to Boost Documents Using Freshness Function in Azure Search?