Sorting resources by search parameters not currently supported by the Android SDK

[Rkareko]

Context
There is a request to sort Medications by master list number (integer). This number is provided by the client and will need to be stored on the Medication resource.
At the moment the list of medications is displayed in a questionnaire using a drop down widget. An x-fhir-query expression is used to fetch the medications from the db.

{
  "url": "http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-answerExpression",
  "valueExpression": {
    "expression": "Medication",
    "language": "application/x-fhir-query"
  }
}

The questionnaire item that houses this drop down is of the type reference. The medications retrieved from the db are represented as shown below

  {
    "reference": "Medication/dc-med-00206501",
    "display": "Atorvastatin"
  }

Sample medication resource

{
    "resourceType": "Medication",
    "id": "dc-med-00206501",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2024-06-05T09:29:43.108+00:00",
        "source": "#258dc701ae615984"
    },
    "text": {
        "status": "generated",
        "div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><div class=\"hapiHeaderText\"> Atorvastatin </div></div>"
    },
    "identifier": [
        {
            "use": "official",
            "value": "00206501"
        }
    ],
    "code": {
        "coding": [
            {
                "system": "http://snomed.info/sct",
                "code": "1145419005",
                "display": "Atorvastatin"
            }
        ]
    }
}

Question 1
How should this master list number be stored?
Is there a need to store the serial number (00206501 in this example) as an identifier?
Suggested approach
Store it as a secondary identifier i.e

{
  "identifier": [
    {
      "use": "official",
      "value": "00206501"
    },
    {
      "use": "secondary",
      "value": "2"
    }
  ]
}

Question 2
How the sorting will be implemented

Suggested approach 1 LOE Large
Update the x-fhir-query expression to sort the medications by the identifier field .i.e

{
  "url": "http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-answerExpression",
  "valueExpression": {
    "expression": "Medication?_sort=identifier",
    "language": "application/x-fhir-query"
  }
}

The issue with this approach is that currently the Android SDk only supports sorting of number, string and date search param types as seen here.
Identifiers are indexed as tokens which is not supported.
We would need to determine how much effort would be required to add this support. This can be done on the android fhir fork as a proof of concept before eventually being moved to the Android SDK repo.
The advantage of this approach is that sorting with be handled at the database level and there aren't any changes to be made on fhircore.
Issues
When a resource has more than 1 identifier, the sort will be affected since all the identifiers are extracted to the TokenIndexEntity table with the same index_name

Suggested approach 2 - LOE Medium
Use the functionality provided by the rulesEngine sortResources function to sort the resources in the profile config.
After sorting is complete, the references would then be extracted from the resources and then loaded onto the questionnaire for pre-population.
The issue with this approach is that pre-population of drop down widgets is not currently supported.
Processing pre-population values of type reference is not currently handled as seen here. This will have to be implemented.
Issues

1. Potential performance hit. Two loops are required 1 for sorting the resources and another for extracting the references for each of the resources
2. Datatype complexity - Finding the right data type to be used for pre-population

Suggested approach 3 (Thanks @pld and @LZRS for the insights) - LOE Small
Add an extension to the Medication resource that contains the sort value. The value will be of type valueInteger

"extension": [
    {
      "url": "http://hl7.org/fhir/5.0/StructureDefinition/extension-medication-sort-value",
      "valueInteger": 1
    }
  ]

Add a custom search param in the configService.provideCustomSearchParameters() function.

    val medicationSortSearchParameter =
      SearchParameter().apply {
        url = "http://smartregister.org/SearchParameter/sort"
        addBase("Medication")
        name = SORT_SEARCH_PARAM
        code = SORT_SEARCH_PARAM
        type = Enumerations.SearchParamType.NUMBER
        expression = "Medication.extension.value"
        description = "Search the sort field"
      }

This sort (find appropriate name for this field) value would be extracted into the NumberIndexEntity as shown below.
Issues
Note that custom search parameters require a fresh install.

In the questionnaire update the x-fhir-query to use the custom search param added above i.e

{
  "url": "http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-answerExpression",
  "valueExpression": {
    "expression": "Medication?_sort=sort",
    "language": "application/x-fhir-query"
  }
}

This particular request has some tight timelines. For the short term the question is which approach should we invest time in so as to get this feature out.
For the long term we can share ideas on a more robust solution.

[pld]

Cool thanks. Some questions:

1. Do have an idea on the LOE required for the 2 approaches? Approach 2 seems like it would have lower LOE to me.
2. For approach #/1, if that value is being used for sorting, calling it "sort-order" or something specific vs "secondary" makes sense.
3. For approach #/1, if the sort is set to "identifier" how does it know which "identifier" to use for sorting?
4. For approach #/1, the only sorting numbers... issue, is there a field in the medication that stores a number value we can use instead of identifier? Looking at the resource I guess not, what if we added an extension, could we sort on that?
5. For approach #/2, do we not need to worry about the only sort by number issue here, can we sort tokens with the sortResources function?

[Rkareko]

2. For approach #/1, if that value is being used for sorting, calling it "sort-order" or something specific vs "secondary" makes sense.

Identifier.use is limited to usual, official, temp, secondary and old as defined here . In order to define that value as sort order we would have to add a type to that idenitifier i.e

 {
      "use": "secondary",
      "value": "2",
      "type": {
        "coding": [
          {
            "system": "https://smartregister.org/",
            "code": "sort-order",
            "display": "Sort order"
          }
        ]
      }
    }

3. For approach #/1, if the sort is set to "identifier" how does it know which "identifier" to use for sorting?

It seem that it is not possible to specify which identifier to use for sorting. All the values for the different identifiers are stored in the TokenIndexEntity table on different rows each having the same index_name which is identifier as shown below. The sort would be applied to all the values.

4. For approach #/1, the only sorting numbers... issue, is there a field in the medication that stores a number value we can use instead of identifier? Looking at the resource I guess not, what if we added an extension, could we sort on that?

See approach #/3 . This will leverage on use of an extension and custom search param.

5. For approach #/2, do we not need to worry about the only sort by number issue here, can we sort tokens with the sortResources function?

The sortResources take a fhirPathExpression which an be used to extract the sort value from the medication resource e.g Medication.identifier.where(use = 'secondary').value.toInteger() would extract the value 2 from

{
  "identifier": [
    {
      "use": "official",
      "value": "00206501"
    },
    {
      "use": "secondary",
      "value": "2"
    }
  ]
}

If the type has been added then the expression Medication.identifier.where(type.coding.code = 'sort-order').value.toInteger() would extract the value 2 from

{
  "identifier": [
    {
      "use": "official",
      "value": "00206501"
    },
    {
      "use": "secondary",
      "value": "2",
      "type": {
        "coding": [
          {
            "system": "https://smartregister.org/",
            "code": "sort-order",
            "display": "Sort order"
          }
        ]
      }
    }
  ]
}

The sortResources function supports sorting by string, integer, date, datetime, decimal types

[f-odhiambo]

Here are my 2 cents

1. Identifier
   Purpose: An Identifier represents a unique identifier for a resource or element. It is commonly used to assign and share key pieces of information like patient IDs, encounter numbers, or insurance policy numbers.

Key Characteristics:

• Represents a unique instance of a resource.
• Typically used for medical record numbers (MRN), Social Security numbers, etc.
• Always related to identifying specific real-world entities.
• Contains fields such as system (the authority or context of the identifier) and value (the actual ID or number).

Example: Meidcation number to uniquely identify them in a hospital’s system.

{
  "system": "http://hospital.org/medno",
  "value": "001"
}

2. Extension
   Purpose: An Extension is used to introduce additional information to a resource when the standard FHIR specification does not support that data. It allows for flexibility in cases where the base FHIR model needs to be extended for a specific use case.

Key Characteristics:

• Used for custom data not covered by core elements.
• Can represent any additional information that is not part of the standard resource definition.
• Must include a URL to define the context and meaning of the extension.
• Extensions can be simple (a single additional field) or complex (multiple fields).
• Example: A medication with a custom identifier to sort

Example

{
  "url": "http://example.org/fhir/StructureDefinition/medication-id",
  "valueString": "001"
}

Both Identifier and Extension can be used as search parameters in FHIR but in different ways. The identifier is a standard search parameter in FHIR. Extension requires custom search parameters to be explicitly defined to be searchable.

1. Identifier as a Search Parameter

GET [base]/Medication?identifier=http://hospital.org/mrn|001

2. Extension as a Search Parameter

Extension is not automatically a standard search parameter in FHIR. However, you can define custom search parameters for extensions when needed.
Usage: To search by extension, you typically need to define a custom search parameter in your FHIR server configuration, which will point to the extension element you want to search by. FHIR provides support for searching extensions via their URL.

GET [base]/Medication?extension=http://example.org/fhir/StructureDefinition/medication-id|001
Summary:

Sorting Locally using FHIR Path expression and custom code

1. Sorting by Identifier
   Pros: No new code, works best for Analytics if needed later on HAPI
   Con: Not properly indexed on SDK, needs update on SDK to work locally

2. Sorting by Extension
   Pros: We can reuse the current code we use to check Patient/Group is active and indexed and make it work locally with no SDK changes
   Con: New code on FHIR Core to work locally, does not work well for Analytics if we are relying on inbuilt HAPI search params and FHIR API endpoints

[Rkareko]

Sorting by Extension
Pros: We can reuse the current code we use to check Patient/Group is active and indexed and make it work locally with no SDK changes
Con: New code on FHIR Core to work locally, does not work well for Analytics if we are relying on inbuilt HAPI search params and FHIR API endpoints

Concenring the con above, the client has confirmed that the number is only used for sorting the medication list on the app

[ageryck]

@Rkareko for the short term depending on deliverable timelines, using the readily available sortfn will yield quicker but dirty result (There's the potential of performance hit especially depending on the list size to be sorted).
Approach #/1 sounds like the ultimate since it provides performance gains at DB level sort and a high potential for reuse to sort resources other than medication.

I therefore suggest quick and dirty for project timelines however we create a separate ticket/roadmap item to track the ultimate sort solution.

Another random question: other than identifier what other generic attribute can we use for sorting (looking beyond medication), if we combine the introduction of an extended attribute to sort and custom search param can that leverage DB level sort?

[Rkareko]

Approach #/3 employs the use of an extension and custom search param that leverages DB level sort

[ageryck]

well then this becomes the go-to option for now

[Rkareko]

There appears to be consensus on using the suggested approach #/3. The following tickets track the changes required to be made :-

• Add custom search parameter for Medication sort . This is required.
• Enhance the process of adding custom search parameters to be configurable . This is a nice to have for future use cases.
• Migrate existing data when new search params are added in order to avoid fresh app install . This is a nice to have especially for apps that are already in production.