Advanced search (Remote search)

[allan-on]

Context

This is the ability to search for Resources from the Hapi FHIR server i.e. a remote search

Example

For instance, on registering new patients, we might need to search whether the patient already exists to get their data and related resources before service provision for instance before conducting a screening.

Requirements

• UI/UX design to facilitate specification of the Resource and attributes combinations to search on the server
• Search functionality:
  • Search Resource by attribute e.g identifier
  • Get the Related resources e.g For patients, $everything
  • Search Partial resources

Grey areas

• Handling the Resource Practitioner, Team, and Location assignments updates once retrieved (moving to catchment area)

Additional context

• There's an SDK feature request to have this as part of the Google FHIR SDK as documented here Allow remote FHIR search through FHIR engine #1912

[allan-on]

Hi @rowo @HenryRae We'll need help with the User Interface and Experience design for this feature

[f-odhiambo]

Shared Mocks

[pld]

Not sure about "Globally" as the label, why note use "Remote search" and "Remotely"?

What happens if someone is not connected to the internet? Does the "Search Globally" link

• not show at all
• or showed greyed with an info box
• or show as normal but when clicked there's an error toast?

can we handle this case in the mocks

[HenryRae]

• do want the global search to work offline? @pld @rowo

[pld]

no that's not possible, it's not a global search it's a search through the remote records on the remote FHIR API, so by definition it can only function while online

[rowo]

Some context on the above, ZEIR's version of advanced search (screen capture on Drive):

• Since the register search only supported a rudimentary search by surname, first name, or ID, we had an advanced search that could search both locally and out of area. The idea is the patient/client might not know what information was entered, so you might need to search even locally in a facility setting with a lot of people by information beyond name and ID. I have not heard this happening too much since then, so the idea of using this kind of advanced search locally might not be something we need to support.
• Accessing this search came from a button that was easy to access. This is because in the area of ZEIR, having out of area people would be fairly common. In this case, I don't remember clearly, but the health workers would record the services done, but not necessarily create a medical record because the expectation is they wouldn't be tracking that person for long. Not sure about the use case in this situation in terms of ease of access.
• To support local and out of area search, the search form has a selector between my clinic vs. out of area, with out of area requiring a connection to function. I want to say that option was disabled if there were no connection available, but maybe it just threw a notification.
• There were limits to what we knew about search results. And there were permissions issues to deal with once you found someone from out of area. For someone out of area, you could only record information on out of area forms. To do more (or see their medical record/profile), you had to move them to your area (which removed them from the other person's client list). See below screens:
  
  
• The labeling in the mockups were just placeholder since the functionality wasn't clear. I don't think "remote search" is a common term — probably would need to see what makes sense in the specific scenario, which I'm not clear on. You should ask the client if the users would say it a certain way, or how locals describe it currently.

I don't know if any of the above changes work for this project, but there are some undefined details around record ownership/permission and technically what can appear at different times in the workflow in search result data and offline/online detection, especially if we're not building anything new.

[pld]

Thanks @rowo this approach makes sense, but the scope in the description and discussions w/partner so far for this issue centered around the remote part.

This would expand the scope of the issue, but it does feel like the better way to handle it anyhow. "Advanced search" with the selector for my/out is also very clear naming.

@allan-on -- I wonder if we can use the "advanced search" functionality here to cover the scope of filtering? That would justify this, but maybe we want to do it regardless. The other thing to follow-up on is what happens when you find a patient with the remote search? We talked with the partner about (1) moving the patient over, like Roger mentioned but this would add new scope for removing patients from other apps (that's ok, we need this feature anyhow) and (2) adding the patient to a Practitioner specific group so they exist in 2 places, wherever they are originally and for this specific Practitioner. (2) probably seems easier. The UI would be the same (or very similar) but with text saying "add" instead of "move".

[HenryRae]

1. During sync by 2 different apps(incase patient can be duplicated on another device), which HW device would be considered as source of truth?
2. Is there a possibility of merging patient data?
3. can the merged data be identified by who uploaded which data?
4. Does the user need to know who published which data?

[pld]

1. I think right now we take the most record that with the most recent changed date, if that device syncs, then a device with an older date syncs, the older devices data will be over-written. This seems fine. It would be nice to keep a copy of the 2nd devices changed data on the server, before over-writing it, so we can add merge on top of that in future work.
   2, 3) Not in scope
2. I don't think so

If we're showing a patient in 2 places, the main assumption here has to be that the patient data will not be changing, or if it those change it's ok for 2 separate changes to conflict and to choose the latest and overwrite the older ones. This seems reasonable because the main use will be adding other data (screening encounters) to a patient, and those will not conflict because they are separate records.

[Gental-Giant]

@rowo @pld I believe the above applies for for WDF as we would like to do the following:

• Be able to filter the patients in the register by a dynamic field, e.g. see all patients for a subset of the GNs I am assigned to (Filter by specific location where a pratiner is assigned to multiple)
• Execute multi-field search, e.g. show all Patients with this name and phone number, by specifying those in different fields
• Filter through registers for patients at risk
• Filter through registers for patients with a follow-up Task

[allan-on]

Hi, @pld Regarding option (1),

Moving the patient over

We would need to create a service on the server and in the app to remove Patient records, similar to how we previously managed Events/Clients in OpenSRP v1. However, this would significantly increase the LOE.

I agree with the idea proposed in (2),

Adding the patient to a Practitioner-specific group so that they exist in two places: wherever they were initially and for that specific Practitioner.

Since we're dealing with duplicates, we should link them as described in Duplicate Patient records.

• This involves using a link of type replaced-by to mark the record containing the link as a duplicate and point to a record that should be used instead.

If there are any related Resources such as Encounters, Observations etc, we can update the metadata for those Resources when saving them locally

[pld]

I think instead of updating the metadata we'd rather refactor the code to use the relationships and ignore the metadata

[pld]

Yea on dedupe that should be handled separately, so we'd have to replace the patients automatically and potentially lose data (which seems reasonable in this use case if we have versioning) then not create duplicates in the first place, or where multiple patients are created off line dedupe would need to be manual for now. @dubdabasoduba in the web UI can you delete patients?

[allan-on]

From discussions with @dubdabasoduba yesterday, it is not currently possible to delete patients from the Web UI

[allan-on]

Looks like we're almost aligned on use cases supported by this feature.
We haven't clarified the resources to fetch during the search. The options are:

1. Keeping it simple and just getting the Patient Resource
   This might complicate workflows tied to additional Resources such as Households (Groups) but suffices for WDF where clients (Patients) are not tied to households

2. Getting the Patient Resource and all or specific related Resources using $everything or only including the specific ones with _include/_revinclude. For instance, it might be necessary to get the Patient and their CarePlans, Tasks etc to facilitate new interactions with the current practitioner.

Any thoughts on the above based on possible use-cases to be supported on the product for most of the projects?
@pld @dubdabasoduba @f-odhiambo

[pld]

I think we have to go with (2.), not sure of the scope of related resources, like not getting families should be OK, but not getting obs and encounters will break things related to any FHIR Path queries that assume those resources are synced (which is a reasonable assumption), this would lead to behavior where the profile for a patient on one device would not look the same as the profile for the same patient on another device (because e.g. the upcoming tasks are not synced).

What's the downside to using $everything? That feels like the simplest solution

[allan-on]

Yeah, using $everything is the simplest solution. No significant downside really.
The filtering alters the size of the payload but it wouldn't be significant since the details will be related to a single Patient/Client record and we'll definitely need to avoid breaking any queries as well as have similar profile views after syncing to the device

[pld]

Based on the last conversation we decided we will split this feature into 1) the remote search part, which renders a Questionnaire for the search fields, then submits a query to the API, fetches a list of data based on the query and displays a list on the device of the data returned. This data cannot be interacted with. This would be generic to remote search across any resource, we'd define the resource in the Questionnaire, the rendering of results will be defined by a Register config. Next step here is to write up an implementation plan as an new issue, @allan-on when can we expect that?

And
2) Managing the interaction with the search results. This needs more scoping discussion, including whether you can view more details on the returned results or submit data for them without moving (or alternatively adding) them to the user's device

[allan-on]

For v1, we'll work with this design @HenryRae :

Defined workflow

1. Local search - As currently exists in the app.
   a. This would filter local records on that register.
2. If there aren't any results from the local search, there should be an option to either:
   a. Add/Register a new client
   b. Search online
3. Searching online launches a Questionnaire as mentioned in the above discussion.
4. Results of the search query will be shown in a register as mentioned in the above discussion.

The rest of the requirements will be defined from:

2. Managing the interaction with the search results. This needs more scoping discussion, including whether you can view more details on the returned results or submit data for them without moving (or alternatively adding) them to the user's device

[HenryRae]

which register will the global search check for results?
is it all registers? main register with all clients i.e All clients register or HH register?
@pld cc: @f-odhiambo

[HenryRae]

if we have this type of search field,

if CHA search by any number e.g 487,

1. are we able to query in PHNs, Phone numbers and all types of IDs we may have?
2. and show matching results based on PHN, Phone number, and different IDs?
   cc: @pld @f-odhiambo @dubdabasoduba

[pld]

I believe we discussed this on the last call and said we should be able to do that, @allan-on you can let us know if this seems correct.

You would do a disjunctive query for whatever is entered in either of those fields, so like if someones PHN no is someone else phone no you'd return both people

[pld]

on which registers, that's flexible, it can be constrained specific registers or be overall patient data, we can think of these as hard-coded pre-filters that we're adding the search query, like within this subset, search for things with these attributes, but how that subset is defined is flexible, and not constrained by where the search originates (but it can be if we want to)

[HenryRae]

Okay great thanks for the clarity, my next items to work with are clear now.

[HenryRae]

We defined some sorting for search results:

1. Alphabetically: Arrange search results in alphabetical order, which is helpful when users are looking for specific items or content in a structured manner.

2. Last modified: Sort results by the date of publication or creation. This option is beneficial when users are looking for the most recent information or want to see the latest updates first.

3. Best matching: Use a proprietary algorithm or machine learning to determine the "best match" based on various factors like user behavior, historical data, and contextual relevance.

4. Relevance: Sort search results based on relevance to the user's query. This option prioritizes items that best match the search terms and user intent, providing the most accurate and useful results first.

Question: which options are we able to add and can we have sort options configurable(if we can have more than 1 implemented)?

[pld]

I would only be confident we can do (1) now.

We might be able to do a simple version of (2) but it would be based on the time last synced to server, which is not really the same as "last modified" date. To do a more complex version we could get the most recent encounter and use the date on that as last modified, but that would add complexity, reduce performance, and maybe require an separate micro-service or plug-in to the info gateway if we wanted to do it server side.

(3) and (4) are in the same category as the more complex version of (2), we can do it but it would need a new round of R&D and scoping and would require new pieces of tech. One longer-term way to do this would be to attach a search-indexer to our fhir-repo and then run these queries against the search indexer, there are project like https://lucene.apache.org/ for generic document search indexing, which would have 2, 3, and 4 built into it. We can discuss this with the Google team, it hasn't come up that that I'm aware of, it's big project.

[HenryRae]

Okay, for the first implementation we will do 1, I’ll put together some ideas for future work for further revision on design team side if we ever want to go with the versions termed as complex.

[HenryRae]

Here's a link the the search feature update.
Have one search field for all search inputs, and one search results page. Inform users through toasts incase of "no internet connection" or incase remote results couldn't be loaded for some reason.

The online search will only be initiated when the search or enter button is hit on the keyboard. Before the CTA is hit, we can show/filter out local results as user is typing.

Search results:

• The idea is to merge both local and remote search and results in the same flow i.e when user makes a search, we query both local database and remote database. The results page with show results from both local and remote search.
• Default to the local search result, for cases were same patient/HH data is also available online. thus consider local database as most upto date source.

Sort:

• the results will be sorted alphabetically based on the patient/HH name.

Other configurations:

• Have/Not have remote/online search is based on project requirements. i.e projects that don't need remote search will only show local results always, and those that require this feature will show both.
• Registers to query from should be flexible i.e the search can be constrained to specific registers or be overall patient data based on the project. reference
• For V1, only show the online results, and shouldn’t be actionable thus remote results will have "move to catchment CTA but disabled"
• won’t always show user input e.g incase they search by ID, in the search result based on the register, but in future we would want to highlight user search input in the results.

Further documentation on how the feature would work is included in the designs
cc: @allan-on @Gental-Giant

[pld]

Cool this looks good, for the "move to catchment" we should have text be set through the configs

[dubdabasoduba]

@allan-on

@qiarie brought up a good point on whether we would want to add a configuration to decide how deep the search goes. In some cases, we might want to search within the location where the user is assigned. This is mostly for apps syncing using Teams or CareTeams.

The other option is to search the whole DB.

[dubdabasoduba]

This is definitely not in scope right now but we can consider it as an enhancement later.

[pld]

Yeah agree good point, this is both modulo users desire and  permissions they haveOn Oct 6, 2023, at 07:28, Benjamin Mwalimu ***@***.***> wrote: This is definitely not in scope right now but we can consider it as an enhancement later. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>