You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As described in #640, our current OpenSearch implementation writes to both the SearchIdentifier table and the OpenSearch index.
Describe the desired state/solution
Eventually, we should refactor away, or otherwise reduce the scope of the SearchIdentifier table. To do so, we would need to either unify the data populated by the link checker and modified dates back into the Identifier table or limit SearchIdentifier to only include these or other necessary fields. These values must be preserved to recreate the OpenSearch index in the event of failure. Recreating relative to the table revision would also involve updating our reindexing logic/script relative to the revised table's final state, synthesizing data from both it and the Identifier table.
Additional notes
Our ability to undertake the above is dependent on the state of production monitoring for OpenSearch. We should act conservatively with regard to table changes until this is fully implemented. This will give us the most latitude with regard to falling back to the previous search implementation in the event of failure or unforeseen performance issues in production.
The text was updated successfully, but these errors were encountered:
adambuttrick
changed the title
[MAINTENANCE] Refactor SearchIdentifier table and OpenSearch index to reduce duplication
[MAINTENANCE] Refactor SearchIdentifier table and OpenSearch to reduce duplication
Jun 18, 2024
Describe the current state/issue
As described in #640, our current OpenSearch implementation writes to both the SearchIdentifier table and the OpenSearch index.
Describe the desired state/solution
Eventually, we should refactor away, or otherwise reduce the scope of the SearchIdentifier table. To do so, we would need to either unify the data populated by the link checker and modified dates back into the Identifier table or limit SearchIdentifier to only include these or other necessary fields. These values must be preserved to recreate the OpenSearch index in the event of failure. Recreating relative to the table revision would also involve updating our reindexing logic/script relative to the revised table's final state, synthesizing data from both it and the Identifier table.
Additional notes
Our ability to undertake the above is dependent on the state of production monitoring for OpenSearch. We should act conservatively with regard to table changes until this is fully implemented. This will give us the most latitude with regard to falling back to the previous search implementation in the event of failure or unforeseen performance issues in production.
The text was updated successfully, but these errors were encountered: