Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MAINTENANCE] Refactor SearchIdentifier table and OpenSearch to reduce duplication #655

Open
adambuttrick opened this issue Jun 18, 2024 · 0 comments
Labels
maintenance Routine or one-off maintenance or clean-up task

Comments

@adambuttrick
Copy link

adambuttrick commented Jun 18, 2024

Describe the current state/issue

As described in #640, our current OpenSearch implementation writes to both the SearchIdentifier table and the OpenSearch index.

Describe the desired state/solution

Eventually, we should refactor away, or otherwise reduce the scope of the SearchIdentifier table. To do so, we would need to either unify the data populated by the link checker and modified dates back into the Identifier table or limit SearchIdentifier to only include these or other necessary fields. These values must be preserved to recreate the OpenSearch index in the event of failure. Recreating relative to the table revision would also involve updating our reindexing logic/script relative to the revised table's final state, synthesizing data from both it and the Identifier table.

Additional notes

Our ability to undertake the above is dependent on the state of production monitoring for OpenSearch. We should act conservatively with regard to table changes until this is fully implemented. This will give us the most latitude with regard to falling back to the previous search implementation in the event of failure or unforeseen performance issues in production.

@adambuttrick adambuttrick added the maintenance Routine or one-off maintenance or clean-up task label Jun 18, 2024
@adambuttrick adambuttrick changed the title [MAINTENANCE] Refactor SearchIdentifier table and OpenSearch index to reduce duplication [MAINTENANCE] Refactor SearchIdentifier table and OpenSearch to reduce duplication Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Routine or one-off maintenance or clean-up task
Projects
None yet
Development

No branches or pull requests

1 participant