Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database or Era files corrupt on Mainnet #6059

Open
jakubgs opened this issue Mar 11, 2024 · 0 comments · Fixed by #6164
Open

Database or Era files corrupt on Mainnet #6059

jakubgs opened this issue Mar 11, 2024 · 0 comments · Fixed by #6164

Comments

@jakubgs
Copy link
Member

jakubgs commented Mar 11, 2024

Describe the bug
Recently we've been seeing errors like this on Mainnet nodes, in this case libp2p-01 on linux-07.ih-eu-mda1.nimbus.mainnet:

{"lvl":"FAT","ts":"2024-03-11 11:20:37.394+00:00","msg":"Era summaries don't lead up to backfill, database or era files corrupt?","bid":"c78a68b6:8518954"}

A previous instance of this happening was with unstable-02 on linux-03.ih-eu-mda1.nimbus.mainnet:

{"lvl":"FAT","ts":"2024-03-08 08:11:18.637+00:00","msg":"Era summaries don't lead up to backfill, database or era files corrupt?","bid":"bd80c153:8503305"}

Which was purged and allowed to resync. The libp2p-01 was left in a broken state for debugging.

Additional context
The unstable-02 was running v24.2.2-3b4b1c-stateofus when issue was discovered(probably).
The libp2p-02 was running v24.2.2-1fef38-stateofus when issue was discovered.

No issues were identified by verifyEra command.
etan-status added a commit that referenced this issue Apr 2, 2024
@etan-status
allow frontfilling finalized CP block from era file
6726baf 
Add support for using era file for the initial checkpoint block.
This should also avoid an error when the beacon node is restarted
before the backfill process has made any progress (#6059).
@etan-status etan-status mentioned this issue Apr 2, 2024
Merged
@etan-status etan-status linked a pull request Apr 2, 2024 that will close this issue
allow frontfilling finalized CP block from era file #6164
Merged
etan-status added a commit that referenced this issue Apr 10, 2024
@etan-status
allow frontfilling finalized CP block from era file (#6164)
9bffc51 
Add support for using era file for the initial checkpoint block.
This should also avoid an error when the beacon node is restarted
before the backfill process has made any progress (#6059).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

allow frontfilling finalized CP block from era file status-im/nimbus-eth2
1 participant
@jakubgs