[Bug]: two cn were hung during sysbench 10w benchmark test on daily regression on tke

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

1.2-dev

Commit ID

2ab791a

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
3*LOG: 3C 7G
- OS type:
- Others:

Actual Behavior

job link:
https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9820003789/job/27118972412

during sysbench 10w benchmark test on daily regression, two cns can not be connected, and it caused the test was hung for serveral houses

goroutines:
cn_hung.tar.gz

and there are lots of the following errors:

metrics:
https://grafana.ci.matrixorigin.cn/d/c69513f9-b211-40f5-a35c-c7ce5100abc2/txn-metrics?orgId=1&from=1720302727000&to=1720319405000&var-interval=1m&var-namespace=mo-branch-nightly-2ab791abf-20240706&var-pod=All

mo-log:
https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22GDT%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-2ab791abf-20240706%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221720303059347%22,%22to%22:%221720307671168%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

run daily regression on tke

Additional information

No response

[volgariver6]

@m-schen please help look into it. This issue has not reappeared after reverting the PR（#17362）.

[volgariver6]

fixed

[aressu1985]

fixed