TCP i/o timeout #16599

mazha20 · 2023-09-15T17:17:48Z

mazha20
Sep 15, 2023

hello everyone, i have clear installation etcd cluster with kubespray, but my cluster in unhealthly because server can't connect to other server in cluster with this error:
error":"read tcp 10.10.10.102:2380->10.10.10.101:43132: i/o timeout
i don't have iptables rules and i can connect from 10.10.10.102 machine to 10.10.10.101 2380 and 2379 port
etcd Version: 3.5.6
Go Version: go1.16.15

Answered by mazha20

Oct 6, 2023

We founded solution. Mtu on different nodes was wrong

View full answer

jmhbnz · 2023-09-19T19:35:10Z

jmhbnz
Sep 19, 2023
Maintainer

Hey @mazha20 - Thanks for your question, can you please provide the logs for your etcd instances so we can confirm cluster health and networking, thanks.

0 replies

mazha20 · 2023-09-23T15:12:14Z

mazha20
Sep 23, 2023
Author

@jmhbnz Thank you for the answer, i think we find problem with our network.

0 replies

mazha20 · 2023-09-28T15:57:36Z

mazha20
Sep 28, 2023
Author

@jmhbnz Hi, i reopened this discussion because I searched problem

I try to setup etcd cluster on Proxmox Cluster and when all etcd nodes works on one node of Proxmox - cluster is healthy. But, when i try to setup etcd cluster on different nodes of Proxmox cluster, i have the same error with tcp i/o timeout.
Also, i wrote problem in Proxmox Forum, because i don't know - this problem with Proxmox or etcd

{"level":"warn","ts":"2023-09-28T10:56:55.443-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51498","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.137:51498: read: connection reset by peer"}
2023-09-28T10:56:55.446562-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:55.443-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51610","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.137:51610: read: connection reset by peer"}
2023-09-28T10:56:55.446596-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:55.443-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:55106","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.106:55106: read: connection reset by peer"}
2023-09-28T10:56:55.446635-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:55.443-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51506","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.137:51506: read: connection reset by peer"}
2023-09-28T10:56:55.446669-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:55.443-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51710","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.137:51710: read: connection reset by peer"}
2023-09-28T10:56:55.446698-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:55.442-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:55368","server-name":"","error":"read tcp 10.10.10.112:2379->10.10.10.106:55368: read: connection reset by peer"}
2023-09-28T10:56:56.872874-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:56.872-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51936","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:51936: i/o timeout"}
2023-09-28T10:56:56.872999-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:56.872-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:51934","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:51934: i/o timeout"}
2023-09-28T10:56:57.469092-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:57.468-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57610","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57610: i/o timeout"}
2023-09-28T10:56:57.469250-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:57.469-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57608","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57608: i/o timeout"}
2023-09-28T10:56:58.739338-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:58.738-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57628","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57628: i/o timeout"}
2023-09-28T10:56:58.745260-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:58.745-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57644","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57644: i/o timeout"}
2023-09-28T10:56:58.750153-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:58.750-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50654","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50654: i/o timeout"}
2023-09-28T10:56:58.751220-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:56:58.751-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50652","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50652: i/o timeout"}
2023-09-28T10:57:01.147006-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:01.146-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57652","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57652: i/o timeout"}
2023-09-28T10:57:01.877375-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:01.877-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50658","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50658: i/o timeout"}
2023-09-28T10:57:01.877526-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:01.877-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50672","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50672: i/o timeout"}
2023-09-28T10:57:02.473688-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:02.473-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57670","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57670: i/o timeout"}
2023-09-28T10:57:02.473944-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:02.473-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:57660","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:57660: i/o timeout"}
2023-09-28T10:57:03.743815-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:03.743-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:51188","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:51188: i/o timeout"}
2023-09-28T10:57:03.749337-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:03.749-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.106:51196","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.106:51196: i/o timeout"}
2023-09-28T10:57:03.755028-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:03.754-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50684","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50684: i/o timeout"}
2023-09-28T10:57:03.758957-05:00 pxm-kube-etcd02 etcd[130581]: {"level":"warn","ts":"2023-09-28T10:57:03.758-0500","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.10.10.137:50696","server-name":"","error":"read tcp 10.10.10.112:2380->10.10.10.137:50696: i/o timeout"}

This is all logs after restart etcd service

0 replies

mazha20 · 2023-09-30T13:41:14Z

mazha20
Sep 30, 2023
Author

i found this error with etcdctl command

{"level":"warn","ts":"2023-09-30T08:39:51.849-0500","logger":"client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000454c40/10.10.10.112:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
https://10.10.10.112:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

0 replies

mazha20 · 2023-09-30T14:27:54Z

mazha20
Sep 30, 2023
Author

and this error with etcd cluster health command:
{"level":"warn","ts":"2023-09-30T09:25:23.818-0500","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000358a80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}

0 replies

mazha20 · 2023-10-06T21:51:06Z

mazha20
Oct 6, 2023
Author

We founded solution. Mtu on different nodes was wrong

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TCP i/o timeout #16599

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

TCP i/o timeout #16599

mazha20 Sep 15, 2023

Replies: 6 comments

jmhbnz Sep 19, 2023 Maintainer

mazha20 Sep 23, 2023 Author

mazha20 Sep 28, 2023 Author

mazha20 Sep 30, 2023 Author

mazha20 Sep 30, 2023 Author

mazha20 Oct 6, 2023 Author

mazha20
Sep 15, 2023

jmhbnz
Sep 19, 2023
Maintainer

mazha20
Sep 23, 2023
Author

mazha20
Sep 28, 2023
Author

mazha20
Sep 30, 2023
Author

mazha20
Sep 30, 2023
Author

mazha20
Oct 6, 2023
Author