maya without etcd?

[aep]

mayastor uses etcd which appears to be very fragile.
is this mandatory? it doesnt appear to store much, so any k/v would work?

i can reliably take down the mayastor cluster by testing power failure events
eventually you end up with

panic: tocommit(423) is out of range [lastIndex(206)]. Was the raft log corrupted, truncated, or lost?

which appears top require manual intervention

see also #1318 (comment)

power failure is something that i'd prefer to recover from automatically.
could something like cockroachdb work?

[tiagolobocastro]

is this mandatory? it doesnt appear to store much, so any k/v would work?

We have an interface pstor which has currently only 1 implementation (etcd) so for the moment etcd is the option.
There's a backlog item for adding a pstor service proxy which every service will connect to and this service proxy will in turn be the one talking directly to etcd. This will make it easier to add other options, other than etcd.

panic: tocommit(423) is out of range [lastIndex(206)]. Was the raft log corrupted, truncated, or lost?

Quick search indicates etcd data was deleted, are you using persistent storage for etcd?

[inful]

Would it not be possible to use the NATS built in K/V capability (as it seems you are already depending on NATS)?

[aep]

I would volunteer to contribute that! Nats is battle tested in prod here.

Unfortunately currently blocked due to other issues with maya that I haven't had time to figure out yet

[tiagolobocastro]

Interesting, I was not aware NATS had a builtin kv store! Does it have leases as well?

[inful]

k3s are currently adding nats as an alternative to etcd (see k3s-io/k3s#7451). That work might serve as an inspiration :)

[dm3ch]

By the way can't be k8s api used as a KV storage to not to spin up separate KV storage?

[tiagolobocastro]

Not at the moment, we interact with etcd directly, which can't be done with k8s etcd.

[dm3ch]

Am I right to understand that it's possible in case of pstor driver creation?

Because I'm interested why to not use k8s as kv storage if mayastor always executed in k8s, where there's already working k8s API? :)

[tiagolobocastro]

Mayastor is actually not tied to k8s, which is why it cannot only reply on k8s api;
Today each service may talk to etcd directly, so it becomes rather messy to have to change each service to know different storage types.

[aep]

i'm hoping to be able to share a solution with nats soon, that is directly usable for mayastor.
~1 week maybe.

in the meantime this might help with regards to leases:
nats-io/nats-server#4803 (comment)