application remains in container creating state with fsck errors when container runtime service is stopped

[mittachaitu]

Description

• Deployed the application with liveness probe(it will create a file and delete filed) and wrote some data on a volume using the dd command.
• Performed sync operation to flush IOs.
• Stoped contained service for 5min on node where application pod was scheduled.
• Post recovery from chaos i.e starting contained service application pod remains in a container creating a state with fsck errors.

Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Warning  FailedMount  30m (x7 over 31m)  kubelet            MountVolume.MountDevice failed for volume "pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf" : rpc error: code = Internal desc = Volume pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf still mounted on node d2iq-node4
  Normal   Scheduled    27m                default-scheduler  Successfully assigned cri-failure-csi-cstor/app-busybox-7946c9758c-9662g to d2iq-node1
  Warning  FailedMount  11m (x2 over 22m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[data-vol], unattached volumes=[default-token-4xbpg data-vol]: timed out waiting for the condition
  Warning  FailedMount  10m (x6 over 30m)  kubelet            MountVolume.MountDevice failed for volume "pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf" : rpc error: code = Internal desc = 'fsck' found errors on device /dev/disk/by-path/ip-10.0.8.159:3260-iscsi-iqn.2016-09.com.openebs.cstor:pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf-lun-0 but could not correct them: fsck from util-linux 2.31.1
ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/sdp is mounted.
/dev/sdp: Superblock needs_recovery flag is clear, but journal has data.
/dev/sdp: Run journal anyway

/dev/sdp: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
           (i.e., without -a or -p options)
  Warning  FailedMount  8m51s (x8 over 29m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[data-vol], unattached volumes=[data-vol default-token-4xbpg]: timed out waiting for the condition
  Warning  FailedMount  41s (x8 over 29m)    kubelet  MountVolume.MountDevice failed for volume "pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf" : rpc error: code = Internal desc = rpc error: code = Internal desc = object is being deleted: cstorvolumeattachments.cstor.openebs.io "pvc-71f1ac8d-2984-4090-82c2-17b16f86fbdf-d2iq-node1" already exists

Pipeline link: openebs.ci