BUG | Kubernetes provider won't get credentials refresh

[juanmatias]

Describe the Bug

When running a layer with a Kubernetes provider, this error arises randomly:

╷
│ Error: Invalid credentials
│ 
│   with kubernetes_manifest.mixed_routing_ingress,
│   on mixed-routing-ingress.tf line 1, in resource "kubernetes_manifest" "mixed_routing_ingress":
│    1: resource "kubernetes_manifest" "mixed_routing_ingress" {
│ 
│ The credentials configured in the provider block are not accepted by the
│ API server. Error: Unauthorized
│ 
│ Set TF_LOG=debug and look for '[InvalidClientConfiguration]' in the log to
│ see actual configuration.
╵

Expected Behavior

The Kubernetes resources are applied with no issue.

Steps to Reproduce

Steps to reproduce the behavior:

1. Go to a layer with Kubernetes provider
2. Run leverage tf plan or leverage tf apply
3. Some times this error happens (specially in a CI pipeline)

Screenshots

N/A

Environment (please complete the following information):

The error was seen in Gitlab CI pipelines.

Additional Context

As per these posts:

• v2.0.1 Authentication failures with token retrieved via aws_eks_cluster_auth hashicorp/terraform-provider-kubernetes#1131
• In-place update of aws_eks_cluster causes recreation of other resources hashicorp/terraform-provider-aws#10269 (comment)

And this Kubernetes API specification: https://kubernetes.io/docs/reference/config-api/client-authentication.v1/

...a possible cause is the token not being updated here:

  token                  = data.aws_eks_cluster_auth.cluster.token

If this is the case, it has to be confirmed, the possible solution is to use some way to force the token update, e.g. replace the token line with:

  exec {
    api_version = "client.authentication.k8s.io/v1"
    args        = [
      "eks",
      "get-token",
      "--cluster-name",
      data.terraform_remote_state.eks-cluster.outputs.cluster_id,
      "--region",
      var.region,
      "--profile",
      var.profile
    ]
    command     = "aws"
  }

So, the full provider definition is:

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  exec {
    api_version = "client.authentication.k8s.io/v1"
    args        = [
      "eks",
      "get-token",
      "--cluster-name",
      data.terraform_remote_state.eks-cluster.outputs.cluster_id,
      "--region",
      var.region,
      "--profile",
      var.profile
    ]
    command     = "aws"
  }
}

Today (2024-01-26) this is being tested in a client using binbash Leverage.