Backport of [NET-9500] Cleanup orphaned inline-certs and acl role/policy into release/1.2.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #4067 to be assessed for backporting due to the inclusion of the label backport/1.2.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@jm96441n
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul-k8s/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Changes proposed in this PR

• Add a goroutine on start up that loops and tries to clean up inline certs that are no longer referenced by any gateway and the original shared acl role/policy that is now specific to each gateway. Once all those are cleaned up the goroutine will exit and stop polling

How I've tested this PR

Run/Write tests

Manual Testing:

1. clone the following gist: https://gist.github.com/jm96441n/b92a80c2251c7a54d1feada320c5eac6
2. run the start.sh script in the gist (this will install consul 1.18-ent so ensure you have an ent license setup)
3. port forward port 8501 to the consul server instance
4. once the cluster is up run the following in another terminal:

export CONSUL_HTTP_ADDR="127.0.0.1:8501"
export CONSUL_HTTP_TOKEN=$(kubectl get --namespace consul secrets/consul-consul-bootstrap-acl-token --template={{.data.token}} | base64 -d)
export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_SSL_VERIFY=false

5. run consul acl role list and you will see the shared "managed-gateway-acl-role"
6. run consul acl policy list and you will see the shared "api-gateway-token-policy"
7. run consul acl binding-rule list and you will see the shared binding-rules referencing both managed-gateway-acl-role
8. run consul config list -kind inline-certificate and you will see an inline certificate listed
9. in the consul_values.yaml uncomment lines 3 and 4 and comment lines 4 and 5 (also make sure you build consul-k8s from this branch and have an up to date build of main of consul)
10. in the same terminal run export CONSUL_K8S_CHARTS_LOCATION="$HOME/hashi/consul-k8s/charts/consul" and replace the value being set with the location of the helm charts in your local copy of consul-k8s
11. run the following helm upgrade --install consul $CONSUL_K8S_CHARTS_LOCATION -f ./consul_values.yaml -n consul --create-namespace --wait to install the new version of consul-k8s that you built
12. force a re-reconciliation of your gateways (this should happen by default due to the change in dataplane version, if it does not you can delete and recreate the gateways)
13. run consul acl role list and you will not see the shared "managed-gateway-acl-role"
14. run consul acl policy list and you will not see the shared "api-gateway-token-policy"
15. . run consul acl binding-rule list and you will not see the shared binding-rules referencing managed-gateway-acl-role
16. run consul config list -kind inline-certificate and you will see no inline certificates listed

How I expect reviewers to test this PR

read the code
run the tests
do the above steps

Checklist

• Tests added
• CHANGELOG entry added

---

Overview of commits

• 74ee754 - eed31ad - 1eeeec4 - 120d2c8 - 743d568 - 49339cd - 8a932cb - 26a77f1 - e564c7a - 474afbe - 386c11d - e97a47c - 00125a5 - 6c2a380

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

1 out of 2 committers have signed the CLA.

• jm96441n
• temp

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[jm96441n]

already backported, closing #4133