403 Client Error: Forbidden for url

[gregfriedland]

Hello,
I've installed kube-janitor as instructed: git clone, kubectl apply -f deploy/common/, then kubectl apply -f deploy/deployment/. However, I'm getting the following error in the pod logs. I'm not sure where 10.222.0.1 is coming from as there aren't any nodes in cluster (inlcuding masters) with that IP.
Any ideas?
Best,
Greg

(This says 0.6 but I've also tried master).

2019-08-06 05:39:49,669 INFO: Janitor v0.6 started with debug=True, delete_notification=None, dry_run=True, exclude_namespaces=kube-system, exclude_resources=events,controllerrevisions, include_namespaces=all, include_resources=all, interval=60, once=False, rules_file=/config/rules.yaml
2019-08-06 05:39:49,669 INFO: **DRY-RUN**: no deletions will be performed!
2019-08-06 05:39:49,688 INFO: Loaded 2 rules from file /config/rules.yaml
2019-08-06 05:39:49,697 DEBUG: Starting new HTTPS connection (1): 10.222.0.1:443
2019-08-06 05:39:49,714 DEBUG: https://10.222.0.1:443 "GET /api/v1/namespaces HTTP/1.1" 403 294
2019-08-06 05:39:49,715 ERROR: Failed to clean up: 403 Client Error: Forbidden for url: https://10.222.0.1:443/api/v1/namespaces
Traceback (most recent call last):
  File "/kube_janitor/main.py", line 51, in run_loop
    dry_run=dry_run)
  File "/kube_janitor/janitor.py", line 201, in clean_up
    for namespace in Namespace.objects(api):
  File "/usr/local/lib/python3.7/site-packages/pykube/query.py", line 148, in __iter__
    return iter(self.query_cache["objects"])
  File "/usr/local/lib/python3.7/site-packages/pykube/query.py", line 138, in query_cache
    cache["response"] = self.execute().json()
  File "/usr/local/lib/python3.7/site-packages/pykube/query.py", line 123, in execute
    r.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.222.0.1:443/api/v1/namespaces

[hjacobs]

The RBAC manifest clearly allows get/list on namespaces: https://github.com/hjacobs/kube-janitor/blob/master/deploy/common/rbac.yaml --- do you really use the latest repo master branch? (deployment.yaml shows version 0.7, not 0.6 like your output says)

[hjacobs]

I just deployed both deploy/common and deploy/deployment to a local kind (Kubernetes in Docker) cluster to test. It works:

./kubectl --kubeconfig ~/.kube/kind-config-kube-janitor-test logs deploy/kube-janitor | head
2019-08-06 08:11:33,926 INFO: Janitor v0.7 started with debug=True, delete_notification=None, dry_run=True, exclude_namespaces=kube-system, exclude_resources=events,controllerrevisions, include_namespaces=all, include_resources=all, interval=60, once=False, rules_file=/config/rules.yaml
2019-08-06 08:11:33,926 INFO: **DRY-RUN**: no deletions will be performed!
2019-08-06 08:11:33,930 INFO: Loaded 2 rules from file /config/rules.yaml
2019-08-06 08:11:33,932 DEBUG: Starting new HTTPS connection (1): 10.96.0.1:443
2019-08-06 08:11:33,938 DEBUG: https://10.96.0.1:443 "GET /api/v1/namespaces HTTP/1.1" 200 1144
2019-08-06 08:11:33,938 DEBUG: Skipping Namespace kube-system
2019-08-06 08:11:33,940 DEBUG: https://10.96.0.1:443 "GET /api/v1/ HTTP/1.1" 200 None
2019-08-06 08:11:33,942 DEBUG: https://10.96.0.1:443 "GET /api/v1/configmaps HTTP/1.1" 200 None
2019-08-06 08:11:33,943 DEBUG: Skipping ConfigMap kube-system/coredns
2019-08-06 08:11:33,943 DEBUG: Skipping ConfigMap kube-system/extension-apiserver-authentication

[gregfriedland]

Yes, that 0.6 was my attempt to try the 0.7 git tag to see if the "release" helped (not sure why it didn't report 0.7). I get the same result when I try master however. Yeah, I'm also not sure what's going on since the RBAC looks ok to my untrained eye.

[hjacobs]

@gregfriedland which cluster are you deploying to? Maybe the cluster has additional auth/restrictions (webhook, ..)?

[stevesea]

having the same issue in a GKE cluster -- 403 on trying to hit the k8s API for namespaces. RBAC looks right to me. also tried getting onto the pod and doing a curl manually to confirm

$ wget --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --no-check-certificate https://IPADDR:443/api/v1/namespaces
Connecting to IPADDR:443 (IPADDR:443)
wget: server returned error: HTTP/1.1 403 Forbidden

[craig410]

Had a similar issue with a 403 on GKE. I chose to install it in a 'kube-janitor' namespace. The ClusterRoleBinding https://github.com/hjacobs/kube-janitor/blob/master/deploy/common/rbac.yaml#L38 grants the 'kube-janitor' role to the 'kube-janitor' service account in the 'default' namespace. Changing this to point to the correct namespace account worked and resolved the issue for me.

[hjacobs]

@craig410 do you mind doing a PR for the README and the rbac.yaml to add appropriate comments for the next user?

[sryabkov]

@hjacobs Please see #57