Implement 'endpoint role' for service discovery

[knatsakis]

https://netdata-cloud.slack.com/archives/CSD2D3NTG/p1605874967006100

[cakrit]

@ilyam8 please plan this.

[ilyam8]

This is not WIP, because, i believe, i wasn't able to explain difference between endpoint and pod roles to @knatsakis. Prometheus has both endpoint and pod configured by default.

Let me clarify the difference and explain why i did only pod back then.

Difference from a user prospective:

• endpoint: apply prometheus.io/scrape: true annotation to a Service.
• pod: apply prometheus.io/scrape: true annotation to a Pod.

Prometheus discovery (and our) result are targets, they are used to create scrape configs.

---

• role endpoint

Find endpoints, create a target per address per port.

0 ~ $ kubectl get endpoints -n infra
NAME                                         ENDPOINTS                                                     AGE
corp-homepage                                10.1.2.12:80                                                  167d
corp-traefik                                 10.1.0.14:8000,10.1.2.13:8000,10.1.0.14:8443 + 1 more...      197d
crdb-cockroachdb                             10.1.0.24:26257,10.1.2.25:26257,10.1.3.25:26257 + 3 more...   189d
...

In most of the cases those endpoints point to pods. Not common case example: an external to k8s db you want to access from the cluster.

---

• role pod

Find pods, create a target per pod per container per port.

---

What we need is to get IP:PORT and additional info (pod name, namespace, container name, image name, etc) which we can use to identify the application (prometheus don't need any identification) - both roles end result is ~ same. endpoint role discovery looks for the pod backed by the endpoint (targetRef) and extracts all the info from the pod (same as pod discovery do).

Feature	Pod	Endpoint
Allows to find a contaienr IP/Port and metadata	yes	yes
prometheus.io/scrape: true annotation place	pod	service
Requires created Service	no, can discover right after you deploy Deployment/Daemonset/etc	yes
Allows to discover external to k8s resources	no	yes

[ilyam8]

I have bery brief expirience with k8s, but from what i understand pod role works perfect if you need to discover applications running in pods. Adding annotations to pods is not a problem at all, endpoint doesn't provides any benefits.