Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

K8s Clustering Leader Receives Context Deadline Error on Target #393

Closed
EntainAustralia opened this issue Mar 11, 2024 · 5 comments
Closed

K8s Clustering Leader Receives Context Deadline Error on Target #393

EntainAustralia opened this issue Mar 11, 2024 · 5 comments

Comments

@EntainAustralia
Copy link

EntainAustralia commented Mar 11, 2024 •
edited by karimra
Loading

One of our two targets receives a context deadline error very quickly (every 0.1 second). Can this rate be reduced somewhere or can we change something else to fix it?

Config

    log: true

    loader:
      type: file
      path: /app/targets-config.yaml
      interval: 30s
      enable-metrics: false

    clustering:
      cluster-name: cluster1
      targets-watch-timer: 30s
      locker:
        type: k8s
        namespace: gnmic
        lease-duration: 30s
        renew-period: 
        retry-timer: 30s
        debug: true

    subscriptions:
      interfaces:
        prefix:
        target:
        set-target:
        paths:
          - interfaces/interface/state/oper-status
        models: []
        mode: STREAM
        stream-mode: SAMPLE
        encoding:  PROTO
        qos:
        sample-interval: 10s
        heartbeat-interval:
        suppress-redundant:
        updates-only:
        outputs:
    outputs:
      prom:
        type: prometheus
        listen: "0.0.0.0:9804" 
        path: /metrics 
        expiration: 60s 
        metric-prefix: "gnmic" 
        append-subscription-name: false 
        override-timestamps: false
        export-timestamps: false 
        strings-as-labels: true
        cache: 
        timeout: 10s
        debug: true 
        add-target: 
        target-template:
        event-processors:
        num-workers: 1

Logs

2024/03/11 02:18:42.401237 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:42.502200 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:42.602368 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:42.703191 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:42.803523 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:42.904003 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.004392 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.105292 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.205649 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.306297 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.406453 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.507565 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.607855 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.708056 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.808335 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:43.909354 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.009764 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.109974 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.210221 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.310872 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.411229 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.512322 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.612520 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.713079 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.813434 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:44.913785 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:45.014109 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:45.115067 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
2024/03/11 02:18:45.215465 [gnmic] failed getting value of {{ TARGET NAME }}: client rate limiter Wait returned an error: context deadline exceeded
@karimra
Copy link
Collaborator

karimra commented Mar 11, 2024

This might be a bug, gNMIc is trying to list all leases with a context that already reached its deadline. The 0.1 seconds is an internal retry timer, changing it won't solve this problem.
Can you share which version of gNMI you are running ?

@Monkman08
Copy link

Seeing this on 0.36.2, This also could be a misconfiguration due to the limited documentation around the k8s locker. We configured the k8s locker based on a comment in karimra/gnmic#560 (comment)

@EntainAustralia
Copy link
Author

We are on version 0.36.2

version : 0.36.2
 commit : a7844a6d
   date : 2024-03-05T20:10:26Z
 gitURL : https://github.com/openconfig/gnmic
   docs : https://gnmic.openconfig.net

@EntainAustralia
Copy link
Author

The solution was to make sure the ENV variables were configured correctly. These were removed originally because they were not documented or explained in any way.

            - name: GNMIC_CLUSTERING_INSTANCE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: GNMIC_CLUSTERING_SERVICE_ADDRESS
              value: "$(GNMIC_CLUSTERING_INSTANCE_NAME).gnmic-svc.gnmic.svc.cluster.local"

@karimra
Copy link
Collaborator

karimra commented Mar 12, 2024

The ENV variables follow a naming pattern explained here: https://gnmic.openconfig.net/user_guide/configuration_env/#configuration-file-to-environment-variables-mapping

GNMIC_CLUSTERING_INSTANCE_NAME=XXX and GNMIC_CLUSTERING_SERVICE_ADDRESS=YYY are equivalent to:

clustering:
  instance-name: XXX
  service-address: YYY

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Monkman08 @karimra @EntainAustralia