External metric not working for Eventhubs #90

Vishal2696 · 2020-02-27T09:32:20Z

Describe the bug
I've setup the metric adapter successfully and have established external metrics for services such as service bus, application gateway. But however When I tried to setup one for eventhub for the metric "IncomingMessages" I was displayed with an error "Error from server (ServiceUnavailable): the server is currently unable to handle the request"

Below is my external metric YAML.

apiVersion: azure.com/v1alpha2
kind: ExternalMetric
metadata:
  name: myextmetric
  namespace: mynamespace
spec:
  type: azuremonitor
  azure:
    resourceGroup: myrg
    resourceName: evhubnamespacename
    resourceProviderNamespace: Microsoft.EventHub
    resourceType: namespaces
  metric:
    metricName: IncomingMessages
    aggregation: Total
    filter: EntityName eq 'customname'

The text was updated successfully, but these errors were encountered:

jsturtevant · 2020-03-08T16:19:50Z

related to #84

Vishal2696 · 2020-03-08T16:28:41Z

related to #84

May I know how is this related to #84 ?? Kindly read my description, I've setup external metrics for other services and they are working fine. Now in the same cluster using the same adapter I'm unable to implement using event hubs. As per your suggestion if launching in different namespace was the problem nothing should have worked for me. I would like to have this issue reopened as I feel there has been a misunderstanding. I've even raised a case with Azure support regarding this and even they were not sure what the problem is. My case is still open there too

jsturtevant · 2020-03-09T03:01:51Z

The error reported above "Error from server (ServiceUnavailable): the server is currently unable to handle the request" was the exact same error message as #84 which was resolved by making sure everything was properly namespaced.

Happy to help resolve if it is not related. Please be mindful of our contributing guidelines and code of conduct.

Could you provide the logs of the metric adapter (specifically after a request for the eventhubs metric was queried) and the following info:

Kubernetes version (kubectl version):

Running on AKS

Logs (kubectl logs <metric adapter pod id>)

Additional context
kubectl describe hpa

Adapter version:

Vishal2696 · 2020-03-09T08:26:21Z

@jsturtevant
I've corrected my last reply according to the code of conduct. Sorry for that. Below are the details you asked for.

Adapter Version: 0.5.0
Kubernetes Version: 1.14.8
Yes, The cluster is running on AKS

Logs from metric-adapter Pod:

I0303 06:05:07.749429 1 controller.go:149] adding item to queue for 'mynamespace/extmetric03' with kind 'ExternalMetric'
I0303 06:05:07.754483 1 handler.go:82] processing item 'extmetric02' in namespace 'mynamespace'
I0303 06:05:07.754607 1 handler.go:111] adding to cache item 'extmetric02' in namespace 'mynamespace'
I0303 06:05:07.754625 1 controller.go:134] succesfully proccessed item '{mynamespace/extmetric02 ExternalMetric}'
I0303 06:05:07.754634 1 controller.go:91] processing next item
I0303 06:05:07.754640 1 controller.go:98] processing item
I0303 06:05:07.754579 1 handler.go:82] processing item 'extmetric01-eventhub' in namespace 'mynamespace'
I0303 06:05:07.754729 1 handler.go:111] adding to cache item 'extmetric01-eventhub' in namespace 'mynamespace'
I0303 06:05:07.754737 1 controller.go:134] succesfully proccessed item '{mynamespace/extmetric01-eventhub ExternalMetric}'

In the logs, the external metric "extmetric01-eventhub" is the one using event-hubs. You can safely ignore others as they use other Azure services and they are working fine too.

When I enter the command kubectl get hpa I'm getting the same error "Error from server (ServiceUnavailable): the server is currently unable to handle the request"

jsturtevant · 2020-03-10T16:12:14Z

So it looks like the HPA is not able to find the metric adapter. It is interesting that it is working for the other metrics in the cluster.

A few thoughts:

Is this HPA and External metric in a specific namespace? Is extmetric02 working as expected? it looks like it is in the same namespace.
Is it possible there is RBAC or networking policy is blocking access to the api server?
Can you share the HPA definition that is pointing at the external metric for event hub?

Can you hit the metric adapter directly using the following command:

kubectl  get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/extmetric01-eventhub"

jsturtevant · 2020-03-10T20:05:37Z

Another thing to check is that the metric adapter is registered:

kubectl describe apiservices v1beta1.external.metrics.k8s.io

jsturtevant · 2020-03-10T23:44:46Z

I was able to track down the issue. It appears that IncomingMessages doesn't return any values through the Metric API at the Eventhub level which is where you are filtering with filter: EntityName eq 'customname'. I am looking into why that metric is not being reported at the EventHub level. It is reported at the Namespace level.

The reason you are getting the Error from server (ServiceUnavailable): the server is currently unable to handle the request message is because the code for the Metric API checks for null but not length so end up with the following exception:

    1 monitor_client.go:50] resource uri: /subscriptions/b9d9436a-0c07-4fe8-b779-2c1030bd7997/resourceGroups/sb-external-example/providers/Microsoft.EventHub/namespaces/samplejjs
E0310 20:36:05.708112       1 runtime.go:71] Observed a panic: runtime error: index out of range
goroutine 62133 [running]:
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc0004db560)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0xda
panic(0x1701480, 0x2bdaa40)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics.extractValue(0xc0004f7f30, 0x10, 0xc000580bd0, 0x24, 0xc0004f7f40, 0xc, 0xc0004f7f13, 0x9, 0xc000667900, 0x12, ...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics/monitor_client.go:81 +0xd56
github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics.(*monitorClient).GetAzureMetric(0xc000530460, 0xc0004f7f30, 0x10, 0xc000580bd0, 0x24, 0xc0004f7f40, 0xc, 0xc0004f7f13, 0x9, 0xc000667900, ...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/azure/externalmetrics/monitor_client.go:60 +0x4a0
github.com/Azure/azure-k8s-metrics-adapter/pkg/provider.(*AzureProvider).GetExternalMetric(0xc00052e600, 0xc0000d6491, 0x7, 0x1c15360, 0xc000530320, 0xc0000d6499, 0xf, 0x3860f5559f5, 0xc00cfdb8d9, 0xcfdb8d900000000)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/pkg/provider/provider_external.go:41 +0x4f1
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/external_metrics.(*REST).List(0xc000593ab0, 0x1c04140, 0xc0001aac30, 0xc00037b580, 0x1918386, 0x2, 0x1bd9340, 0xc00037b580)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/registry/external_metrics/reststorage.go:79 +0x209
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x1bf68c0, 0xc00000e368, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:268 +0xc3e
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer.restfulListResource.func1(0xc0001aab40, 0xc0004fac60)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/kubernetes-incubator/custom-metrics-apiserver/pkg/apiserver/installer/installer.go:288 +0x8f
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc0001aab40, 0xc0004fac60)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:272 +0x254
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc000320480, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful/container.go:277 +0x985
github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(...)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/github.com/emicklei/go-restful/container.go:199
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x192efce, 0x16, 0xc000320480, 0xc000299880, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4e4
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4fa
net/http.HandlerFunc.ServeHTTP(0xc00007b2c0, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x5c7
net/http.HandlerFunc.ServeHTTP(0xc0003b07e0, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ec3
net/http.HandlerFunc.ServeHTTP(0xc00007b300, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931400)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fbf4ccb44c8, 0xc00000e348, 0xc000931300)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:81 +0x527
net/http.HandlerFunc.ServeHTTP(0xc00037fb30, 0x7fbf4ccb44c8, 0xc00000e348, 0xc000931300)
        /usr/local/go/src/net/http/server.go:1995 +0x44
github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc0004db560, 0xc000215a40, 0x1c04480, 0xc00000e348, 0xc000931300)
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:111 +0xb3
created by github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
        /go/src/github.com/Azure/azure-k8s-metrics-adapter/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:98 +0x1b1

At a more meta level, I am not sure you would want to scale on just the number of messages incoming. I would think you would either want scale based on the lag of the messages being processed or number of unprocessed messages. It doesn't look like either of those are captured via the Azure Metrics api. KEDA does have support for all the metrics you are currenlty scaling on, including supporting the number of unprocessed messages in eventhub.

Vishal2696 · 2020-03-11T06:37:14Z

@jsturtevant
Following is the output when I enter the command
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/mynamespace/extmetric01-eventhub"
Output:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

kubectl describe apiservices v1beta1.external.metrics.k8s.io
Output:

Name: v1beta1.external.metrics.k8s.io
Namespace:
Labels:
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2020-02-13T13:10:23Z
Resource Version: 15729871
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
UID: 31955796-4e62-11ea-ad67-1a32613f77cc
Spec:
Group: external.metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: azureadapter-azure-k8s-metrics-adapter
Namespace: custom-metrics
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2020-03-03T23:50:49Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events:

Coming to your point "It appears that IncomingMessages doesn't return any values through the Metric API at the Eventhub level which is where you are filtering with filter: EntityName eq 'customname'" I'm able to see metric graph for the metric "IncomingMessages" from the portal at eventhub level too. Below is a screenshot for your reference.

Let me know if I've got anything wrong here and coming to KEDA I cannot do that since it supports only if I enable capture, as it can currently get from storage account only. This won't work for me for many reasons and cost is also one of them.

Your efforts on this are to be appreciated!

jsturtevant · 2020-03-13T01:03:28Z

It seems when there are no messages for a period of time the result metric api for IncomingMessage returns an empty time series. After putting load on it for awhile a while I was getting a correct result back but after letting it sit for awhile I am getting the error again. The initial time for that value to come back was over several hours though.

Although I understand your current cost constraint, the delay in the initial metrics showing up initially causes me to hesitate that this metric would be correct metric to scale on. As well, this metric might not work well as it will return the current Messages coming in which might peak then drop zero even though you haven't made a dent in the queue. Using this metric, you could be in a situation where there are no/low number of Incoming Messages which would cause the HPA to scale down but the processing pods and yet you could have a really high quantity in the event queue.

KEDA does support the Azure Monitor metrics so you could use it to scale on this same metric there if you wanted thought I think you will run into the same challenges with this particular metric.

Vishal2696 · 2020-03-13T06:35:17Z

@jsturtevant Yeah saw that KEDA has launched 1.3.0 recently with the Azure monitor scaler. Gonna give it a try now. Will update you how that goes, Also with respect to the metric behavior I think that is how it is supposed to behave as there are checkpoints and other stuff. Anyway I'll discuss with my team and update you.

Vishal2696 · 2020-03-25T10:05:56Z

@jsturtevant Just now saw the exact same error in KEDA too. They closed your issue but it seems like they haven't fixed it. I've requested to reopen your issue again there.

jsturtevant closed this as completed Mar 8, 2020

jsturtevant reopened this Mar 9, 2020

jsturtevant mentioned this issue Mar 13, 2020

Azure Monitor Scaler will panic if the timeseries has no data kedacore/keda#679

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External metric not working for Eventhubs #90

External metric not working for Eventhubs #90

Vishal2696 commented Feb 27, 2020 •

edited

Loading

jsturtevant commented Mar 8, 2020

Vishal2696 commented Mar 8, 2020 •

edited

Loading

jsturtevant commented Mar 9, 2020

Vishal2696 commented Mar 9, 2020 •

edited

Loading

jsturtevant commented Mar 10, 2020

jsturtevant commented Mar 10, 2020

jsturtevant commented Mar 10, 2020 •

edited

Loading

Vishal2696 commented Mar 11, 2020 •

edited

Loading

jsturtevant commented Mar 13, 2020

Vishal2696 commented Mar 13, 2020

Vishal2696 commented Mar 25, 2020

External metric not working for Eventhubs #90

External metric not working for Eventhubs #90

Comments

Vishal2696 commented Feb 27, 2020 • edited Loading

jsturtevant commented Mar 8, 2020

Vishal2696 commented Mar 8, 2020 • edited Loading

jsturtevant commented Mar 9, 2020

Vishal2696 commented Mar 9, 2020 • edited Loading

jsturtevant commented Mar 10, 2020

jsturtevant commented Mar 10, 2020

jsturtevant commented Mar 10, 2020 • edited Loading

Vishal2696 commented Mar 11, 2020 • edited Loading

jsturtevant commented Mar 13, 2020

Vishal2696 commented Mar 13, 2020

Vishal2696 commented Mar 25, 2020

Vishal2696 commented Feb 27, 2020 •

edited

Loading

Vishal2696 commented Mar 8, 2020 •

edited

Loading

Vishal2696 commented Mar 9, 2020 •

edited

Loading

jsturtevant commented Mar 10, 2020 •

edited

Loading

Vishal2696 commented Mar 11, 2020 •

edited

Loading