Unable to configure opentelemetry and prometheus plugins in standalone mode

[ronnybremer]

Description

Apisix is running fine in standalone mode, providing upstream access to some API routes as well as a PHP middleware. However, I would like to integrate Opentelemetry for tracing and Prometheus for metrics. Having tried various approaches, I can't get it to work.

Opentelemetry:
Plugin is enabled as I can see the OT parent span information arriving in the API backend
Plugin cannot transmit data to the OTEL collector, all I can see in the docker logs is:

apisix-1  | 2024/06/25 07:59:56 [error] 34#34: *1719 [lua] http_client.lua:46: do_request(): request failed: connection refused, context: ngx.timer, client: 172.31.0.1, server: 0.0.0.0:9443

172.31.0.1 is the IP address of the running docker image of Apisix. I can not observe any traffic to port 4318 from inside the docker network, so to me it looks like the default collector address of 127.0.0.1:4318 is used. But the LUA error in the log does not say anything about which address it tried to connect to.

I will paste my full configuration (redacted slightly) below.

Prometheus:
Since I am in standalone mode I guess I need to export the plugin URI via public-api. I tried that but can't access that URL afterwards. I am not sure what to specify as the upstream URI as the API port is not listening in standalone mode.

My current config:

ssls:
  -
    cert: |
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
    key: |
      -----BEGIN PRIVATE KEY-----
    snis:
      - "test.local.lan"

global_rules:
  - id: 1
    plugins:
      response-rewrite:
        headers:
          set:
            X-Content-Type-Options: nosniff
          remove:
            - Server

# I tried configuring the plugin here (found this in How to configure plugins in the standalone section of the documentation)
plugins:
  - name: opentelemetry
    resource:
      service.name: APISIX          # Set the service name for OpenTelemetry traces.
      service.version: 3.9.0
      service.instance.id: test-01
    collector:
      address: otelcollect.local.lan:4318
  - name: prometheus

# I tried configuring the plugin here (found this in the opentelemetry plugin section of the documentation)
plugin_attr:
  opentelemetry:      # Plugin: opentelemetry
    resource:
      service.name: APISIX          # Set the service name for OpenTelemetry traces.
      service.version: 3.9.0
      service.instance.id: test-01
    collector:
      address: otelcollect.local.lan:4318
  prometheus:
    enable_export_server: false

routes:
  -
    methods: ["GET", "POST"]
    uris: [ "/portal" ]
    plugins:
      redirect:
        http_to_https: true
      opentelemetry: # the plugin does get activated as the OT headers are present in each request
        sampler:
          name: always_on
    upstream:
        nodes:
            "upstream.local.lan:80": 1
        type: chash
        hash_on: cookie
        key: cookie_PHPSESSID

Any help would be highly appreciated.

Environment

• APISIX version (run apisix version): 3.9.0
• Operating system (run uname -a): Debian docker image running on Rocky Linux 8
• OpenResty / Nginx version (run openresty -V or nginx -V): 1.25.3.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): none, standalone mode
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):