[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 · 2024-04-25T17:11:09Z

Description of the changes

Part of [jaeger-v2] Support sampling strategies #5531
adds static sampling support for otel-based jaeger-v2.
supports hot reload
added unit tests

How was this change tested?

go run -tags=ui ./cmd/jaeger --config ./cmd/jaeger/config-badger.yaml
make test

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: yarn lint and yarn test

codecov · 2024-04-25T17:20:47Z

Codecov Report

Attention: Patch coverage is 26.73611% with 211 lines in your changes missing coverage. Please review.

Project coverage is 95.15%. Comparing base (1c1bc08) to head (1a2c505).

Files	Patch %	Lines
...ger/internal/extension/remotesampling/extension.go	29.33%	91 Missing and 15 partials ⚠️
.../internal/processors/adaptivesampling/processor.go	0.00%	35 Missing ⚠️
...er/internal/processors/adaptivesampling/factory.go	0.00%	20 Missing ⚠️
...jaeger/internal/extension/remotesampling/config.go	15.00%	15 Missing and 2 partials ⚠️
...ugin/sampling/strategyprovider/adaptive/options.go	0.00%	13 Missing ⚠️
...n/sampling/strategyprovider/adaptive/aggregator.go	0.00%	10 Missing ⚠️
pkg/clientcfg/clientcfghttp/handler.go	0.00%	7 Missing ⚠️
...ger/internal/processors/adaptivesampling/config.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5389      +/-   ##
==========================================
- Coverage   96.38%   95.15%   -1.23%     
==========================================
  Files         329      335       +6     
  Lines       16060    16347     +287     
==========================================
+ Hits        15479    15555      +76     
- Misses        404      598     +194     
- Partials      177      194      +17

Flag	Coverage Δ
badger_v1	`8.04% <ø> (ø)`
badger_v2	`1.92% <0.00%> (ø)`
cassandra-3.x-v1	`16.60% <ø> (ø)`
cassandra-3.x-v2	`1.84% <0.00%> (ø)`
cassandra-4.x-v1	`16.60% <ø> (ø)`
cassandra-4.x-v2	`1.84% <0.00%> (ø)`
elasticsearch-7.x-v1	`18.87% <ø> (-0.02%)`	⬇️
elasticsearch-8.x-v1	`19.07% <ø> (ø)`
elasticsearch-8.x-v2	`19.07% <ø> (-0.02%)`	⬇️
grpc_v1	`9.47% <ø> (ø)`
grpc_v2	`7.47% <0.00%> (-0.02%)`	⬇️
kafka	`9.76% <ø> (ø)`
opensearch-1.x-v1	`18.93% <ø> (ø)`
opensearch-2.x-v1	`18.93% <ø> (ø)`
opensearch-2.x-v2	`18.92% <ø> (ø)`
unittests	`93.04% <26.73%> (-1.19%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cmd/jaeger/internal/extension/jaegerstaticsampling/config.go

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md

Pushkarm029 · 2024-04-26T21:16:26Z

I am thinking of this approach.:

create a gRPC sampling server
then, register api_v2.RegisterSamplingManagerServer(server, sampling.NewGRPCHandler(SamplingStore))
similarly, create an HTTP server and register a sampling API.

That's how it works in the current Jaeger collector, but we are serving this API in the collector server. Here, we have to create a new one.

suggestions

Pushkarm029 · 2024-04-26T21:17:34Z

One more question:

How can I run hotrod with jaeger-v2 binary?
I tried this, but Jaeger UI was not able to detect Hotrod service.
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 go run ./main.go all
go run -tags=ui ./cmd/jaeger --config ./cmd/jaeger/config-elasticsearch.yaml

yurishkuro · 2024-04-27T01:15:44Z

We need a remotesampling extension which performs the following duties:

creates HTTP server that returns sampling strategies as JSON
allows user a choice of a sampling strategy store
- file: a static file with auto-reload
- adaptive: a backend storage that implements interfaces in sampling/strategystore package

We also need an adaptivesampling processor that

uses remotesampling to obtain storage backend (similar how jaegerexporter uses jaegerextension to get storage)
process spans and updates strategies in that storage (which then will be read by the extension's server)

All of this should require very minimal implementation, primarily just wiring up the existing components already implemented in app/collector/main

yurishkuro · 2024-04-29T15:59:38Z

The existing component (from jaeger v1) are in blue:

flowchart LR
    Receiver --> AdaptiveSamplingProcessor --> BatchProcessor --> Exporter
    Exporter -->|"(1) get storage"| JaegerStorageExension
    Exporter -->|"(2) write trace"| TraceStorage
    AdaptiveSamplingProcessor -->|"getStorage()"| StorageConfig

    OTEL_SDK[OTEL
             SDK]
    OTEL_SDK -->|"(1) GET /sampling"| HTTP_endpoint
    HTTP_endpoint -->|"(2) getStrategy()"| StrategiesProvider
    style HTTP_endpoint fill:blue,color:white

    subgraph Jaeger Collector
        Receiver
        BatchProcessor[Batch
                       Processor]
        Exporter        
        TraceStorage[(Trace
                      Storage)]
        AdaptiveSamplingProcessor[Adaptive
                                  Sampling
                                  Processor]
        AdaptiveSamplingProcessorV1[Adaptive
                                    Sampling
                                    Processor_v1]
        style AdaptiveSamplingProcessorV1 fill:blue,color:white
        AdaptiveSamplingProcessor -->|"[]*model.Span"| AdaptiveSamplingProcessorV1
        AdaptiveSamplingProcessorV1 ---|use| SamplingStorage

        subgraph JaegerStorageExension[Jaeger Storage Exension]
            Storage[[Storage
                     Config]]
        end
        subgraph RemoteSamplingExtension[Remote Sampling Extension]
            StrategiesProvider -->|"(3b) getStrategy()"| AdaptiveProvider
            StrategiesProvider -->|"(3a) getStrategy()"| FileProvider
            FileProvider --> FileConfig
            AdaptiveProvider --> StorageConfig
    
            HTTP_endpoint[HTTP
                          endpoint]
            StrategiesProvider[Strategies
                               Provider]
            FileProvider[File
                         Provider]
            AdaptiveProvider[Adaptive
                             Provider]
            style StrategiesProvider fill:blue,color:white
            style FileProvider fill:blue,color:white
            style AdaptiveProvider fill:blue,color:white
            subgraph Config
                FileConfig[[File Config]]
                StorageConfig[[Storage Config]]
            end
            StorageConfig --- SamplingStorage
            SamplingStorage[(Sampling
                             Storage)]
            style SamplingStorage fill:blue,color:white
        end
    end

cmd/jaeger/config-badger.yaml

Pushkarm029 · 2024-05-04T14:38:36Z

i haven;t pushed latest changes yet.

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md

pkg/clientcfg/clientcfghttp/handler.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

cmd/jaeger/internal/extension/remotesampling/extension_test.go

cmd/jaeger/config-badger.yaml

yurishkuro

looks good, in the right direction

cmd/jaeger/config-badger.yaml

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/extension/remotesampling/factory.go

cmd/jaeger/internal/extension/remotesampling/internal/http.go

cmd/jaeger/internal/extension/remotesampling/testdata/strategy.json

cmd/jaeger/internal/extension/remotesampling/config.go

pkg/clientcfg/clientcfghttp/handler.go

cmd/jaeger/internal/extension/remotesampling/extension.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

cmd/jaeger/internal/extension/remotesampling/internal/http.go

cmd/jaeger/internal/extension/remotesampling/config.go

## Which problem is this PR solving? - part of #5389 ## Description of the changes - processor is co-located in strategy_store and aggregator. - In aggregator to run `generateStrategyResponses`, `runCalculationLoop`. - In strategy_store to run `loadProbabilities`, `runUpdateProbabilitiesLoop` ## How was this change tested? - `make test` ## Checklist - [x] I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md - [x] I have signed all commits - [x] I have added unit tests for the new functionality - [x] I have run lint and test steps successfully - for `jaeger`: `make lint test` - for `jaeger-ui`: `yarn lint` and `yarn test` --------- Signed-off-by: Pushkar Mishra <[email protected]> Signed-off-by: pushkarm029 <[email protected]>

yurishkuro

how do you plan to test it?

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/extension/remotesampling/factory.go

cmd/jaeger/internal/processors/adaptivesampling/config.go

plugin/sampling/strategystore/adaptive/aggregator.go

cmd/jaeger/internal/extension/remotesampling/extension.go

yurishkuro · 2024-05-25T21:34:54Z

cmd/jaeger/config-badger.yaml

+    # file:
+    #   path: ./cmd/jaeger/sampling-strategies.json


maybe we should default to file, not to more complex adaptive?

## Which problem is this PR solving? - #5389 (comment) ## Description of the changes - Refactored `handleRootSpan` logic into a helper method in aggregator.go.

cmd/jaeger/internal/extension/remotesampling/config.go

cmd/jaeger/internal/processors/adaptivesampling/processor.go

yurishkuro

please fix the DCO and keep it green

cmd/jaeger/internal/extension/remotesampling/extension.go

## Which problem is this PR solving? - jaegertracing#5389 (comment) ## Description of the changes - Refactored `handleRootSpan` logic into a helper method in aggregator.go. Signed-off-by: Vamshi Maskuri <[email protected]>

yurishkuro · 2024-06-06T16:43:10Z

@Pushkarm029 are you able to fix DCO? It's confused by many unrelated changes.

I would recommend squashing all commit, e.g.

git reset --soft HEAD~40  # to the start of your changes
git stash # save the diffs
git rebase main # (make sure main is up to date before doing this)
git stash apply # may need to resolve a couple of conflicts

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 · 2024-06-06T18:14:51Z

Fixed

Signed-off-by: Yuri Shkuro <[email protected]>

cmd/jaeger/internal/extension/remotesampling/extension.go

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro · 2024-06-18T01:16:29Z

Note for future: I restored most config files to original state, it's too early to introduce adaptive sampling parts into them, we should first validate that with memstore. We can come back to that later - commit adda2b2

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro

@Pushkarm029 this looks to be in a good shape, we just need to bump test coverage. I verified that the server starts and serves sampling strategies over HTTP.

Signed-off-by: Yuri Shkuro <[email protected]>

Pushkarm029 · 2024-06-18T05:17:36Z

@Pushkarm029 this looks to be in a good shape, we just need to bump test coverage. I verified that the server starts and serves sampling strategies over HTTP.

thanks, I will start writing tests now.

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 · 2024-06-30T09:17:45Z

Halfway🛣️

Pushkarm029 requested a review from a team as a code owner April 25, 2024 17:11

Pushkarm029 requested a review from pavolloffay April 25, 2024 17:11

yurishkuro reviewed Apr 25, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/config.go Outdated Show resolved Hide resolved

yurishkuro reviewed Apr 25, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md Outdated Show resolved Hide resolved

Pushkarm029 changed the title ~~[jaeger-v2] Add static sampling extension~~ [jaeger-v2] Add remotesampling extension May 4, 2024

Pushkarm029 marked this pull request as draft May 4, 2024 08:11

yurishkuro reviewed May 4, 2024

View reviewed changes

cmd/jaeger/config-badger.yaml Outdated Show resolved Hide resolved

yurishkuro reviewed May 4, 2024

View reviewed changes

cmd/jaeger/internal/extension/jaegerstaticsampling/README.md Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

pkg/clientcfg/clientcfghttp/handler.go Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/internal/processors/adaptivesampling/processor.go Outdated Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/extension_test.go Show resolved Hide resolved

Pushkarm029 commented May 6, 2024

View reviewed changes

cmd/jaeger/config-badger.yaml Outdated Show resolved Hide resolved

yurishkuro added the changelog:new-feature Change that should be called out as new feature in CHANGELOG label May 7, 2024

yurishkuro reviewed May 7, 2024

View reviewed changes

yurishkuro reviewed May 10, 2024

View reviewed changes

Pushkarm029 mentioned this pull request May 11, 2024

Refactor Adaptive Sampling Aggregator & Strategy Store #5441

Merged

4 tasks

yurishkuro reviewed May 23, 2024

View reviewed changes

Pushkarm029 mentioned this pull request May 23, 2024

[refactor] move root span handler into aggregator #5478

Merged

4 tasks

yurishkuro reviewed May 25, 2024

View reviewed changes

yurishkuro reviewed May 26, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/config.go Show resolved Hide resolved

yurishkuro reviewed May 26, 2024

View reviewed changes

cmd/jaeger/internal/processors/adaptivesampling/processor.go Outdated Show resolved Hide resolved

yurishkuro reviewed Jun 2, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/extension.go Outdated Show resolved Hide resolved

rebase

e4ab125

Signed-off-by: pushkarm029 <[email protected]>

Pushkarm029 force-pushed the sampling_extension branch from 153f9ab to e4ab125 Compare June 6, 2024 18:12

Merge branch 'main' into sampling_extension

61ee980

yurishkuro and others added 4 commits June 14, 2024 13:57

Merge branch 'main' into sampling_extension

4d188a9

Merge branch 'main' into sampling_extension

0b6b276

fix lint issues

0da4655

Signed-off-by: Yuri Shkuro <[email protected]>

fix

aff467e

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro mentioned this pull request Jun 16, 2024

[adaptive] Separate Provider and Aggregator settings #5639

Closed

4 tasks

yurishkuro reviewed Jun 17, 2024

View reviewed changes

cmd/jaeger/internal/extension/remotesampling/extension.go Outdated Show resolved Hide resolved

yurishkuro added 3 commits June 17, 2024 20:41

Merge branch 'main' into sampling_extension

1aaf084

Signed-off-by: Yuri Shkuro <[email protected]>

refactor

75f0ad1

Signed-off-by: Yuri Shkuro <[email protected]>

restore most configs

adda2b2

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro added 4 commits June 17, 2024 22:04

make config sections optional

fe93d6e

Signed-off-by: Yuri Shkuro <[email protected]>

fmt

ae73998

Signed-off-by: Yuri Shkuro <[email protected]>

better error logging

c5f9544

Signed-off-by: Yuri Shkuro <[email protected]>

add tests

88ec66f

Signed-off-by: Yuri Shkuro <[email protected]>

yurishkuro reviewed Jun 18, 2024

View reviewed changes

fix

ff73c57

Signed-off-by: Yuri Shkuro <[email protected]>

Pushkarm029 added 2 commits June 23, 2024 10:56

Merge branch 'main' into sampling_extension

175e682

fix lint

1a2c505

Signed-off-by: pushkarm029 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jaeger-v2] Add `remotesampling` extension #5389

[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 commented Apr 25, 2024 •

edited by yurishkuro

Loading

codecov bot commented Apr 25, 2024 •

edited

Loading

Pushkarm029 commented Apr 26, 2024

Pushkarm029 commented Apr 26, 2024

yurishkuro commented Apr 27, 2024

yurishkuro commented Apr 29, 2024

Pushkarm029 commented May 4, 2024

yurishkuro left a comment

yurishkuro left a comment

yurishkuro May 25, 2024

yurishkuro left a comment

yurishkuro commented Jun 6, 2024

Pushkarm029 commented Jun 6, 2024

yurishkuro commented Jun 18, 2024

yurishkuro left a comment

Pushkarm029 commented Jun 18, 2024

Pushkarm029 commented Jun 30, 2024

[jaeger-v2] Add remotesampling extension #5389

Are you sure you want to change the base?

[jaeger-v2] Add remotesampling extension #5389

Conversation

Pushkarm029 commented Apr 25, 2024 • edited by yurishkuro Loading

Description of the changes

How was this change tested?

Checklist

codecov bot commented Apr 25, 2024 • edited Loading

Codecov Report

Pushkarm029 commented Apr 26, 2024

Pushkarm029 commented Apr 26, 2024

yurishkuro commented Apr 27, 2024

yurishkuro commented Apr 29, 2024

Pushkarm029 commented May 4, 2024

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro May 25, 2024

Choose a reason for hiding this comment

yurishkuro left a comment

Choose a reason for hiding this comment

yurishkuro commented Jun 6, 2024

Pushkarm029 commented Jun 6, 2024

yurishkuro commented Jun 18, 2024

yurishkuro left a comment

Choose a reason for hiding this comment

Pushkarm029 commented Jun 18, 2024

Pushkarm029 commented Jun 30, 2024

[jaeger-v2] Add `remotesampling` extension #5389

[jaeger-v2] Add `remotesampling` extension #5389

Pushkarm029 commented Apr 25, 2024 •

edited by yurishkuro

Loading

codecov bot commented Apr 25, 2024 •

edited

Loading