Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASM] Standalone Billing Propagation #5743

Draft
wants to merge 27 commits into
base: master
Choose a base branch
from
Draft

[ASM] Standalone Billing Propagation #5743

wants to merge 27 commits into from
+503 −224

Conversation

e-n-0
Copy link
Member

@e-n-0 e-n-0 commented Jun 26, 2024

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

@e-n-0 e-n-0 force-pushed the flavien/asm/asm/standalone-billing-part2 branch from 1838696 to 1b69b2b Compare June 26, 2024 13:39
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024 •
edited
Loading

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

169 occurrences of :

-      _sampling_priority_v1: 1.0
+      _sampling_priority_v1: 2.0

@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jun 26, 2024 •
edited
Loading

Datadog Report

Branch report: flavien/asm/asm/standalone-billing-part2
Commit report: d62d1ce
Test service: dd-trace-dotnet

✅ 0 Failed, 431052 Passed, 2658 Skipped, 25h 35m 15.84s Total Time

@e-n-0 e-n-0 force-pushed the flavien/asm/asm/standalone-billing-part2 branch from 1b69b2b to 39278c7 Compare June 27, 2024 11:44
@andrewlock
Copy link
Member

andrewlock commented Jun 27, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (73ms)  : 64, 82
     .   : milestone, 73,
    master - mean (74ms)  : 63, 85
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (1,022ms)  : 1003, 1042
     .   : milestone, 1022,
    master - mean (1,019ms)  : 997, 1040
     .   : milestone, 1019,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (109ms)  : 105, 114
     .   : milestone, 109,
    master - mean (109ms)  : 107, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (713ms)  : 690, 735
     .   : milestone, 713,
    master - mean (706ms)  : 685, 728
     .   : milestone, 706,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (93ms)  : 90, 96
     .   : milestone, 93,
    master - mean (93ms)  : 89, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (667ms)  : 637, 698
     .   : milestone, 667,
    master - mean (665ms)  : 644, 687
     .   : milestone, 665,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (191ms)  : 188, 194
     .   : milestone, 191,
    master - mean (191ms)  : 188, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (1,118ms)  : 1095, 1142
     .   : milestone, 1118,
    master - mean (1,116ms)  : 1090, 1141
     .   : milestone, 1116,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (277ms)  : 271, 282
     .   : milestone, 277,
    master - mean (275ms)  : 271, 280
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (884ms)  : 862, 907
     .   : milestone, 884,
    master - mean (879ms)  : 849, 909
     .   : milestone, 879,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5743) - mean (265ms)  : 260, 269
     .   : milestone, 265,
    master - mean (265ms)  : 259, 271
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5743) - mean (863ms)  : 839, 887
     .   : milestone, 863,
    master - mean (863ms)  : 842, 885
     .   : milestone, 863,
Loading

@andrewlock
Copy link
Member

andrewlock commented Jun 27, 2024 •
edited
Loading

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5743) (11.879M)   : 0, 11878565
    master (11.956M)   : 0, 11956052
    benchmarks/2.9.0 (11.967M)   : 0, 11966853

    section Automatic
    This PR (5743) (8.030M)   : 0, 8029898
    master (8.134M)   : 0, 8133534
    benchmarks/2.9.0 (8.405M)   : 0, 8404892

    section Trace stats
    master (8.463M)   : 0, 8462714

    section Manual
    This PR (5743) (10.197M)   : 0, 10196530
    master (10.444M)   : 0, 10443993

    section Manual + Automatic
    This PR (5743) (7.626M)   : 0, 7626342
    master (7.695M)   : 0, 7694938

    section Version Conflict
    master (6.851M)   : 0, 6850859
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5743) (9.648M)   : 0, 9648212
    master (9.502M)   : 0, 9502331
    benchmarks/2.9.0 (9.255M)   : 0, 9255366

    section Automatic
    This PR (5743) (6.616M)   : 0, 6616123
    master (6.589M)   : 0, 6589204

    section Trace stats
    master (6.630M)   : 0, 6630265

    section Manual
    This PR (5743) (8.170M)   : 0, 8170451
    master (8.176M)   : 0, 8176343

    section Manual + Automatic
    This PR (5743) (6.162M)   : 0, 6161959
    master (6.047M)   : 0, 6047122

    section Version Conflict
    master (5.500M)   : 0, 5500346
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5743) (9.511M)   : 0, 9511182
    master (10.201M)   : 0, 10201473
    benchmarks/2.9.0 (10.302M)   : 0, 10302396

    section Automatic
    This PR (5743) (6.866M)   : 0, 6866396
    master (7.200M)   : 0, 7200433
    benchmarks/2.9.0 (7.635M)   : 0, 7634994

    section Trace stats
    master (7.395M)   : 0, 7395068

    section Manual
    This PR (5743) (8.581M)   : 0, 8581282
    master (9.027M)   : 0, 9026978

    section Manual + Automatic
    This PR (5743) (6.686M)   : 0, 6686091
    master (6.888M)   : 0, 6888149

    section Version Conflict
    master (6.270M)   : 0, 6269938
Loading

@andrewlock
Copy link
Member

andrewlock commented Jun 27, 2024 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #5743 compared to master:

  • 4 benchmarks are faster, with geometric mean 1.150
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.75μs 38.8ns 178ns 0.0149 0.00745 0 5.43 KB
master StartStopWithChild netcoreapp3.1 10.3μs 56.5ns 348ns 0.0199 0.00995 0 5.61 KB
master StartStopWithChild net472 16.7μs 49.3ns 191ns 1.02 0.317 0.0916 6.07 KB
#5743 StartStopWithChild net6.0 7.65μs 43.4ns 316ns 0.0146 0.00365 0 5.43 KB
#5743 StartStopWithChild netcoreapp3.1 9.97μs 56ns 384ns 0.0149 0.00498 0 5.62 KB
#5743 StartStopWithChild net472 16.3μs 60.2ns 225ns 1.01 0.301 0.095 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 467μs 275ns 1.06μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 652μs 116ns 418ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 820μs 323ns 1.21μs 0.411 0 0 3.3 KB
#5743 WriteAndFlushEnrichedTraces net6.0 459μs 307ns 1.15μs 0 0 0 2.7 KB
#5743 WriteAndFlushEnrichedTraces netcoreapp3.1 646μs 279ns 1.04μs 0 0 0 2.7 KB
#5743 WriteAndFlushEnrichedTraces net472 840μs 251ns 940ns 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 170μs 212ns 822ns 0.17 0 0 18.45 KB
master SendRequest netcoreapp3.1 192μs 414ns 1.6μs 0.191 0 0 20.61 KB
master SendRequest net472 3.03E‑05ns 2.61E‑05ns 0.000101ns 0 0 0 0 b
#5743 SendRequest net6.0 168μs 214ns 829ns 0.169 0 0 18.45 KB
#5743 SendRequest netcoreapp3.1 193μs 306ns 1.19μs 0.191 0 0 20.61 KB
#5743 SendRequest net472 0.000787ns 0.00031ns 0.0012ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 561μs 651ns 2.52μs 0.556 0 0 41.69 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 647μs 937ns 3.63μs 0.326 0 0 41.65 KB
master WriteAndFlushEnrichedTraces net472 888μs 4.24μs 16.4μs 8.3 2.62 0.437 53.25 KB
#5743 WriteAndFlushEnrichedTraces net6.0 539μs 331ns 1.28μs 0.534 0 0 41.74 KB
#5743 WriteAndFlushEnrichedTraces netcoreapp3.1 666μs 1.21μs 4.68μs 0.331 0 0 41.59 KB
#5743 WriteAndFlushEnrichedTraces net472 841μs 2.67μs 10.3μs 8.39 2.52 0.419 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.3μs 0.588ns 2.2ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.7μs 1.5ns 5.83ns 0.0135 0 0 1.02 KB
master ExecuteNonQuery net472 1.94μs 0.493ns 1.91ns 0.156 0 0 987 B
#5743 ExecuteNonQuery net6.0 1.24μs 0.325ns 1.22ns 0.0142 0 0 1.02 KB
#5743 ExecuteNonQuery netcoreapp3.1 1.71μs 0.989ns 3.83ns 0.0138 0 0 1.02 KB
#5743 ExecuteNonQuery net472 1.96μs 3.62ns 13.5ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.13μs 0.645ns 2.5ns 0.0135 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.57μs 2.44ns 9.44ns 0.0133 0 0 976 B
master CallElasticsearch net472 2.62μs 2.43ns 9.43ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.25μs 0.356ns 1.33ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.66μs 0.617ns 2.31ns 0.0133 0 0 1.02 KB
master CallElasticsearchAsync net472 2.61μs 2.31ns 8.96ns 0.167 0 0 1.05 KB
#5743 CallElasticsearch net6.0 1.19μs 0.35ns 1.36ns 0.0137 0 0 976 B
#5743 CallElasticsearch netcoreapp3.1 1.51μs 0.407ns 1.52ns 0.0128 0 0 976 B
#5743 CallElasticsearch net472 2.53μs 1.32ns 4.96ns 0.157 0 0 995 B
#5743 CallElasticsearchAsync net6.0 1.24μs 0.385ns 1.44ns 0.013 0 0 952 B
#5743 CallElasticsearchAsync netcoreapp3.1 1.6μs 0.583ns 2.1ns 0.0137 0 0 1.02 KB
#5743 CallElasticsearchAsync net472 2.69μs 1.4ns 5.41ns 0.167 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.21μs 1.96ns 7.61ns 0.0135 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.62μs 1.35ns 5.21ns 0.013 0 0 952 B
master ExecuteAsync net472 1.89μs 1.15ns 4.29ns 0.145 0 0 915 B
#5743 ExecuteAsync net6.0 1.35μs 0.643ns 2.49ns 0.0134 0 0 952 B
#5743 ExecuteAsync netcoreapp3.1 1.66μs 0.682ns 2.64ns 0.0131 0 0 952 B
#5743 ExecuteAsync net472 1.9μs 1.46ns 5.64ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 2.27ns 8.48ns 0.0302 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.09μs 3.31ns 12.8ns 0.0357 0 0 2.76 KB
master SendAsync net472 7.65μs 4.69ns 18.2ns 0.5 0 0 3.15 KB
#5743 SendAsync net6.0 4.12μs 3.3ns 12.3ns 0.031 0 0 2.22 KB
#5743 SendAsync netcoreapp3.1 5.09μs 2.41ns 9.34ns 0.0356 0 0 2.76 KB
#5743 SendAsync net472 7.61μs 2.14ns 8.02ns 0.498 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.42μs 3.83ns 14.8ns 0.0232 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.23μs 0.743ns 2.68ns 0.0213 0 0 1.64 KB
master EnrichedLog net472 2.48μs 1.66ns 6.43ns 0.249 0 0 1.57 KB
#5743 EnrichedLog net6.0 1.51μs 0.79ns 2.96ns 0.0227 0 0 1.64 KB
#5743 EnrichedLog netcoreapp3.1 2.1μs 1.78ns 6.88ns 0.0227 0 0 1.64 KB
#5743 EnrichedLog net472 2.52μs 1.77ns 6.62ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 127ns 490ns 0.0566 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 120μs 229ns 826ns 0.0597 0 0 4.28 KB
master EnrichedLog net472 146μs 94ns 352ns 0.659 0.22 0 4.46 KB
#5743 EnrichedLog net6.0 114μs 251ns 972ns 0.0567 0 0 4.28 KB
#5743 EnrichedLog netcoreapp3.1 118μs 113ns 436ns 0 0 0 4.28 KB
#5743 EnrichedLog net472 149μs 137ns 530ns 0.675 0.225 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.99μs 1.05ns 4.08ns 0.0301 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.12μs 2.28ns 8.54ns 0.0288 0 0 2.2 KB
master EnrichedLog net472 4.77μs 2.47ns 9.57ns 0.319 0 0 2.02 KB
#5743 EnrichedLog net6.0 3.08μs 1.27ns 4.56ns 0.0308 0 0 2.2 KB
#5743 EnrichedLog netcoreapp3.1 4.11μs 2.46ns 9.54ns 0.029 0 0 2.2 KB
#5743 EnrichedLog net472 4.91μs 1.66ns 6.44ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.4μs 0.611ns 2.37ns 0.016 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.72μs 0.782ns 3.03ns 0.0154 0 0 1.14 KB
master SendReceive net472 2.11μs 0.712ns 2.67ns 0.184 0 0 1.16 KB
#5743 SendReceive net6.0 1.34μs 1.27ns 4.91ns 0.016 0 0 1.14 KB
#5743 SendReceive netcoreapp3.1 1.73μs 0.515ns 1.93ns 0.0149 0 0 1.14 KB
#5743 SendReceive net472 2.01μs 1.6ns 6.22ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.69μs 0.846ns 3.28ns 0.0215 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.75μs 1.18ns 4.41ns 0.0226 0 0 1.65 KB
master EnrichedLog net472 4.27μs 2.1ns 7.88ns 0.323 0 0 2.04 KB
#5743 EnrichedLog net6.0 2.77μs 1.69ns 6.54ns 0.0221 0 0 1.6 KB
#5743 EnrichedLog netcoreapp3.1 3.99μs 3.85ns 14.9ns 0.0223 0 0 1.65 KB
#5743 EnrichedLog net472 4.36μs 1.37ns 5.32ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5743

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.195 701.08 586.76
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.133 793.55 700.29

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 466ns 0.134ns 0.5ns 0.00815 0 0 576 B
master StartFinishSpan netcoreapp3.1 598ns 0.399ns 1.49ns 0.00795 0 0 576 B
master StartFinishSpan net472 701ns 0.517ns 2ns 0.0915 0 0 578 B
master StartFinishScope net6.0 543ns 0.38ns 1.47ns 0.00978 0 0 696 B
master StartFinishScope netcoreapp3.1 793ns 0.355ns 1.37ns 0.00955 0 0 696 B
master StartFinishScope net472 832ns 0.505ns 1.96ns 0.104 0 0 658 B
#5743 StartFinishSpan net6.0 446ns 0.136ns 0.529ns 0.00804 0 0 576 B
#5743 StartFinishSpan netcoreapp3.1 577ns 0.333ns 1.29ns 0.00779 0 0 576 B
#5743 StartFinishSpan net472 587ns 0.651ns 2.52ns 0.0917 0 0 578 B
#5743 StartFinishScope net6.0 520ns 0.2ns 0.748ns 0.00979 0 0 696 B
#5743 StartFinishScope netcoreapp3.1 700ns 0.316ns 1.22ns 0.00962 0 0 696 B
#5743 StartFinishScope net472 833ns 0.639ns 2.47ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5743

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.147 733.02 639.17
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 1.127 955.57 848.13

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 733ns 0.77ns 2.67ns 0.00952 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 956ns 0.743ns 2.88ns 0.00957 0 0 696 B
master RunOnMethodBegin net472 1.21μs 0.341ns 1.32ns 0.104 0 0 658 B
#5743 RunOnMethodBegin net6.0 639ns 0.186ns 0.72ns 0.00962 0 0 696 B
#5743 RunOnMethodBegin netcoreapp3.1 848ns 0.526ns 2.04ns 0.00972 0 0 696 B
#5743 RunOnMethodBegin net472 1.1μs 0.531ns 2.06ns 0.104 0 0 658 B

@andrewlock
Copy link
Member

andrewlock commented Jun 27, 2024 •
edited
Loading

Benchmarks Report for appsec 🐌

Benchmarks for #5743 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.228
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5743

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.228 191.12 155.66

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 72.1μs 93.8ns 363ns 0.0715 0 0 6 KB
master AllCycleSimpleBody netcoreapp3.1 62.8μs 96.4ns 374ns 0.0623 0 0 6.95 KB
master AllCycleSimpleBody net472 46.9μs 33.8ns 127ns 1.31 0 0 8.33 KB
master AllCycleMoreComplexBody net6.0 77.2μs 101ns 393ns 0.115 0 0 9.51 KB
master AllCycleMoreComplexBody netcoreapp3.1 70.1μs 95.3ns 369ns 0.105 0 0 10.36 KB
master AllCycleMoreComplexBody net472 54.1μs 14.7ns 51ns 1.86 0.0274 0 11.85 KB
master ObjectExtractorSimpleBody net6.0 191ns 0.0842ns 0.315ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 207ns 0.11ns 0.41ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 173ns 0.163ns 0.631ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.05μs 1.58ns 5.91ns 0.0533 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.05μs 2.87ns 10.8ns 0.0489 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.77μs 3.54ns 12.8ns 0.602 0.00569 0 3.8 KB
#5743 AllCycleSimpleBody net6.0 71.6μs 46.4ns 161ns 0.0715 0 0 6.01 KB
#5743 AllCycleSimpleBody netcoreapp3.1 61.6μs 76.1ns 295ns 0.0916 0 0 6.95 KB
#5743 AllCycleSimpleBody net472 46.8μs 38.7ns 145ns 1.31 0 0 8.34 KB
#5743 AllCycleMoreComplexBody net6.0 76.7μs 82ns 318ns 0.115 0 0 9.51 KB
#5743 AllCycleMoreComplexBody netcoreapp3.1 68.4μs 77.6ns 290ns 0.136 0 0 10.37 KB
#5743 AllCycleMoreComplexBody net472 54.5μs 27.3ns 98.6ns 1.88 0.0272 0 11.85 KB
#5743 ObjectExtractorSimpleBody net6.0 156ns 0.0997ns 0.373ns 0.00394 0 0 280 B
#5743 ObjectExtractorSimpleBody netcoreapp3.1 211ns 0.25ns 0.901ns 0.00367 0 0 272 B
#5743 ObjectExtractorSimpleBody net472 171ns 0.221ns 0.856ns 0.0446 0 0 281 B
#5743 ObjectExtractorMoreComplexBody net6.0 3.28μs 3ns 11.6ns 0.0526 0 0 3.78 KB
#5743 ObjectExtractorMoreComplexBody netcoreapp3.1 3.94μs 1.44ns 5.19ns 0.0514 0 0 3.69 KB
#5743 ObjectExtractorMoreComplexBody net472 3.98μs 7.36ns 28.5ns 0.603 0.00591 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 38.8μs 22ns 85.3ns 0.446 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.1μs 23.1ns 89.5ns 0.43 0 0 32.4 KB
master EncodeArgs net472 67.4μs 38.6ns 145ns 5.14 0.0672 0 32.5 KB
master EncodeLegacyArgs net6.0 79.3μs 26.9ns 104ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 110μs 415ns 1.61μs 0 0 0 2.14 KB
master EncodeLegacyArgs net472 155μs 56ns 217ns 0.31 0 0 2.15 KB
#5743 EncodeArgs net6.0 38.5μs 26.8ns 104ns 0.448 0 0 32.4 KB
#5743 EncodeArgs netcoreapp3.1 54.4μs 24ns 92.8ns 0.434 0 0 32.4 KB
#5743 EncodeArgs net472 66.4μs 52.9ns 205ns 5.14 0.0663 0 32.5 KB
#5743 EncodeLegacyArgs net6.0 77.6μs 20ns 71.9ns 0.036 0 0 2.14 KB
#5743 EncodeLegacyArgs netcoreapp3.1 106μs 218ns 786ns 0 0 0 2.14 KB
#5743 EncodeLegacyArgs net472 156μs 138ns 536ns 0.311 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 181μs 173ns 668ns 0 0 0 2.42 KB
master RunWafRealisticBenchmark netcoreapp3.1 194μs 248ns 960ns 0 0 0 2.37 KB
master RunWafRealisticBenchmark net472 207μs 88.6ns 343ns 0.309 0 0 2.43 KB
master RunWafRealisticBenchmarkWithAttack net6.0 121μs 116ns 450ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 130μs 95.9ns 371ns 0 0 0 1.45 KB
master RunWafRealisticBenchmarkWithAttack net472 140μs 243ns 908ns 0.208 0 0 1.48 KB
#5743 RunWafRealisticBenchmark net6.0 181μs 173ns 670ns 0 0 0 2.42 KB
#5743 RunWafRealisticBenchmark netcoreapp3.1 192μs 143ns 534ns 0 0 0 2.37 KB
#5743 RunWafRealisticBenchmark net472 210μs 85.4ns 331ns 0.313 0 0 2.43 KB
#5743 RunWafRealisticBenchmarkWithAttack net6.0 123μs 193ns 746ns 0 0 0 1.46 KB
#5743 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 205ns 795ns 0 0 0 1.45 KB
#5743 RunWafRealisticBenchmarkWithAttack net472 139μs 39.4ns 152ns 0.21 0 0 1.48 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5743

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 255.35 KB 262.63 KB 7.28 KB 2.85%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 253.64 KB 255.95 KB 2.31 KB 0.91%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 58.5μs 712ns 7.01μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 54.4μs 293ns 1.5μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 38.1μs 89.6ns 323ns 0 0 0 59.65 KB
master StringConcatAspectBenchmark net6.0 295μs 1.66μs 11.3μs 0 0 0 253.64 KB
master StringConcatAspectBenchmark netcoreapp3.1 333μs 1.73μs 7.93μs 0 0 0 255.35 KB
master StringConcatAspectBenchmark net472 308μs 6.69μs 64.2μs 0 0 0 278.53 KB
#5743 StringConcatBenchmark net6.0 51.7μs 123ns 461ns 0 0 0 43.44 KB
#5743 StringConcatBenchmark netcoreapp3.1 63.3μs 832ns 8.28μs 0 0 0 42.64 KB
#5743 StringConcatBenchmark net472 38.3μs 126ns 455ns 0 0 0 59.39 KB
#5743 StringConcatAspectBenchmark net6.0 307μs 1.7μs 10.2μs 0 0 0 255.95 KB
#5743 StringConcatAspectBenchmark netcoreapp3.1 340μs 1.68μs 11.4μs 0 0 0 262.63 KB
#5743 StringConcatAspectBenchmark net472 322μs 7.25μs 71.1μs 0 0 0 278.53 KB

@e-n-0 e-n-0 force-pushed the flavien/asm/asm/standalone-billing-part2 branch 14 times, most recently from d6a3d2f to 68e58b9 Compare July 4, 2024 16:01
e-n-0 added 20 commits July 4, 2024 22:07
@e-n-0
Add new env var key to config_norm_rules.json
d3d0a98
@e-n-0
Fix some tests
a8dfc49
@e-n-0
fix rebase
53932d5
@e-n-0
update snapshots
2c2d516
@e-n-0
Fix agent exception by not using Tracer.Instance singleton
1e6540a
@e-n-0
Change Tags to Metrics
5825a67
@e-n-0
Remove Experimental from the name + Apply bunch of comments
9c58bd7
@e-n-0
Add telemetry record when StatsComputationEnabled was explicitely ena…
4ab4dd8 
…bled by customer but disabled by Standalone
@e-n-0
Fix
e2554e8
@e-n-0
Update snapshots to reflect the change to UserKeep for IAST
ba3639f
@e-n-0
Revert changes, to make it consistant with current implementation
84300af
@e-n-0
Fix snapshot (Don't set to 2, where there is no vuln)
4fe623b
@e-n-0
Add mising Iast set sampling priority on web vuln
5ac33f1
@e-n-0
Apply nit comment: use args names to raw values when more than 1 values
44531a4
@e-n-0
Correctly set the computed stats header when standalone is enabled
6c3aebd
@e-n-0
Change to local root for _dd.apm.enabled:0 metric
d75817c
@e-n-0
fix rebase
eb8915e
@e-n-0
Remove Experimental from the name + Apply bunch of comments
7090d48
@e-n-0
Revert changes, to make it consistant with current implementation
96c10c2
@e-n-0
Implement Propagation
d62d1ce
@e-n-0 e-n-0 force-pushed the flavien/asm/asm/standalone-billing-part2 branch from 68e58b9 to d62d1ce Compare July 4, 2024 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area:asm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@e-n-0 @andrewlock